If the balance of opinion of scientists and policymakers (or those who had briefly heard arguments) was that AI catastrophic risk is high, and that this should be a huge social priority, then you could do a lot of things. For example, you could get budgets of tens of billions of dollars for interpretability research, the way governments already provide tens of billions of dollars of subsidies to strengthen their chip industries. Top AI people would be applying to do safety research in huge numbers. People like Bill Gates and Elon Musk who nominally take AI risk seriously would be doing stuff about it, and Musk could have gotten more traction when he tried to make his case to government.

My perception based on many areas of experience is that policymakers and your AI expert survey respondents on the whole think that these risks are too speculative and not compelling enough to outweigh the gains from advancing AI rapidly (your survey respondents state those are much more likely than the harms). In particular, there is much more enthusiasm for the positive gains from AI than your payoff matrix suggests (particularly among AI researchers), and more mutual fear (e.g. the CCP does not want to be overthrown and subjected to trials for crimes against humanity as has happened to some other regimes, and the rest of the world does not want to live under oppressive CCP dictatorship indefinitely).

But you're proposing that people worried about AI disaster should leapfrog smaller asks of putting a substantial portion of the effort going into accelerating AI into risk mitigation, which we haven't been able to achieve because of low buy-in on the case for risk, to far more costly and demanding asks (on policymakers' views, which prioritize subsidizing AI capabilities and geopolitical competition already). But if you can't get the smaller more cost-effective asks because you don't have buy-in on your risk model, you're going to achieve even less by focusing on more extravagant demands with much lower cost-effectiveness that require massive shifts to make a difference (adding $1B to AI safety annual spending is a big multiplier from the current baseline, removing $1B from semiconductor spending is a miniscule proportional decrease).

When your view is the minority view you have to invest in scientific testing to evaluate your view and make the truth more credible, and better communication. You can't get around failure to convince the world of a problem by just making more extravagant and politically costly demands about how to solve it. It's like climate activists in 1950 responding to difficulties passing funds for renewable energy R&D or a carbon tax by proposing that the sale of automobiles be banned immediately. It took a lot of scientific data, solidification of scientific consensus, and communication/movement-building over time to get current measures on climate change, and the most effective measures actually passed have been ones that minimized pain to the public (and opposition), like supporting the development of better solar energy.

Another analogy in biology: if you're worried about engineered pandemics and it's a struggle to fund extremely cost-effective low-hanging fruit in pandemic prevention, it's not a better strategy to try to ban all general-purpose biomedical technology research.

↑ comment by habryka (habryka4) · 2022-12-23T07:26:00.090Z · LW(p) · GW(p)

I think this comment is overstating the case for policymakers and the electorate actually believing that investing in AI is good for the world. I think the answer currently is "we don't know what policymakers and the electorate actually want in relation to AI" as well as "the relationship of policymakers and the electorate is in the middle of shifting quite rapidly, so past actions are not that predictive of future actions".

I really only have anecdata to go on (though I don't think anyone has much better), but my sense from doing informal polls of e.g. Uber drivers, people on Twitter, and perusing a bunch of Subreddits (which, to be clear, is a terrible sample) is that indeed a pretty substantial fraction of the world is now quite afraid of the consequences of AI, both in a "this change is happening far too quickly and we would like it to slow down" sense, and in a "yeah, I am actually worried about killer robots killing everyone" sense. I think both of these positions are quite compatible with pushing for a broad slow down. There is also a very broad and growing "anti-tech" movement that is more broadly interested in giving less resources to the tech sector, whose aims are at least for a long while compatible with slowing down AGI progress.

My current guess is that policies that are primarily aimed at slowing down and/or heavily regulating AI research are actually pretty popular among the electorate, and I also expect them to be reasonably popular among policymakers, though I also expect their preferences to lag behind the electorate for a while. But again, I really think we don't know, and nobody has run even any basic surveys on the topic yet.

Edit: Inspired by this topic/discussion, I ended up doing some quick google searches for AI opinion polls. I didn't find anything great, but this Pew report has some stuff that's pretty congruent with potential widespread support for AI regulation: https://www.pewresearch.org/internet/2022/03/17/how-americans-think-about-artificial-intelligence/

Replies from: Zach Stein-Perlman, CarlShulman, MichaelA

↑ comment by Zach Stein-Perlman · 2022-12-23T09:24:50.720Z · LW(p) · GW(p)

I ended up doing some quick google searches for AI opinion polls

I collected such polls here, if you want to see more. Most people say they want to regulate AI.

↑ comment by CarlShulman · 2022-12-23T14:52:55.118Z · LW(p) · GW(p)

I agree there is some weak public sentiment in this direction (with the fear of AI takeover being weaker). Privacy protections and redistribution don't particularly favor measures to avoid AI apocalypse.

 I'd also mention this YouGov survey:

But the sentiment looks weak compared to e.g. climate change and nuclear war,  where fossil fuel production and nuclear arsenals continue, although there are significant policy actions taken in hopes of avoiding those problems. The sticking point is policymakers and the scientific community. At the end of the Obama administration the President asked scientific advisors what to make of Bostrom's Superintelligence, and concluded not to pay attention to it because it was not an immediate threat. If policymakers and their advisors and academia and the media think such public concerns are confused, wrongheaded, and not politically powerful they won't work to satisfy them against more pressing concerns like economic growth and national security. This is a lot worse than the situation for climate change, which is why it seems better regulation requires that the expert and elite debate play out differently, or the hope that later circumstances such as dramatic AI progress drastically change views (in favor of AI safety, not the central importance of racing to AI).

Replies from: Kaj_Sotala, peter_hurford

↑ comment by Kaj_Sotala · 2022-12-23T19:24:40.743Z · LW(p) · GW(p)

But the sentiment looks weak compared to e.g. climate change and nuclear war,  where fossil fuel production and nuclear arsenals continue,

That seems correct to me, but on the other hand, I think the public sentiment against things like GMOs was also weaker than the one that we currently have against climate change, and GMOs got slowed down regardless. Also I'm not sure how strong the sentiment against nuclear power was relative to the one against climate change, but in any case, nuclear power got hindered quite a bit too.

I think one important aspect where fossil fuels are different from GMOs and nuclear power is that fossil fuel usage is firmly entrenched across the economy and it's difficult, costly, and slow to replace it. Whereas GMOs were a novel thing and governments could just decide to regulate them and slow them down without incurring major immediate costs. As for nuclear power, it was somewhat entrenched in that there were many existing plants, but society could make the choice to drastically reduce the progress of building new ones - which it did.

Nuclear arsenals don't quite fit this model - in principle, one could have stopped expanding them, but they did keep growing for quite a bit, despite public opposition. Then again, there was an arms race dynamic there. And eventually, nuclear arsenals got cut down in size too.

I think AI is in a sense comparable to nuclear power and GMOs in that there are existing narrow AI applications that would be hard and costly to get rid of, but more general and powerful AI is clearly not yet entrenched due to not having been developed yet. On the other hand, AI labs have a lot of money and there are lots of companies that have significant investments in AI R&D, so that's some level of entrenchment.

Whether nuclear weapons are comparable to AI depends on whether you buy the arguments in the OP for them being different... but seems also relevant that AI arms race arguments are often framed as the US vs. China. That seems reasonable enough, given that the West could probably find consensus on AI as it has found on other matters of regulation, Russia does not seem to be in a shape to compete, and the rest of the world isn't really on the leading edge of AI development. And now it seems like China might not even particularly care about AI [1 [LW(p) · GW(p)], 2 [LW(p) · GW(p)]].

↑ comment by Peter Wildeford (peter_hurford) · 2022-12-25T23:07:11.399Z · LW(p) · GW(p)

I'll shill here and say that Rethink Priorities is pretty good at running polls of the electorate if anyone wants to know what a representative sample of Americans think about a particular issue such as this one. No need to poll Uber drivers or Twitter when you can do the real thing!

Replies from: CarlShulman, Stefan_Schubert

↑ comment by CarlShulman · 2022-12-26T00:21:19.288Z · LW(p) · GW(p)

I'd very much like to see this done with standard high-quality polling techniques, e.g. while airing counterarguments (like support for expensive programs that looks like majority but collapses if higher taxes to pay for them is mentioned). In particular, how the public would react given different views coming from computer scientists/government commissions/panels.

↑ comment by Stefan_Schubert · 2022-12-26T10:16:40.813Z · LW(p) · GW(p)

I think that could be valuable.

It might be worth testing quite carefully for robustness - to ask multiple different questions probing the same issue, and see whether responses converge. My sense is that people's stated opinions about risks from artificial intelligence, and existential risks more generally, could vary substantially depending on framing. Most haven't thought a lot about these issues, which likely contributes. I think a problem problem with some studies on these issues is that researchers over-generalise from highly framing-dependent survey responses.

Replies from: peter_hurford

↑ comment by Peter Wildeford (peter_hurford) · 2022-12-26T15:14:32.616Z · LW(p) · GW(p)

That makes a lot of sense. We can definitely test a lot of different framings. I think the problem with a lot of these kinds of problems is that they are low saliency, and thus people tend not to have opinions already, and thus they tend to generate an opinion on the spot. We have a lot of experience polling on low saliency issues though because we've done a lot of polling on animal farming policy which has similar framing effects.

Replies from: habryka4, Stefan_Schubert

↑ comment by habryka (habryka4) · 2022-12-27T02:33:55.478Z · LW(p) · GW(p)

I would definitely vote in favor of a grant to do this on the LTFF, as well as the SFF, and might even be interested in backstopping it with my personal funds or Lightcone funds.

Replies from: peter_hurford

↑ comment by Peter Wildeford (peter_hurford) · 2022-12-27T17:21:30.076Z · LW(p) · GW(p)

Cool - I'll follow up when I'm back at work.

↑ comment by Stefan_Schubert · 2022-12-26T20:36:26.053Z · LW(p) · GW(p)

I think that's exactly right.

↑ comment by MichaelA · 2023-01-07T13:06:33.762Z · LW(p) · GW(p)

I found this thread interesting and useful, but I feel a key point has been omitted thus far (from what I've read): 

• Public, elite, and policymaker beliefs and attitudes related to AI risk aren't just a variable we (members of the EA/longtermist/AI safety communities) have to bear in mind and operate in light of, but instead also a variable we can intervene on. 
• And so far I'd say we have (often for very good reasons) done significantly less to intervene on that variable than we could've or than we could going forward. 
• So it seems plausible that actually these people are fairly convincible if exposed to better efforts to really explain the arguments in a compelling way.

We've definitely done a significant amount of this kind of work, but I think we've often (a) deliberately held back on doing so or on conveying key parts of the arguments, due to reasonable downside risk concerns, and (b) not prioritized this. And I think there's significantly more we could do if we wanted to, especially after a period of actively building capacity for this. 

Important caveats / wet blankets:

• I think there are indeed strong arguments against trying to shift relevant beliefs and attitudes in a more favorable direction, including not just costs and plausibly low upside but also multiple major plausible downside risks.^[1] 
• So I wouldn't want anyone to take major steps in this direction without checking in with multiple people working on AI safety/governance first. 
• And it's not at all obvious to me we should be doing more of that sort of work. (Though I think whether, how, & when we should is an important question and I'm aware of and excited about a couple small research projects that are happening on that.) 

All I really want to convey in this comment is what I said in my first paragraph: we may be able to significantly push beliefs and opinions in favorable directions relative to where they are now or would be n future by default. 

1. ^{^}

   Due to time constraints, I'll just point to this vague overview.

Thank you, this is a good post.

My main point of disagreement is that you point to successful coordination in things like not eating sand, or not wearing weird clothing. The upside of these things is limited, but you say the upside of superintelligence is also limited because it could kill us.

But rephrase the question to "Should we create an AI that's 1% better than the current best AI?" Most of the time this goes well - you get prettier artwork or better protein folding prediction, and it doesn't kill you. So there's strong upside to building slightly better AIs, as long as you don't cross the "kills everyone" level. Which nobody knows the location of. And which (LW conventional wisdom says) most people will be wrong about.

We successfully coordinate a halt to AI advancement at the first point where more than half of the relevant coordination power agrees that the next 1% step forward is in expectation bad rather than good. But "relevant" is a tough qualifier, because if 99 labs think it's bad, and one lab thinks it's good, then unless there's some centralizing force, the one lab can go ahead and take the step. So "half the relevant coordination power" has to include either every lab agreeing on which 1% step is bad, or the agreement of lots of governments, professional organizations, or other groups that have the power to stop the single most reckless lab.

I think it's possible that we make this work, and worth trying, but that the most likely scenario is that most people underestimate the risk from AI, and so we don't get half the relevant coordination power united around stopping the 1% step that actually creates dangerous superintelligence - which at the time will look to most people like just building a mildly better chatbot with many great social returns.

Katja, many thanks for writing this, and Oliver, thanks for this comment [LW(p) · GW(p)] pointing out that everyday people are in fact worried about AI x-risk.  Since around 2017 when I left MIRI to rejoin academia, I have been trying continually to point out that everyday people are able to easily understand the case for AI x-risk, and that it's incorrect to assume the existence of AI x-risk can only be understood by a very small and select group of people.  My arguments have often been basically the same as yours here: in my case, informal conversations with Uber drivers, random academics, and people at random public social events.  Plus, the argument is very simple: If things are smarter than us, they can outsmart us and cause us trouble.  It's always seemed strange to say there's an "inferential gap" of substance here.

However, for some reason, the idea that people outside the LessWrong community might recognize the existence of AI x-risk — and therefore be worth coordinating with on the issue — has felt not only poorly received on LessWrong, but also fraught to even suggest.  For instance, I tried to point it out in this previous post:

“Pivotal Act” Intentions: Negative Consequences and Fallacious Arguments [LW · GW] 

I wrote the following, targeting multiple LessWrong-adjacent groups in the EA/rationality communities who thought "pivotal acts" with AGI were the only sensible way to reduce AI x-risk:

In fact, before you get to AGI, your company will probably develop other surprising capabilities, and you can demonstrate those capabilities to neutral-but-influential outsiders who previously did not believe those capabilities were possible or concerning.  In other words, outsiders can start to help you implement helpful regulatory ideas, rather than you planning to do it all on your own by force at the last minute using a super-powerful AI system.  

That particular statement was very poorly received, with a 139-karma retort from John Wentworth arguing, 

What exactly is the model by which some AI organization demonstrating AI capabilities will lead to world governments jointly preventing scary AI from being built, in a world which does not actually ban gain-of-function research?

I'm not sure what's going on here, but it seems to me like the idea of coordinating with "outsiders" or placing trust or hope in judgement of "outsiders" has been a bit of taboo here, and that arguments that outsiders are dumb or wrong or can't be trusted will reliably get a lot of cheering in the form of Karma.  

Thankfully, it now also seems to me that perhaps the core LessWrong team has started to think that ideas from outsiders matter more to the LessWrong community's epistemics and/or ability to get things done than previously represented, such as by including material written outside LessWrong in the 2021 LessWrong review posted just a few weeks ago, for the first time:

https://www.lesswrong.com/posts/qCc7tm29Guhz6mtf7/the-lesswrong-2021-review-intellectual-circle-expansion

I consider this a move in a positive direction, but I am wondering if I can draw the LessWrong team's attention to a more serious trend here.  @Oliver, @Raemon, @Ruby, and @Ben Pace, and others engaged in curating and fostering intellectual progress on LessWrong:

Could it be that the LessWrong community, or the EA community, or the rationality community, has systematically discounted the opinions and/or agency of people outside that community, in a way that has lead the community to plan more drastic actions in the world than would otherwise be reasonable if outsiders of that community could also be counted upon to take reasonable actions?

This is a leading question, and my gut and deliberate reasoning have both been screaming "yes" at me for about 5 or 6 years straight, but I am trying to get you guys to take a fresh look at this hypothesis now, in question-form.  Thanks in any case for considering it.

Survey about this question (I have a hypothesis, but I don't want to say what it is yet): https://forms.gle/1R74tPc7kUgqwd3GA

'Nuclear power' seems to me like a weird example because we selectively halted the development of productive use of nuclear power while having comparatively little standing in the way of development of destructive use of nuclear power. If a similar story holds, then we'll still see militarily relevant AIs (deliberately doing adversarial planning of the sort that could lead to human extinction) while not getting many of the benefits along the way.

That... doesn't seem like much of a coordination success story, to me.

The main concrete proposals / ideas that are mentioned here or I can think of are:

1. Work to spread good knowledge regarding AGI risk / doom stuff among AI researchers.
   1. I think everyone is in favor of this, particularly when it’s tied to the less-adversarial takeaway message “there is a big problem, and more safety research is desperately needed”. When it’s tied to the more-adversarial takeaway message “capabilities research should be slowed down”, I think that can be tactically bad, as I think people generally don’t tend to be receptive to people telling them that they and all their scientific idols are being reckless. I think it’s good to be honest and frank, but in the context of outreach, we can be strategic about what we emphasize, and I think emphasizing the “there is a big problem, and more safety research is desperately needed” message is generally the better approach. It also helps that the “there is a big problem, and more safety research is desperately needed” message is pretty clearly correct and easy to argue whereas “capabilities research is harmful” has comparatively more balanced arguments on both sides, and moreover if people can be sufficiently bought into the “there is a is a big problem, and more safety research is desperately needed” message then they can gradually propagate through to the (more complicated) implications on capabilities research.
2. Work to spread good knowledge regarding AGI risk / doom stuff among politicians, the general public, etc.
   1. Basically ditto. Emphasizing “there is a big problem, and more safety research is desperately needed” seems good and is I think uncontroversial. Emphasizing “capabilities research should be slowed down” seems at least uncertain to accomplish what one might hope for it to accomplish, and seems to have many fewer advocates in the community AFAICT. For my part, I do some modest amount of outreach of this sort, with the “there is a big problem, and more safety research is desperately needed” messaging.
3. In support of 1 & 2, do research that may lead to more crisp and rigorous arguments for why AGI doom is likely, if indeed it's likely.
   1. Everyone agrees.
4. Don’t do capabilities research oneself, and try to discourage my friends who are already concerned about AGI risk / doom stuff from doing capabilities research.
   1. Seems to be a very active ongoing debate within the community. I usually find myself on the anti-capabilities-research side of these debates, but it depends on the details.
5. Lay groundwork for possible future capability-slowing-down activities (whether through regulation or voluntary), like efforts to quantify AGI capabilities, track where high-end chips are going, deal with possible future anti-trust issues, start industry groups, get into positions of government influence, etc.
   1. These seem to be some of the main things that “AGI governance” people are doing, right? I’m personally in favor of all of those things, and can’t recall anyone objecting to them. I don’t work on them myself because it’s not my comparative advantage.

So for all of these things, I think they’re worth doing and indeed do them myself to an extent that is appropriate given my strengths, other priorities, etc. [LW · GW]

Are there other things we should be thinking about besides those 5?

The researchers themselves probably don’t want to destroy the world. Many of them also actually agree that AI is a serious existential risk. So in two natural ways, pushing for caution is cooperative with many if not most AI researchers.

I think a possible crux here is that we have different assumptions about how many people think AI existential risk is real & serious versus stupid. I think that right now most people think it’s stupid, and therefore the thing to do right now is to move lots of people from the “it’s stupid” camp to the “it’s real & serious” camp. After we get a lot more people, especially in the AI community but also probably among the general public, out of the “it’s stupid” camp and into the “it’s real & serious” camp, I think a lot more options open up, both governmental and through industry groups or whatever.

I think there are currently too many people in the “it’s stupid” camp and too few in the “it’s real & serious” camp for those types of slowing-down options to be realistically available right now. I could be wrong—I could imagine having my mind changed by survey results or something. Yours says “the median respondent’s probability of x-risk from humans failing to control AI was 10%”, but I have some concerns about response bias, and also a lot of people (even in ML) don’t think about probability the way you or I do, and may mentally treat 10% as “basically 0”, or at least not enough to outweigh the costs of slowing AI. Maybe if there’s a next survey, you could ask directly about receptiveness to broad-based agreements to slow AI research?? I wonder whether you’re in a bubble, or maybe I am, etc.

Anyway, centering the discussion around regulation would (IMO) be productive if we’re in the latter stage where most people already believe that AI existential risk is real & serious, whereas centering the discussion around regulation would at least plausibly be counterproductive if we’re in the former stage where we’re mainly trying to get lots of people to believe that AI existential risk is real & serious as opposed to stupid.

I’m not aware of examples of a technology not getting developed because of concerns over the technology, when most experts and most of the general public thought those concerns were silly. For example, concerns over nuclear power are in fact mostly silly, but had widespread support in the public before they had any impact, I think.

There might be widespread support for regulation to do with social media recommendation algorithm AIs or a few other things like that, but that’s not the topic of concern—AGI timelines are getting shrunk by-and-large by non-public-facing R&D projects, IMO. There is no IRB-like tradition of getting preapproval for running code that’s not intended to leave the four walls of the R&D lab, and I’m not sure what one could do to help such a tradition to get established, at least not before catastrophic AI lab escape accidents, which (if we survive) would be a different topic of discussion.

you are underestimating the degree of unilateralist's curse driving ai progress by quite a bit. it looks like scaling is what does it, but that isn't actually true, the biggest-deal capabilities improvements come from basic algorithms research that is somewhat serially bottlenecked until we improve on a scaling law, and then the scaling is what makes a difference. progress towards dethroning google by strengthening underlying algorithms until they work on individual machines has been swift, and the next generations of advanced basic algorithms are already here, eg https://github.com/BlinkDL/RWKV-LM - most likely, a single 3090 can train a GPT3 level model in a practical amount of time. the illusion that only large agents can train ai is a falsehood resulting from how much easier the current generation of models are to train than previous ones, such that simply scaling them up works at all. but it is incredibly inefficient - transformers are a bad architecture, and the next things after them are shockingly stronger. if your safety plan doesn't take this into account, it won't work.

that said - there's no reason to think we're doomed just because we can't slow down. we need simply speed up safety until it has caught up with the leading edge of capabilities. safety should always be thinking first about how to make the very strongest model's architecture fit in with existing attempts at safety, and safety should take care not to overfit on individual model architectures.

My experience in talking about slowing down AI with people is that they seem to have a can’t do attitude. They don’t want it to be a reasonable course: they want to write it off.

There are a lot of really good reasons why someone would avoid touching the concept of "suppressing AI research" with a ten-foot pole; depending on who they are and where they work, it's tantamount to advocating for treason. Literal treason, with some people. It's the kind of thing where merely associating with people who advocate for it can cost you your job, and certainly promotions.

A lot of this is considered infohazardous, and frankly, I've already said too much here. But it's very legitimate, and even sensible, to have very, very strong and unquestioned misgivings about large numbers of people they're associated with being persuaded to do something as radical as playing a zero-sum game against the entire AI industry.

This was counter to the prevailing narrative at the time, and I think did some of the work of changing the narrative. It's of historical significance, if nothing else.

I think it's a bit hard to tell how influential this post has been, though my best guess is "very". It's clear that sometime around when this post was published there was a pretty large shift in the strategies that I and a lot of other people pursued, with "slowing down AI" becoming a much more common goal for people to pursue.

I think (most of) the arguments in this post are good. I also think that when I read an initial draft of this post (around 1.5 years ago or so), and had a very hesitant reaction to the core strategy it proposes, that I was picking up on something important, and that I do also want to award Bayes points to that part of me given how things have been playing out so far. 

I do think that since I've seen people around me adopt strategies to slow down AI, I've seen it done on a basis that feels much more rhetorical, and often directly violates virtues and perspectives that I hold very dearly. I think it's really important to understand that technological progress has been the central driving force behind humanity's success, and that indeed this should establish a huge prior against stopping almost any kind of technological development.

In contrast to that, the majority of arguments that I've seen find traction for slowing down AI development are not distinguishable from arguments that apply to a much larger set of technologies which to me clearly do not pose a risk commensurable with the prior we should have against slowdown. Concerns about putting people out of jobs, destroying traditional models of romantic relationships, violating copyright law, spreading disinformation, all seem to me to be the kind of thing that if you buy it, you end up with an argument that proves too much and should end up opposed to a huge chunk of technological progress. 

And I can feel the pressure in myself for these things as well. I can see how it would be easy to at least locally gain traction at slowing down AI by allying myself with people who are concerned about these things, and don't care much about the existential risk angle. I think allying is probably even a good call, though it's hard to be in alliances without conflating your beliefs with the beliefs if your alliance.

I think this post did engage with these considerations pretty well, though I am still hesitant and uncertain about the degree to which we can slow down AI without sacrificing enough collective sanity and epistemics, and am a bit worried now that things are in the realm of public debate and advocacy it's too late to have a sane conversation about those sacrifices.

Overall, I think this post is among the most important posts of 2022. 

There are things that are robustly good in the world, and things that are good on highly specific inside-view models and terrible if those models are wrong. Slowing dangerous tech development seems like the former, whereas forwarding arms races for dangerous tech between world superpowers seems more like the latter.

It may seem the opposite to some people. For instance, my impression is that for many adjacent to the US government, "being ahead of China in every technology" would be widely considered robustly good, and nobody would question you at all if you said that was robustly good. Under this perspective the idea that AI could pose an existential risk is a "highly specific inside-view model" and it would be terrible if we acted on the model and it is wrong.

I don't think your readers will mostly think this, but I actually think a lot of people would, which for me makes this particular argument seem entirely subjective and thus suspect.

Thanks for writing!

I want to push back a bit on the framing used here. Instead of the framing "slowing down AI," another framing we could use is, "lay the groundwork for slowing down in the future, when extra time is most needed." I prefer this latter framing/emphasis because:

• An extra year in which the AI safety field has access to pretty advanced AI capabilities seems much more valuable for the field's progress (say, maybe 10x) than an extra year with current AI capabilities, since the former type of year would give the field much better opportunities to test safety ideas and more clarity about what types of AI systems are relevant.
  • One counterargument is that AI safety will likely be bottlenecked by serial time [? · GW], because discarding bad theories and formulating better ones takes serial time, making extra years early on very useful. But my very spotty understanding of the history of science suggests that it doesn't just take time for bad theories to get replaced by better ones--it takes time along with the accumulation of lots of empirical evidence. This supports the view that late-stage time is much more valuable than early-stage time.
• Slowing down in the future seems much more tractable than slowing down now, since many critical actors seem much more likely to support slowing down if and when there are clear, salient demonstrations of its importance (i.e. warning shots).
• Given that slowing down later is much more valuable and much more tractable than just slowing down now, it seems much better to focus on slowing down later. But the broader framing of "slow down" doesn't really suggest that focus, and maybe it even discourages it.

I’ve copied over and lightly edited some comments I left on a draft. Note I haven’t reread the post in detail; sorry if these were addressed somewhere.

Writing down quick thoughts after reading the intro and before reading the rest:

I have two major reasons to be skeptical of actively slowing down AI (setting aside feasibility):

1. It makes it easier for a future misaligned AI to take over by increasing overhangs, both via compute progress and algorithmic efficiency progress. (This is basically the same sort of argument as "Every 18 months, the minimum IQ necessary to destroy the world drops by one point.")

2. Such strategies are likely to disproportionately penalize safety-conscious actors.

(As a concrete example of (2), if you build public support, maybe the public calls for compute restrictions on AGI companies and this ends up binding the companies with AGI safety teams but not the various AI companies that are skeptical of “AGI” and “AI x-risk” and say they are just building powerful AI tools without calling it AGI.)

For me personally there's a third reason, which is that (to first approximation) I have a limited amount of resources and it seems better to spend that on the "use good alignment techniques" plan rather than the "try to not build AGI" plan. But that's specific to me.

After reading the doc I think I agree with most of the specific object-level points, and disagree with the overall thrust.

There seems to be some equivocation between two different viewpoints:

1. We can make a big difference by just not building AGI for a long time (say decades). Maybe this is hard / morally wrong, maybe not, we haven't actually tried, and we aren't even considering it. (To this view my objection would be the two points above; I'm confused why neither one appeared in the doc.)

2. There are simple straightforward things that seem clearly good, so it's wrong for people to say it's too hard / morally dubious. (To this view my objection is that people don't say that, people aren't opposed to it, and people are doing the simple straightforward things. Though perhaps we know different people.)

I feel like as I read this doc I was being pushed towards ambiguating between these two views, so that I come out with a belief like "Wow, there's all this straightforward obvious stuff, but there's a huge taboo against it and no one even considers it". Nowhere do you actually say that but it's very much the vibe I get.

All that being said, I strongly agree that there are simple straightforward things that seem clearly good and we should do them! I just think that people do consider them already. For example, on your list of 11 actions in the “Restraint is not terrorism, usually”, I think I have done or am doing (1), (2), (4), (5), (6), (7), (9), and (11).

(Written now, not previously) Overall my take is that most of the specific object-level arguments make sense and are valuable as a response to some ill-considered memes that get thrown around in this community, but it isn't a great take on the overall question of whether and how to slow down AI because it doesn't consider the best counterarguments. (Which I'll note that I wrote down before reading the full essay.)

The conversation near me over the years has felt a bit like this: 

Some people: AI might kill everyone. We should design a godlike super-AI of perfect goodness to prevent that.

Others: wow that sounds extremely ambitious

Some people: yeah but it’s very important and also we are extremely smart so idk it could work

[Work on it for a decade and a half]

 

Some people: ok that’s pretty hard, we give up

Others: oh huh shouldn’t we maybe try to stop the building of this dangerous AI? 

Some people: hmm, that would involve coordinating numerous people—we may be arrogant enough to think that we might build a god-machine that can take over the world and remake it as a paradise, but we aren’t delusional

There's a sleight of hand going on with the "we" here. "We" as in LessWrong are not building the godlike AI, the trillion dollar technocapitalist machine is doing that. "We" are a bunch of nerds off to the side, some of whom are researching ways to point said AIs at specific targets. If "we" tried to start an AGI company we'd indeed end up in fifth place and speed up timelines by six weeks (generously).

I think there's a major internal tension in the picture you present (though the tension is only there with further assumptions). You write:

Obstruction doesn’t need discernment

[...]
I don’t buy it. If all you want is to slow down a broad area of activity, my guess is that ignorant regulations do just fine at that every day (usually unintentionally). In particular, my impression is that if you mess up regulating things, a usual outcome is that many things are randomly slower than hoped. If you wanted to speed a specific thing up, that’s a very different story, and might require understanding the thing in question.

The same goes for social opposition. Nobody need understand the details of how genetic engineering works for its ascendancy to be seriously impaired by people not liking it. Maybe by their lights it still isn’t optimally undermined yet, but just not liking anything in the vicinity does go a long way.

And you write:

Technological choice is not luddism

Some technologies are better than others [citation not needed]. The best pro-technology visions should disproportionately involve awesome technologies and avoid shitty technologies, I claim. If you think AGI is highly likely to destroy the world, then it is the pinnacle of shittiness as a technology. Being opposed to having it into your techno-utopia is about as luddite as refusing to have radioactive toothpaste there. Colloquially, Luddites are against progress if it comes as technology.¹⁰ [LW · GW] Even if that’s a terrible position, its wise reversal is not the endorsement of all ‘technology’, regardless of whether it comes as progress.

From "obstruction doesn't need discernment", you're proposing a fairly broad dampening. While this may be worth it from an X-risk perspective, it's (1) far more Luddite, (2) far more potentially actually harmful (though we'd agree, still worth it), and therefore (3) far more objectionable, and (4) far more legitimately opposed. 

Curated.  I am broadly skeptical of existing "coordination"-flavored efforts, but this post prompted several thoughts:

1. I have very incomplete models of what those efforts are (much more so than my models of technical alignment efforts, which are also substantially incomplete)
2. It seems likely to me that some people have better models of both existing efforts, and potential future efforts, which have not been propagated very well, and it'd be good for that to change

I think this post does a good job of highlighting representative objections to various proposed strategies and then demonstrating why those objections should not be considered decisive (or even relevant).  It is true that we will not solve the problem of AI killing everyone by slowing it down, but that does not mean we should give up on trying to find +EV strategies for slowing it down, since a successful slowdown, all else equal, is good.

The world has a lot of experience slowing down technological progress already, no need to invent new ways. 

Fusion research slowed to a crawl in the 1970s and so we don't have fusion power. Electric cars have been delayed by a century. IRB is successful at preventing many promising avenues. The FDA/CDC killed most of the novel drug and pandemic prevention research. The space industry is only now catching up to where it was 50 years ago. 

Basically, stigma/cost/red tape reliably and provably does the trick. 

↑ comment by TekhneMakre · 2022-12-23T05:55:11.161Z · LW(p) · GW(p)

Fusion research slowed to a crawl in the 1970s and so we don't have fusion power

Requires huge specialized equipment. Some AI requires huge equipment, but (1) you can do a lot with a little, and (2) the equipment is heavily economically incentivized for other reasons (all the other uses of compute). 
 

 Electric cars have been delayed by a century.

Why was this? I'd've thought it's basically battery tech, blocked on materials tech. Is that not right?
 

  The space industry is only now catching up to where it was 50 years ago. 
 

Wasn't this basically un-investment? That's not a lever we can easily pull with AI. 
 

 IRB is successful at preventing many promising avenues. The FDA/CDC killed most of the novel drug and pandemic prevention research.

Now we're talking! How the heck does that work? I'm surprised enough that actually I'd guess it's not that hard for a private investor to do research, it's just that the research wouldn't be allowed to be applied (enforceable because very public). Is that true? If not, why not?

Replies from: shminux

↑ comment by Shmi (shminux) · 2022-12-23T06:37:41.089Z · LW(p) · GW(p)

With fusion it was mostly defunding, just like with space exploration:

from https://21sci-tech.com/Articles_2010/Winter_2009/Who_Killed_Fusion.pdf

Not sure if this is a possibility with AI. Electric cars and transport in general were apparently killed by the gas automobile industry. Battery tech was just enough for the daily commute, and there were options and workarounds.

I am not an expert on how the government regulation kills innovation, there is probably enough out there, including by Zvi and Scott Alexander.

It seems very odd to have a discussion of arms race dynamics that is purely theoretical exploration of possible payoff matrices, and does not include a historically informed discussion of what seems like the obviously most analogous case, namely nuclear weapons research during the Second World War.

US nuclear researchers famously (IIRC, pls correct me if wrong!) thought there was a nontrivial chance their research would lead to human extinction, not just because nuclear war might do so but because e.g. a nuclear test explosion might ignite the atmosphere. They forged ahead anyway on the theory that otherwise the Nazis were going to get there first, and if they got there first they would use that advantage to lock in Nazi hegemony, and that was so bad an outcome it was worth a significant risk of human extinction to avoid.

Was that the wrong thing for them to have done under the circumstances? If so, why, and what can we say confidently in hindsight that should they have done instead? If not, why is the present situation saliently different? If China gets to AGI first that plausibly locks in CCP hegemony which is arguably similarly bad to locking in Nazi hegemony. Trying to convince the CCP that they will just kill themselves too if they do this, so they shouldn't try, seems about as tractable as persuading Werner Heisenberg and his superiors during WWII that they shouldn't try to build nukes because they might ignite the atmosphere.

The world has actually coordinated on some impressive things, e.g. nuclear non-proliferation

Given the success of North Korea, I am both impressed by the world's coordination on nuclear weapon and depressed that even the impressive coordination is not enough. I feel similarly for the topic the nuclear weapon is metaphor for.

Continuing to think about the metaphor: in terms of existential risk, North Korean nuclear weapon is probably less damaging than United States missile defense, due to reasons related to second strike. It is probable people feel bad about North Korean nuclear weapon because it is normally bad (North Korea is bad so it is bad for North Korea to be powerful), not because it is an existential threat (it increases the probability of the world being destroyed).

Is this analogous to people being against AI because they are "biased", not because they are dangerous? Or are "we" missing some important crucial considerations like second strike in AI, leading us to worry about North Korean nuclear weapon when we really should worry about United States missile defense?

I suspect that part of what is going on is that many in the AI safety community are inexperienced with and uncomfortable with politics and have highly negative views about government capabilities.   

Another potential (and related) issue, is that people in the AI safety community think that their comparative advantage doesn't lie in political action (which is likely true) and therefore believe they are better off pursuing their comparative advantage (which is likely false).  

As a datapoint, I remember briefly talking with Eliezer in July 2021, where I said "If only we could make it really cringe to do capabilities/gain-of-function work..." (I don't remember which one I said). To which, I think he replied "That's not how human psychology works." 

I now disagree with this response. I think it's less "human psychology" and more "our current sociocultural environment around these specific areas of research." EG genetically engineering humans seems like a thing which, in some alternate branches, is considered "cool" and "exciting", while being cringe in our branch. It doesn't seem like a predestined fact of human psychology that that field had to end up being considered cringe. 

Your own examples of technologies that aren't currently pursued but have huge upsides are a strong case against this proposition. These lines of research have some risks, but if there was sufficient funding and coordination, they could be tremendously valuable. Yet the status quo is to simply ban them without investing much at all in building a safe infrastructure to pursue them.

If you should succeed in achieving the political will needed to "slow down tech," it will come from idiots, fundamentalists, people with useless jobs, etc. It will not be a coalition pushing to make aligned, democratic, open AI possible. It will be a coalition dedicated to preserving the status quo and rent-seeking. Perhaps you believe, SBF-style, that "means justify the ends," and this dogmatic coalition will still serve you well. Perhaps it does, and it reduces existential risk. (This is a big assumption.) Even so, most humans don't care about existential risk much. Humans value others by proximity to themselves, temporally and otherwise. This is asking everyone alive, their children, their grandchildren, etc. to suffer so that "existential risk" integrated over an infinite time interval is reduced. This is not something most people want, and it's the reason you will only find allies in the unproductive rent-seekers and idiots.

Someone used CRISPR on babies in China

Nitpick: the CRISPR was on the embryos, not the babies.

Although I don't think what He Jiankui did was scientifically justified (the benefit of HIV resistance wasn't worth the risk – he should have chosen PCSK9 instead of CCR5), I think the current norms against human genetic enhancement really are stifling a lot of progress. 

(1) The framing of all this as military technology (and the DoD is the single largest purchasing agent on earth) reminds me of nuclear power development. Molten Salt reactors and Pellet Bed reactors are both old tech which would have created the dream of safe, small-scale nuclear power. However, in addition to not melting down and working at relatively small scales, they also don't make weapons-usable materials. Thus they were shunted in favor of the kinds of reactors we mostly have now. In an alternative past without the cold war driving us to make new and better weapons, we hit infinite free energy for everyone back in 1986 and in 2022 we finally got our Dyson Sphere running.

So yeah, the sad thing about it being an arms race with China and DARPA et al that is the AGIs will look a certain way, and that might become """what AGI is""" for several decades. And the safeguards and controls that get put around those military-grade AGIs will prevent some other kind, the equivalent of Molten Salt reactors, from getting developed and built.

But we have to accept the world of incentives we have, not the one we wish for.

(2) As a PD defector/exploiter without much in the way of morals or shame, what I like about all this slowing down is that it gives smaller players a little opportunity to catch up and play with the big guys. I suspect at least a few smaller players (along with the open source community) would make some hay while the sun is shining and everyone else is moving slow and ensuring alignment, which is democratizing and cool. I put this up here as a selling point for those people who crave the pedal to the metal on AI dev. The slowness the OP is talking about allows a different pedal to different metal if you are looking for it, perhaps with your foot on it.

I arrogantly think I could write a broadly compelling and accessible case for AI risk

I recently asked [LW · GW] whether such a resource exists and didn't find one that meets what I'm looking for. Currently trying to write my own version.

Thank you for writing this post! I agree completely, which is perhaps unsurprising given my position [LW · GW] stated back in 2020. Essentially, I think we should apply the precautionary principle for existentially risky technologies: do not build unless safety is proven.

A few words on where that position has brought me since then.

First, I concluded back then that there was little support for this position in rationalist or EA circles. I concluded as you did, that this had mostly to do with what people wanted (subjective techno-futurist desires), and less with what was possible or the best way to reduce human extinction risk. So I went ahead and started the Existential Risk Observatory anyway, a nonprofit aiming to reduce human extinction risk by informing the public debate. We think public awareness is essentially the bottleneck for effective risk reduction, and we hope more awareness will lead to increased amounts of talent, funding, institutes, diversity, and robustness for AI Safety, and increased support for constructive regulation. This can be in the form of software, research, data, or hardware regulation, with each having their own advantages and disadvantages. Our intuition is that with 50% awareness, countries should be able to implement some combination of the above that would effectively reduce AI existential risk, while trying to keep economic damage to a minimum (an international treaty may be needed, or a US-China deal, or using supply chain leverage, or some smarter idea). To our knowledge, no-one has worked out a detailed regulation proposal for this (perhaps this comes kind of close). If true, we think that's embarrassing and regulation proposals should be worked out (and this work should be funded) with urgency. If there are regulation proposals which are not shared, we think people should share them and be less infohazardy about it.

So how did informing the societal debate go so far?

We started from a super crappy position: self-funded, hardly any connection to the xrisk space (that was also partially hostile to our concept), no media network to speak of, located in Amsterdam, far from everything. I had only some founding experience with a previous start-up. Still, I have to say that on balance, things went better than expected:

1. Setting up the organization went well. It was easy to attract talent through EA networks. My first lesson: even if some senior EAs and rationalists were not convinced about informing the societal debate, many juniors were.
2. We were successful in slowly working our way into the Dutch societal debate. One job opening led to another podcast led to another drink led to another op-ed, etc. It took a few months and lots of meetings with usually skeptical people, but we definitely made progress.
3. We published our first op-eds in leading Dutch newspapers after about six months. We are now publishing about one article per month, and have been in four podcasts as well. We have reached out to a few million people by readership, mostly in the Netherlands but also in the US.
4. We are now doing our first structured survey research measuring how effective our articles are. According to our first preliminary measurement data (report will be out in a few months), conversion rates for newspaper articles and youtube videos (the two interventions we measured so far) are actually fairly high (between ~25% and 65%). However, there aren't too many good articles on the topic out there yet relative to population sizes, so if you just crunch the numbers, it seems likely that most people still haven't heard of the topic. There's also a group that has heard the arguments but doesn't find them convincing. According to first measurements, this doesn't correlate too much with education level or field. Our data is therefore pointing away from the idea that only brilliant people can be convinced of AI xrisk.
5. We obtained funding from SFF and ICFG. Apparently, getting funding for projects aiming to raise AI xrisk awareness, despite skepticism of this approach by some, was already doable last year. We seem to observe a shift towards our approach, so we would expect this to become easier.
6. There's a direct connection between publishing articles and influencing policy. It wasn't our goal to directly influence policy, but co-authors, journalists, and others are automatically asking when you write an article: so what do you propose? One can naturally include regulation proposals (or proposals for e.g. more AI Safety funding) into articles. It is also much easier to get meetings with politicians and policy makers after publishing articles. Our PA person has had meetings with three parliamentarians (two of parties in government) in the last few weeks, so we are moderately optimistic that we can influence policy in the medium term.

We think that if we can do this, many more people can. Raising awareness is constrained by many things, but most of all by manpower. Although there are definitely qualities that makes you better at this job (xrisk expertise, motivation, intelligence, writing and communication skills, management skills, network), you don't need to be a super genius or have a very specific background to do communication. Many in the EA and rationalist communities who would love to do something about AI xrisk but aren't machine learning experts could work in this field. With only about 3 FTE, I'm positive our org can inform millions of people. Imagine what dozens, hundreds, or thousands of people working in this field could achieve.

If we would all agree that AI xrisk comms is a good idea, I think humanity would have a good chance of making it through this century.

Broadly agree, but: I think we're very confused about the social situation. (Again, I agree this argues to work on deconfusing, not to give up!) For example, one interpretation of the propositions claimed in this thread

https://twitter.com/soniajoseph_/status/1597735163936768000 

if  they are true, is that AI being dangerous is more powerful in terms of moving money and problem-solving juice as a recruitment tool rather than a dissuasion tool. I.e. in certain contexts, it's beneficial towards the goal of getting funding to include in your pitch "this technology might be extremely dangerous". 
 

I've tried to make a list here
https://www.lesswrong.com/posts/ZePZkkNriPqo9c2wd/why-do-some-people-try-to-make-agi [LW · GW]

to seed theorization about the situation, but didn't get much engagement. 

This post seems like it was quite influential. This is basically a trivial review to allow the post to be voted on.

I see one critical flaw here.

Why does anyone assume ANY progress will be made on alignment if we don't have potentially dangerous AGIs in existence to experiment with?

A second issue is that at least the current model for chatGPT REQUIRES human feedback to get smarter, and the greater the scale of userbase the smarter it can potentially become.

Other systems designed to scale to AGI may have to be trained this way: initial training from test environments and static human text, but refinement from interaction with live humans, where the company with the most users has runaway success because more users means the system learns faster and more users switch to it.

This dependency graph means we may have no choice but to proceed with AGI on the most expedient timescale. Once early systems attempt bad behavior and fail, then research those in large well funded isolated labs to discover ways to mitigate the issues.

How much progress did humans make on heavier than air flight before they had actual airplanes?

Nobody invented jet engines or computational fluid dynamics before they had many generations of aircraft and many air battles to justify the expense.

One big obstacle you didn't mention: you can make porn with that thing. It's too late to stop it.

More seriously, I think this cat may already be out of the bag. Even if the scientific community and the american military-industrial complex and the chinese military-industrial complex agreed to stop AI research, existing models and techniques are already widely available on the internet.

Even if there is no official AI lab anywhere doing AI research, you will still have internet communities pooling compute together for their own research projects (especially if crypto collapses and everybody suddenly has a lot of extra compute on their hands).

And these online communities are not going to be open-minded about AI safety concerns. We've seen that already with the release of Stable Diffusion 2.0: the internet went absolutely furious that the model was limited in (very limited ways) that impacted performance. People wanted their porn machine to be as good as it could possibly be and had no sympathy whatsoever for the developers' PR / safety / not-wanting-to-be-complicit-with-nonconsensual-porn-fakes concerns.

Of course, if we do get to the point only decentralized communities do AI research, it will be a pretty big win for the "slowing down" strategy. I get your general point about "we should really exhaust all available options even if we think it's nigh impossible". I just think you're underestimating a bit how nigh-impossible it is. We can barely stop people from using fossil fuels, and that's with an infinitely higher level of buy-in from decision-makers.

I'm not convinced this line of thinking works from the perspective of the structure of the international system. For example, not once are international security concerns mentioned in this post. 

My post here draws out some fundamental flaws in this thinking:
https://www.lesswrong.com/posts/dKFRinvMAHwvRmnzb/issues-with-uneven-ai-resource-distribution

I very strongly agree with this post. Thank you very much for writing it!

I think to reach a general agreement on not doing certain stupid things, we need to better understand and define what exactly those things are that we shouldn't do. For example, instead of talking about slowing down the development of AGI, which is a quite fuzzy term, we could talk about preventing uncontrollable AI [LW · GW]. Superintelligent self-improving AGI would very likely be uncontrollable, but there could be lesser forms of AI that could be uncontrollable, and thus very dangerous, as well. It should also be easier to reach agreement that uncontrollable AI is a bad thing. At least, I don't think that any leader of an AI lab would proudly announce to the public that they're trying to develop uncontrollable AI.

Of course, it isn't clear yet what exactly makes an AI uncontrollable. I would love to see more research in that direction.

Bravo! I especially agree wrt people giving too many galaxy brain takes for why current ML labs speeding along is good.

I believe this post is one of the best to grace the front page of the LW forum this year. It provides a reasoned counterargument to prevailing wisdom in an accessible way, and has the potential to significantly update views. I would strongly support pinning this post to the front page to increase critical engagement.

Thank you for calling attention to this. It always seems uncontroversial that some things could speed up AGI timelines, yet it is assumed that very little can be done to slow them down. The actual hard part is figuring out what could, in practice, slow down timelines with certainty.

Finding ways to slow down timelines is exactly why I wrote this post on Foresight for AGI Safety Strategy: Mitigating Risks and Identifying Golden Opportunities [LW · GW].

• AI is pretty safe: unaligned AGI has a mere 7% chance of causing doom, plus a further 7% chance of causing short term lock-in of something mediocre
• Your opponent risks bad lock-in: If there’s a ‘lock-in’ of something mediocre, your opponent has a 5% chance of locking in something actively terrible, whereas you’ll always pick good mediocre lock-in world (and mediocre lock-ins are either 5% as good as utopia, -5% as good)
• Your opponent risks messing up utopia: In the event of aligned AGI, you will reliably achieve the best outcome, whereas your opponent has a 5% chance of ending up in a ‘mediocre bad’ scenario then too.
• Safety investment obliterates your chance of getting to AGI first: moving from no safety at all to full safety means you go from a 50% chance of being first to a 0% chance
• Your opponent is racing: Your opponent is investing everything in capabilities and nothing in safety
• Safety work helps others at a steep discount:  your safety work contributes 50% to the other player’s safety 

This is more a personal note / call for somebody to examine my thinking processes, but I've been thinking really hard about putting hardware security methods to work. Specifically, spreading knowledge far and wide about how to:

1. allow hardware designers / manufacturers to have easy, total control over who uses their product for what for how much throughout the supply chain
2. make it easy to secure AI related data (including e.g. model weights and architecture) and difficult to steal.

This sounds like it would improve every aspect of the racey-environment conditions, except:

Your opponent is racing: Your opponent is investing everything in capabilities and nothing in safety

The exact effect of this is unclear. On the one hand, if racey, zero-sum thinking actors learn that you're trying to "restrict" or "control" AI hardware supply, they'll totally amp up their efforts. On the other hand, you've also given them one more thing to worry about (their hardware supply).

I would love to get some frames on how to think about this.

So, a number of issues stand out to me, some have been noted by others already, but:

My impression is that there are also less endorsable or less altruistic or more silly motives floating around for this attention allocation.

A lot of this list looks to me like the sort of heuristics where, societies that don't follow them inevitably crash, burn and become awful. A list of famous questions where the obvious answer is horribly wrong, and there's a long list of groups who came to the obvious conclusion and became awful, and it's become accepted wisdom to not do that, except among the perpetually stubborn "It'll be different this time" crowd, and doomers who insist "well, we just have to make it work this time, there's no alternative".

if anyone chooses to build, everything is destroyed

The problem with our current prisoner's dilemma is that China has already openly declared their intentions. You're playing against a defect bot. Also, your arguments are totally ineffective against them, because you're not writing in Chinese. And, the opposition is openly malicious, and if alignment turns out to be easy, this ends with hell on earth, which is much worse than the false worst case of universal annihilation.

On the inevitability of AI: I find current attempts at AI alignment to be spaceships with sliderules silliness and not serious. Longer AI timelines are only useful if you can do something with the extra time. You're missing necessary preconditions to both AI and alignment, and so long as those aren't met, neither field is going to make any progress at all.

On qualia: I expect intelligence to be more interesting in general than the opposition expects. There are many ways to maximize paperclips, and even if technically, one path is actually correct, it's almost impossible to produce sufficient pressure to direct a utility function directly at that. I expect an alien super intelligence that's a 99.9999% perfect paperclip optimizer, and plays fun games on the side, to play above 99% of the quantity of games that a fun game optimizer would get. I accuse the opposition of bigotry towards aliens, and assert that the range of utility functions that produce positive outcomes is much larger than the opposition believes. Also, excluding all AI that would eliminate humanity, excludes lots of likable AI that would live good lives, but reach the obviously correct conclusion that humans are worse than them and need to go, while failing exclude any malicious AI that values human suffering.

On anthropics: We don't actually experience the worlds that we fail to make interesting, so there's no point worrying about them anyway. The only thing that actually matters is the utility ratio. It is granted that, if this worldline looked particularly heaven-oriented, and not hellish, it would be reasonable to maximize the amount of qualia attention by being protective of local reality, but just looking around me, that seems obviously not true.

On Existential Risk: I hold that the opposition massively underestimates current existential risks excluding AI, most of which AI is the solution to. The current environment is already fragile. Any stable evil government anywhere means that anything that sets back civilization threatens stagnation or worse, aka, every serious threat, even those that don't immediately wipe out all life, most notably nuclear weapons, constitutes an existential risk. Propaganda and related can easily drive society into an irrecoverable position using current techniques. Genetics can easily wipe us out, and worse, in either direction. Become too fit, and we're the ones maximizing paperclips. Alternatively, there's the grow giant antlers and die problem where species trap themselves in a dysgenic spiral. Evolution does not have to be slow, and especially if social factors accelerate the divide between losers and winners, we could easily breed ourselves to oblivion in a few generations. Almost any technology could get us all killed. Super pathogens with a spread phase and a kill phases. Space technology that slightly adjusts the pathing of large objects. Very big explosions. Cheap stealth, guns that fire accurately across massive distances, fast transportation, easy ways to produce various poison gasses. There seems to be this idea that just because it isn't exotic it won't kill you.

In sum: I fully expect that this plan reduces the chances of long term survival of life, while also massively increasing the probability of artificial hell.

I could be wrong, but I’d guess convincing the ten most relevant leaders of AI labs that this is a massive deal, worth prioritizing, actually gets you a decent slow-down. I don’t have much evidence for this.

…

Delay is probably finite by default 

Convincing OpenAI, Anthropic, and Google seems moderately hard - they've all clearly already considered the matter carefully, and it's fairly apparent from their actions what conclusions they've reached about pausing now. Then convincing Meta, Mistral, Cohere, and the next dozen-or-so would-be OpenAI replacements seems harder. Then, also doing so for the Chinese and UAE equivalents and their governments seems like it might be harder still, not so much because talking to their AI researchers is hard, but because you also have to persuade members of their governments, which are top-down power structures headed by people used to getting their way. Then you need to convince A19z, EleutherAI, Hugging Face and the entire open-source AI community — that seems harder still, mostly because there are so many of them. After that we're looking at people like the Russians and the North Koreans, then the academic AI research community, and after that probably things like criminal organizations and warlords.

Somewhere on that sequence, you're going to find a nut you can't crack. Having multiple major governments on your side would help a lot, they might well be able to achieve everything before North Korea and the Russians. At current speed of improvements in Moore's Law for GPUs and algorithmic improvements and monetary investments in AI, each step on that ladder probably buys you something like 6 months of delay (this is a rough estimate, I'm sure there are people in the AI governance community who could give better ones.) If you got several steps down, you might even be able to somewhat slow the growth of training compute and algorithmic improvements, so that might stretch to more like 12 months per rung. So with massive effort backed by major governments, we might be able to slow things down by up to about 4 years. If we're all going to die at the end, that's ~4 years of extra life for ~8 billion people, clearly a huge positive achievement. If those ~4 years would make a large improvement to Alignment fieldbuilding and progress, that could make an even bigger difference. But the downside is that if someone builds AGI in secret during those ~4 years, it's going to be a criminal organization or a failed state, and that near the end of it, you're now in a race with people like the North Koreans. We'd also have built up some overhang: on my progress-slowing assumption, only about 2 years' worth at current rates, since the world would actually be behind the unslowed track, but still, some overhang. Overhang presumably leads to a potential for rapid growth, which is extremely bad for safety.

The obvious next question is, how far away is AGI? My personal current bests guess, off rough parallels to human synapse counts, is about 2-3 orders of magnitude in parameter counts (which is ~2-3 GPT generations, so ~GPT-6 or GPT-7, which sounds about right based on generational progress so far). That's ~4-6 orders of magnitude in total training compute cost before any algorithmic and Moore's Law discounts. At the GPT-1 through 4 progress rate, that's around 2.5-4 years. Clearly it's hard to be sure, this could be only one GPT-generation away (about a year), or four (about 5.5 years). So an additional ~4 year delay is quite a significant amount of extra time, probably more than doubling, but not a huge difference.

The next question is, if we're going to delay, when? Coordinating a delay is probably going to be easiest when the danger is near and most tangible and demonstrable, and as the OP notes the gains for an extra up-to-4 years of Alignment research will be biggest with near-AGI systems to work with and more Alignment fieldbuilding done. There will obviously also be a lot more political capital for a pause once the significant white-collar job losses start. So there would seem to be reasons to maintain a significant lead until you pause, and then pause for as long as you can get away with (while closely monitoring anyone still racing).

The recent blitz to get governments involved and set up regulatory frameworks could be interpreted as laying the groundwork for a future government-led pause (and for starting to slow smaller players) without actually initiating one yet. So it looks like the intention is to build models around the GPT-5 level, do some safety analysis, and then see what they look like. At that point, if they were showing really scary capabilities ("High" across the board on OpenAI's Preparednesss framework, for example), then that would put the labs in a good position to go back to the governments and yell "Crisis! Pull the emergency brake!" They would also probably by then have a much better idea if they already have AGI on their hands, or it's still 1-3 generations away.

"I arrogantly think I could write a broadly compelling and accessible case for AI risk"

Please do so. Your current essay is very good, so chances are your "arrogant" thought is correct.

Edit: I think this is too pessimistic about human nature, but maybe we should think about this more before publishing a "broadly compelling and accessible case for AI risk".

https://www.lesswrong.com/posts/xAzKefLsYdFa4SErg/accurate-models-of-ai-risk-are-hyperexistential-exfohazards [LW · GW]

Thank you for writing this. On your section 'Obstruction doesn't need discernment' - see also this post that went up on LW a while back called The Regulatory Option: A response to near 0% survival odds [LW · GW]. I thought it was an excellent post, and it didn't get anywhere near the attention it deserved, in my view.

People have been writing stories about the dangers of artificial intelligences arguably since Ancient Greek time (Hephaistos built artificial people, including Pandora), certainly since Frankenstein. There are dozens of SF movies on the theme (and in the Hollywood ones, the hero always wins, of course). Artificial intelligence trying to take over the world isn't a new idea, by scriptwriter standard it's a tired trope. Getting AI as tightly controlled as nuclear power or genetic engineering would not, politically, be that hard -- it might take a decade or two of concerted action , but it's not impossible. Especially if not-yet-general AI is also taking people's jobs. The thing is, humans (and especially politicians) mostly worry about problems that could kill them in the next O(5) years. Relatively few people in AI/universities/boardrooms/government/on the streets think we're O(5) years from GAI, and after more of them have talked to ChatGPT/etc for a while, they're going to notice the distinctly sub-human-level mistakes it makes, and eventually internalize that a lot of its human-level-appearing abilities are just pattern-extrapolated wisdom of crowds learned from most of the Internet.

So I think the questions are:

1. Is slowing down progress on GAI actually likely to be helpful, beyond the obvious billions of person-years per year gained from delaying doom? (Personally, I'm having difficulty thinking of a hard technical problem where having more time to solve it doesn't help.)
2. If so, when should we slow down progress towards GAI? Too late is disastrous, too soon risks people deciding you're crying wolf, either when you try it so you fail (and make it harder to slow down later), or else a decade or two after you succeed and progress gets sped up again (as I think is starting to happen with genetic engineering). This depends a lot on how soon you think GAI might happen, and what level of below-general AI would most enhance your doing alignment research on it. (FWIW, my personal feeling is that until recently we didn't have any AI complex enough for alignment research on it to be interesting/informative, and the likely answer is "just before any treacherous turn is going to happen" -- which is a nasty gambling dilemma. I also personally think GAI is still some number of decades away, and the most useful time to go slowly is somewhere around the "smart as a mouse/chimp/just sub-human level" -- close enough to human that you're not having to extrapolate a long way what you learn from doing alignment research on it up to mildly-superhuman levels.)
3. Whatever you think the answer to 2. is, you need  to start the political process a decade or two earlier: social change takes time.

I'm guessing a lot of the reluctance in the AI community is coming from "I'm not the right sort of person to run a political movement". In which case, go find someone who is, and explain to them that this is an extremely hard technical problem, humanity is doomed if we get it wrong, and we only get one try.

(From a personal point of view, I'm actually more worried about poorly-aligned AI than non-aligned AI. Everyone beng dead and having the solar system converted into paperclips would suck, but at least it's probably fairly quick. Partially aligned AI that keeps us around but doesn't understand how to treat us could make Orwell's old quote about a boot stamping on a human face forever look mild – and yes, I'm on the edge of Godwin's Law.)

Very useful post and discussion! Let's ignore the issue that someone in capabilities research might be underestimating the risk and assume they have appropriately assessed the risk. Let's also simplify to two outcomes of bliss expanding in our lightcone and extinction (no value). Let's also assume that very low values of risk are possible but we have to wait a long time. It would be very interesting to me to hear how different people (maybe with a poll) would want the probability of extinction to be below before activating the AGI. Below are my super rough guesses:

1x10^-10: strong longtermist

1x10^-5: weak longtermist

1x10^-2 = 1%: average person (values a few centuries?)

1x10^-1 = 10%: person affecting: currently alive people will get to live indefinitely if successful

30%: selfish researcher

90%: fame/power loving older selfish researcher

I was surprised that my estimate was not more different for a selfish person. With climate change, if an altruistic person affecting individual thinks the carbon tax should be $100 per ton carbon, a selfish person should act as if the carbon tax is about 10 billion times lower, so ten orders of magnitude different versus ~one order for AGI. So it appears that AGI is a different case in that the risk is more internalized to the actors. Most of the variance for AGI appears to be from how longtermist one is vs whether one is selfish or altruistic.

Chiming in on toy models of research incentives:  Seems to me like a key feature is that you start with an Arms Race then, after some amount of capabilities accumulate, transitions to the Suicide Race.  But players have only vague estimates of where that threshold is, have widely varying estimates, and may not be able to communicate estimates effectively or persuasively.  Players have a strong incentive to push right up to the line where things get obviously (to them) dangerous, and with enough players, somebody's estimate is going to be wrong.

Working off a model like that, we'd much rather be playing the version where players can effectively share estimates and converge on a view of what level of capabilities makes things get very dangerous.  Lack of constructive conversations with the largest players on that topic do sound like a current bottleneck. 

It's unclear to me to what extent there's even a universal clear distinction understood between mundane weak AI systems with ordinary kinds of risks and superhuman AGI systems with exotic risks that software and business people aren't used to thinking about outside of sci-fi.  That strikes me as a key inferential leap that may be getting glossed over.  

There's quite a lot of effort spent in technology training people that systems are mostly static absent human intervention or well defined automations that some person ultimately wrote, anything else being a fault that gets fixed.  Computers don't have a mind of their own, troubleshoot instead of anthropomorphizing, etc., etc.  That this intuition will at some point stop working or being true of a sufficiently capable system (and that this is fundamentally part of what we mean by human level AGI) is something that probably needs to be focused on more as it's explicitly contrary to the basic induction that's part of usefully working in/on computers.

I'd like to ask a few questions about slowing down AGI as they may turn out to be cruxes for me.

1. How popular/unpopular is AI slowdown? Ideally, we'd get AI slowdown/AI progress/Neutral as choices in a poll. I also ideally would like different framings of the problem, to test how well frames affect people's choices. But I do want at least a poll on how popular/unpopular AI slowdown is.

2. How much does the government want AI to be slowed down? Is Trevor's story about the US government not willing to countenance AI slowdown correct, and instead speed it up the norm in interacting with the government?

3. How much will AI companies lobby against AI slowdown? Because if this is a repeat of the fossil fuel situation where AI is viewed by the public as extremely good, I probably would not support too much object work in AI governance, and instead go meta. In other words, I'd be doing more meta things. But if AI companies support AI slowdown or at least not oppose it, than things could be okay, depending on the answers to 1 and 2.

I am interested in getting feedback on whether it seems worthwhile to advocate for better governance mechanisms (like prediction markets) in the hopes that this might help civilization build common knowledge about AI risk more quickly, or might help civilization do a more "adequate" job of slowing AI progress by, restricting unauthorized access to compute resources. Is this a good cause for me to work on, or is it too indirect and it would be better to try and convince people about AI risk directly? See a more detailed comment here: https://www.lesswrong.com/posts/PABtHv8X28jJdxrD6/racing-through-a-minefield-the-ai-deployment-problem?commentId=ufXuR5xtMGeo5bjon [LW(p) · GW(p)]

But empirically, the world doesn’t pursue every technology—it barely pursues any technologies.

While it's true that not all technologies are pursued, history shows that humans have actively sought out transformative technologies, such as fire, swords, longbows, and nuclear energy. The driving factors have often been novelty and utility. I expect the same will happen with AI. 

Very nice exploration of this idea. But I don't see mention, in the main post, and couldn't find by simple search, in the comments, any mention of my biggest concern about this notion of an AI development slowdown (or complete halt, as EY recently called for in Time):

Years of delay would put a significant kink in the smooth exponential progress charts of e.g. Ray Kurzweil. Which is a big deal, in my opinion, because:

(1) They're not recognised as laws of nature. But these trends, across so many technologies and the entire history of complexity, in the universe, have been so consistent, that diverting them, at will, feels like commanding the tide to not come in. It's probably less avoidable than we realise.

(2) If such a measure were somehow enacted, why wouldn't one expect civilizational collapse? Why should current society necessarily be able to be statically stable? All stagnant empires have historically collapsed. And the contemporary world is as tightly linked as one, with developments moving faster than ever, at bigger scale than ever. I feel sure there are a growing wake of known and unknown issues close behind, ready to wash over us all, if we come to a sudden slowdown.

Also, AI is not a single application to be suppressed, like any of the examples mentioned. Those, I'd liken to mushrooms, sprouting from the hidden fungal network of technological capabilities hidden out of sight. Whereas AI has already become a pervasive research tool underpinning so many continued advances.

While AGI looks set to jump in to extend the exponential upwards trend of ever more brains, now human population is plateauing. Preventing these being 'born' would, in my view, be equivalent to insanely harsh population controls. Or, a relative culling, compared to what's needed to continue on trend.

[Edit: I'd appreciate explicit critique of these thoughts, please.]

So, a couple general thoughts. 
-In the 80s/90s we were obsessing over potential atom-splitting devastation and the resulting nuclear winter. Before that there were plagues wiping out huge swaths of humanity. 
-Once a technology genie is out of the bottle you can't stop it. Human nature. -In spite of the 'miracles' of science there's A LOT humanity does not know. "There are know unknowns and unknown unknowns" D.R. 
-In a few billion years our sun will extinguish and all remaining life on space rock Earth will perish. All material concerns are temporary. Sorry. 
-Four thousand years of good advice suggests that the most interesting & meaningful stage of life starts when we begin to Know Thyself. Now in mid-life I'm starting to suspect there is some truth to this. Happy holidays everyone!

I completely agree, and I've said similar stuff to people. Let's just..pause?

From a longtermist viewpoint, even the tactic of terrorism to prevent X-Risk cannot and should not be ruled out. If the future of humanity is at stake, any laws and deontological rules can be overruled.