Where's the foom?

Scientific-style risk assessments should be positioned in the frame of some science. For all the examples that you gave (atomic bomb, vacuum decay, etc.) that was obviously physics.

To formally assess and quantify the risks of misaligned AI, the calculations should be done within a frame of the science of intelligence (i.e., "abstract" cognitive science, not tied to the specifics of the human brain) and agency.

No such general science of intelligence exists at the moment. There are frameworks that aspire to describe intelligence, agency, and sentience in general, such as Active Inference (Friston et al., 2022), thermodynamic ML (Boyd et al., 2022), MCR^2 (Ma et al., 2022), and, perhaps, Vanchurin's "neural network toolbox for describing the world and agents within it". Of these, I'm somewhat familiar only with Active Inference and Vanchun's work, and it seems to me that both of them are too general frameworks to be the basis of concrete risk calculations. I'm not familiar with thermodynamic ML work and MCR^2 work at all (a recurrent note: if would be very useful if the alignment community had people who are at least deeply familiar with and conversant in these frameworks).

Then, there are more concrete engineering proposals for A(G)I architectures, such as Transformer-based LLMs, RL agents of specific architectures, or LeCun's H-JEPA architecture [LW · GW]. Empirical and more general theoretical papers have been published about the risks of goal misgeneralisation for RL agents (I believe these works are referenced in Ngo et al.'s "The Alignment Problem from a Deep Learning Perspective"). I hope at least in part due to this (although he doesn't explicitly acknowledge this, as far as I know), LeCun is sceptical about RL and advocates for minimising RL from intelligence architectures.

Theoretical analysis of the risks of the H-JEPA architecture hasn't been done yet (let alone empirical, because H-JEPA agents haven't been constructed yet), however, H-JEPA belongs to a wider class of intelligence architectures that employ energy-based modelling (EBM) of the world. EBM agents have been constructed and could be analysed empirically already for their alignment properties. It's worth noting that energy-based modelling is conceptually rather similar to Active Inference, and Active Inference agents have been constructed, too (Fountas et al., 2020), and thus could be probed empirically for their alignment properties as well.

LLMs are also tested empirically for alignment (e.g., OpenAI's evals, the Machiavelli benchmark). Theoretical analysis of alignment in future LLMs is probably bottlenecked by better theories of transformers and auto-regressive LLMs in general, such as Roberts, Yaida, and Hanin’s deep learning theory (2021), Anthropic’s mathematical framework for transformers (2021), Edelman et al.'s theory of "sparse variable creation" in transformers (2022), Marciano’s theory of DNNs as a semi-classical limit of topological quantum NNs (2022), Bahri et al.’s review of statistical mechanics of deep learning (2022), and other general theories of mechanistic interpretability and representation in DNNs (Räukur et al. 2022). See here [LW · GW] for a more detailed discussion.

The fundamental limitation of analysing the risks of any specific architecture is that once AGI starts to rewrite itself, it could probably easily depart from its original architecture. Then then the question is whether we can organizationally, institutionally, and economically prevent AGIs from taking control of their own self-improvement. This question doesn't have a scientific answer and it's impossible to rigorously put a probability on this event. I would only say that if AGI is available in open-source, I think the probability of this will be approximately 100%: surely, some open-source hacker (or, more likely, dozens of them) will task AGI with improving itself, just out of curiosity or due to other motivations.

Likewise, even if AGI is developed strictly within tightly controlled and regulated labs, it's probably impossible to rigorously assess the probability that the labs will drift into failure (e.g., will accidentally release an unaligned version of AGI through AI), or will "leak" the AGI code or parameter weights. I suspect the majority of AI x-risk in the assessments of MIRI people and others is not even purely technical ("as if we had a perfect plan and executed it perfectly"), but from these kinds of execution, organisation resilience, theft, governmental or military coercion, and other forms of not strictly technical and therefore not formally assessable risk.

AGI Safety from first principles [? · GW] doesn't meet all of Tyler's requirements and desiderata, but I think it's a good introduction for a technical but skeptical / uninitiated audience. 

... in a leading science journal, refereed of course.

Being published and upvoted / endorsed by peers on the Alignment Forum should arguably count as this. Highly-upvoted content on the Alignment Forum is often higher in quality than papers published in even the best traditional academic journals. 

I think the question of where and how to conduct science and review is a separate question from the question of why to care about AI risk though. I am happy to read content published in any venue, though I might be hesitant to spend much time diving into any particular piece, unless it has been endorsed by someone I trust on Twitter, LW, AF, etc. Publication in an academic journal is just a different kind of endorsement, one that, in my experience, is a weaker and less reliable indicator of quality than the others.

This definitely sounds like something we'd need, yes. Just bringing this up that if anyone's interested in this sort of work, I'm game to help! My expertise is in computational physics and software engineering so I would be more help with e.g. modelling or even working with basic ML toy systems. PM for more details if you have any ideas that could use this kind of expertise.

This is a useful video because it is about the limits and possibilities of explanation (because it is an expert on ZKPs), while also (because of the format of the entire youtube series) nicely handling the issue that different audiences need different "levels of explanation":

I consider some version of "foom" to just be obviously true to all experts who care to study it for long enough, and learn the needed background concepts. 

Things like instrumental convergence, and the orthogonality thesis are relevant here, but also probably some security mindset.

Like part of my concern arises from the fact that I'm aware of lots of critical systems for human happiness that are held together with bailing wire, and full of zero-days, and the only reason they haven't been knocked over yet is that no one smart is stupid-or-evil enough to choose to knock them over and then robustly to act on that choice.

At the same time, I don't believe that it is wise to spell out all the zero-days in the critical infrastructure of human civilization.

There is an ongoing debate within the infosec community about proper norms for "responsible disclosure" where "what does 'responsible' even mean?" is somewhat contested.

Like from the perspective of many profit-seeking system maintainers who want to build insecure systems and make money from deploying them, they would rather there just be no work at all by anyone to hack any systems, and maybe all such work is irresponsible, and if the work produces exploits them maybe those should just be censored, and if those people want to publish then instead they should give the exploits, for zero pay, to the people whose systems were buggy.

But then security researchers get no pay, so that's probably unfair as a compensation model. 

What happens in practice is (arguably) that some organizations create "bug bounty systems"... and then all the other organizations that don't have bug bounty systems get hacked over and over and over and over?

But like... if you can solve all the problems for the publication of a relatively simple and seemingly-low-stakes version of "exploits that can root a linux kernel" then by all means, please explain how that should work and let's start doing that kind of thing all over the rest of society too (including for exploits that become possible with AGI).

A key issue here, arguably, is that if a security measure needs democratic buy-in and the median voter can't understand "a safe version of the explanation to spread around in public" then... there's gonna be a bad time.

Options seem logically to include:

1) Just publish lots and lots and lots of concrete vivid workable recipes that could be used by any random idiot to burn the world down from inside any country on the planet using tools available at the local computer shop and free software they can download off the internet.... and keep doing this over and over... until the GLOBAL median voter sees why there should be some GLOBAL government coordination on GLOBAL fire-suppression systems, and then there is a vote, and then the vote is counted, and then a wise ruler is selected as the delegate of the median voter... and then that ruler tries to do some coherent thing that would prevent literally all the published "world burning plans" from succeeding.

2) Give up on legible democracy and/or distinct nation-states and/or many other cherished institutions.

3) Lose.

4) Something creative! <3

Personally, I'm still working off of "option 4" for now, which is something like "encourage global elite culture to courageously embrace their duties and just do the right thing whether or not they will keep office after having done the right thing".

However, as option 3 rises higher and higher in my probability estimate I'm going to start looking at options 1 and 2 pretty closely.

Also, it is reasonable clear to me that option 2 is a shorter abstract super category that includes option 1, because option 1 also sounds like "not how many cherished institutions currently work"!

So probably what I'll do is look at option 2, and examine lots of different constraints that might be relaxed, and lots of consequences, and try to pick the least scary "cherished institutional constraints" to give up on, and this will probably NOT include giving up on my norm against publishing lots and logs of world-burning recipes because "maybe that will convince the fools!"

Publishing lots of world-burning recipes still seems quite foolish to me, in comparison to something normal and reasonable.

Like I personally think it would be wiser to swiftly bootstrap a new "United People's House of Representative Parliament" that treats the existing "United Nation-State Delegates" roughly as if it were a global "Senate" or global "House of Lords" or some such?

Then the "United People" could write legislation using their own internal processes and offer the "United Nations" the right to veto the laws in a single up/down vote?

And I think this would be LESS dangerous than just publishing all the world-burning plans? (But also this is a way to solve ONE of several distinct barriers to reaching a Win Condition (the part of the problem where global coordination institutions don't actually exist at all) and so this is also probably an inadequate plan all by itself.)

Category 2 is a big category. It is full of plans that would cause lots and lots of people to complain about how the proposals are "too weird", or sacrifice too many of their "cherished institutions", or many other complaints.

In summary: this proposals feels like you're personally asking to be "convinced in public using means that third parties can watch, so that third parties will grant that it isn't your personal fault for believing something at variance with the herd's beliefs" and not like your honest private assessment of the real situation is bleak. These are different things. 

My private assessment is that if I talk for anyone for several days face-to-face about all of this, and they even understand what Aumann Agreement is (such as to be able to notice in a verbal formal way when their public performance of "belief" is violating core standards of a minimally tolerable epistemology) that seems like a high cost to pay... but one I would gladly pay three or four or five times if me paying those costs was enough to save the world. But in practice, I would have to put a halt to any pretense of having a day job for maybe 6 weeks to do that, and that hit to my family budget is not wise.

Also: I believe that no humans exist who offer that degree of leverage over the benevolent competent rulership of the world, and so it is not worth it, yet, to me, personally, to pay such costs-of-persuasion with... anyone?

Also: the plausible candidates wouldn't listen to me, because they are currently too busy with things they (wrongly, in my opinion) think are more important to work on than AI stuff.

Also: if you could get the list of people, and plan the conversations, there are surely better people than me to be in those conversations, because the leaders will make decisions based off of lots of non-verbal factors, and I am neither male, nor tall, nor <other useful properties> and so normal-default-human leaders will not listen to me very efficiently.

gwern's clippy story does a reasonable amount of the work, i think. to me, part of why convincing people of AI doom is something that often needs to be done interactively rather than having a comprehensive explanation, is that people get stuck in a variety of different ways rather than always in the same way. that and numerous cognitive biases and stuff.

to me the argument is pretty straightforward:

• recursive self-improvement has a lot of room to get to superintelligence, so the capabilities can get there
• agentic AI totally makes sense
• an intelligent agentic AI will follow the instrumental convergence goals and win
• narrow values means it winning doesn't entail us surviving

i have a lot more ideas about how foom will happen but alas sharing them would be a capability exfohazard.

There are a lot of places which somewhat argue for FOOM -- i.e., very fast intelligence growth in the future, probably not preceded by smooth growth -- but they tend to be deeply out of date ( Yud-Hanson Debate and Intelligence Explosion Microeconomics ) or really cursory (Yud's pararaph in List of Lethalities [LW · GW] ) or a dialogue between two people being confused at each other (Christiano / Yud Discussion [? · GW] ).

I think the last one is probably the best as overview, but none provide like a great overview. Here's Christiano's blog on the topic which was written in 2018, so if its predictions hold up then its evidence for it. (But it is very much not in favor of FOOM... although you really have to read it to see what that actually means.)

The paper should have a formal model or calibration of some sort, working toward the conclusion of showing that the relevant risk is actually fairly high.

I doubt anyone has created this model yet, let alone published it in a journal. I guess that should be a first step 🤔. It seems very hard, since AGI would touch upon every domain.

The closest thing I can think of are the economic models based on hyperbolic growth. They predict a singularity, but don't specify what will cause it.