Late 2021 MIRI Conversations: AMA / Discussion

I feel like I have a broad distribution over worlds and usually answer questions with probability distributions, that I have a complete mental universe (which feels to me like it outputs answers to a much broader set of questions than Eliezer's, albeit probabilistic ones, rather than bailing with "the future is hard to predict").  At a high level I don't think "mainline" is a great concept for describing probability distributions over the future except in certain exceptional cases (though I may not understand what "mainline" means), and that neat stories that fit everything usually don't work well (unless, or often even if, generated in hindsight).

In answer to your "why is this," I think it's a combination of moderate differences in functioning and large differences in communication style. I think Eliezer has a way of thinking about the future that is quite different from mine and I'm somewhat skeptical of and feel like Eliezer is overselling (which is what got me into this discussion), but that's probably smaller than a large difference in communication style (driven partly by different skills, different aesthetics, and different ideas about what kinds of standards discourse should aspire to).

I think I may not understand well the basic lesson / broader point, so will probably be more helpful on object level points and will mostly go answer those in the time I have.

EDIT: I wrote this before seeing Paul's response; hence a significant amount of repetition.

They often seem to emit sentences that are 'not absurd', instead of 'on their mainline', because they're mostly trying to generate sentences that pass some shallow checks instead of 'coming from their complete mental universe.'

Why is this?

Well, there are many boring cases that are explained by pedagogy / argument structure. When I say things like "in the limit of infinite oversight capacity, we could just understand everything about the AI system and reengineer it to be safe", I'm obviously not claiming that this is a realistic thing that I expect to happen, so it's not coming from my "complete mental universe"; I'm just using this as an intuition pump for the listener to establish that a sufficiently powerful oversight process would solve AI alignment.

That being said, I think there is a more interesting difference here, but that your description of it is inaccurate (at least for me).

From my perspective I am implicitly representing a probability distribution over possible futures in my head. When I say "maybe X happens", or "X is not absurd", I'm saying that my probability distribution assigns non-trivial probability to futures in which X happens. Notably, this is absolutely "coming from my complete mental universe" -- the probability distribution is all there is, there's no extra constraints that take 5% probabilities and drive them down to 0, or whatever else you might imagine would give you a "mainline".

As I understand it, when you "talk about the mainline", you're supposed to have some low-entropy (i.e. confident) view on how the future goes, such that you can answer very different questions X, Y and Z about that particular future, that are all correlated with each other, and all get (say) > 50% probability. (Idk, as I write this down, it seems so obviously a bad way to reason that I feel like I must not be understanding it correctly.)

But to the extent this is right, I'm actually quite confused why anyone thinks "talk about the mainline" is an ideal to which to aspire. What makes you expect that? It's certainly not privileged based on what we know about idealized rationality; idealized rationality tells you to keep a list of hypotheses that you perform Bayesian updating over. In that setting "talk about the mainline" sounds like "keep just one hypothesis and talk about what it says"; this is not going to give you good results [LW · GW]. Maybe more charitably it's "one hypothesis is going to stably get >50% probability and so you should think about that hypothesis a lot" but I don't see why that should be true.

Obviously some things do in fact get > 90% probability; if you ask me questions like "what's the probability that if it rains the sidewalk will be wet" I will totally have a mainline, and there will be edge cases like "what if the rain stopped at the boundary between the sidewalk and the road" but those will be mostly irrelevant. The thing that I am confused by is the notion that you should always have a mainline, especially about something as complicated and uncertain as the future.

I presume that there is an underlying unvoiced argument that goes "Rohin, you say that you have a probability distribution over futures; that implies that you have many, many different consistent worlds in mind, and you are uncertain about which one we're in, and when you are asked for the probability of X then you sum probabilities across each of the worlds where X holds. This seems wild; it's such a ridiculously complicated operation for a puny human brain to implement; there's no way you're doing this. You're probably just implementing some simpler heuristic where you look at some simple surface desideratum and go 'idk, 30%' out of modesty."

Obviously I do not literally perform the operation described above, like any bounded agent I have to approximate the ideal. But I do not then give up and say "okay, I'll just think about a single consistent world and drop the rest of the distribution", I do my best to represent the full range of uncertainty, attempting to have all of my probabilities on events ground out in specific worlds that I think are plausible, think about some specific worlds in greater detail to see what sorts of correlations arise between different important phenomena, carry out some consistency checks on the probabilities I assign to events to notice cases where I'm clearly making mistakes, etc. I don't see why "have a mainline" is obviously a better response to our boundedness than the approach I use (if anything, it seems obviously a worse response).

The most recent post [? · GW] has a related exchange between Eliezer and Rohin:

Eliezer: I think the critical insight - though it has a format that basically nobody except me ever visibly invokes in those terms, and I worry maybe it can only be taught by a kind of life experience that's very hard to obtain - is the realization that any consistent reasonable story about underlying mechanisms will give you less optimistic forecasts than the ones you get by freely combining surface desiderata

Rohin: Yeah, I think I do not in fact understand why that is true for any consistent reasonable story.

If I'm being locally nitpicky, I argue that Eliezer's thing is a very mild overstatement (it should be "≤" instead of "<") but given that we're talking about forecasts, we're talking about uncertainty, and so we should expect "less" optimism instead of just "not more" optimism, and so I think Eliezer's statement stands as a general principle about engineering design.

This also feels to me like the sort of thing that I somehow want to direct attention towards. Either this principle is right and relevant (and it would be good for the field if all the AI safety thinkers held it!), or there's some deep confusion of mine that I'd like cleared up.

When I read other people, I often feel like they're operating in a 'narrower segment of their model', or not trying to fit the whole world at once, or something. They often seem to emit sentences that are 'not absurd', instead of 'on their mainline', because they're mostly trying to generate sentences that pass some shallow checks instead of 'coming from their complete mental universe.'

To me it seems like this is what you should expect other people to look like both when other people know less about a domain than you do, and also when you're overconfident about your understanding of that domain. So I don't think it helps distinguish those two cases.

(Also, to me it seems like a similar thing happens, but with the positions reversed, when Paul and Eliezer try to forecast concrete progress in ML over the next decade. Does that seem right to you?)

when Eliezer responded with:

But there's a really really basic lesson here about the different style of "sentences found in political history books" rather than "sentences produced by people imagining ways future politics could handle an issue successfully".

the subject got changed.

I believe this was discussed further at some point - I argued that Eliezer-style political history books also exclude statements like "and then we survived the cold war" or "most countries still don't have nuclear energy".
 

This is a very interesting point! I will chip in by pointing out a very similar remark from Rohin just earlier today [LW(p) · GW(p)]:

And I'll reiterate again because I anticipate being misunderstood that this is not a prediction of how the world must be and thus we are obviously safe; it is instead a story that I think is not ruled out by our current understanding and thus one to which I assign non-trivial probability.

That is all.

(Obviously there's a kinda superficial resemblance here to the phenomenon of "calling out" somebody else; I want to state outright that this is not the intention, it's just that I saw your comment right after seeing Rohin's comment, in such a way that my memory of his remark was still salient enough that the connection jumped out at me. Since salient observations tend to fade over time, I wanted to put this down before that happened.)

Echoing that I loved these conversations and I'm super grateful to everyone who participated — especially Richard, Paul, Eliezer, Nate, Ajeya, Carl, Rohin, and Jaan, who contributed a lot.

I don't plan to try to summarize the discussions or distill key take-aways myself (other than the extremely cursory job I did on https://intelligence.org/late-2021-miri-conversations/), but I'm very keen on seeing others attempt that, especially as part of a process to figure out their own models and do some evaluative work.

I think I'd rather see partial summaries/responses that go deep, instead of a more exhaustive but shallow summary; and I'd rather see summaries that center the author's own view (what's your personal take-away? what are your objections? which things were small versus large updates? etc.) over something that tries to be maximally objective and impersonal. But all the options seem good to me.

I chatted briefly the other day with Rob Bensinger about me turning them into a little book. My guess is I'd want to do something to compress especially the long Paul/Eliezer bet hashing out, that felt super long to me and not all worth the reading.

Interested in other suggestions for compression.

(This is not a commitment to do this, I probably won't.)

Here is a heavily condensed summary of the takeoff speeds thread of the conversation, incorporating earlier points made by Hanson, Grace, etc. https://objection.lol/objection/3262835

:)

(kudos to Ben Goldhaber for pointing me to it)

I wrote this doc a couple of years ago (while I was at CHAI). It's got many rough edges (I think I wrote it in one sitting and never bothered to rewrite it to make it better), but I still endorse the general gist, if we're talking about what systems are being deployed to do and what happens amongst organizations. It doesn't totally answer your question (it's more focused on what happens before we get systems that could kill everyone), but it seems pretty related.

(I haven't brought it up before because it seems to me like the disagreement is much more in the "mechanisms underlying intelligence", which that doc barely talks about, and the stuff it does say feels pretty outdated; I'd say different things now.)

(Not very sure I understood your description right, but here is my take:)

• I think your proposal is not explaining some crucial steps, which are in fact hard. In particular, I understood it as "you have AI which can give you blueprints for nano sized machines". But I think we already have some blueprints, this isn't an issue. How we assemble them is an issue.
• I expect that there will be more issues like this that you would find if you tried writing the plan in more detail.

However, I share the general sentiment behind your post --- I also don't understand why you can't get some pivotal act by combining human intelligence with some narrow AI. I expect that Eliezer have tried to come up with such combinations and came away with some general takeaways on this being not realistic. But I haven't done this exercise, so it seems not obvious to me. Perhaps it would be beneficial if many more people tried doing the exercise and then communicated the takeaways.

they download an existing chip schematic, and scale it down

Uh, how big do you think contemporary chips are?

I got the impression Eliezer's claiming that a dangerous superintelligence is merely sufficient for nanotech.

How would you save us with nanotech? It had better be good given all the hardware progress you just caused!

After reading some of the newer MIRI dialogues, I'm less convinced than I once was that I know what "corrigibility" actually is. Could you say a few words about what kind of behavior you concretely expect to see from a "corrigible" agent, followed by how [you expect] those behaviors [to] fit into the "trajectory-constraining" framework you propose in your post?

EDIT: This is not purely a question for Steven, incidentally (or at least, the first half isn't); anyone else who wants to take a shot at answering should feel free to do so. In particular I'd be interested in hearing answers from Eliezer or anyone else historically involved in the invention of the term.

I'm a little confused what it hopes to accomplish. I mean, to start I'm a little confused by your example of "preferences not about future states" (i.e. 'the pizza shop employee is running around frantically, and I am laughing' is a future state).

But to me, I'm not sure what the mixing of "paperclips" vs "humans remain in control" accomplishes. On the one hand, I think if you can specify "humans remain in control" safely, you've solved the alignment problem already. On another, I wouldn't want that to seize the future: There are potentially much better futures where humans are not in control, but still alive/free/whatever. (e.g. the Sophotechs in the Golden Oecumene are very much in control). On a third, I would definitely, a lot, very much, prefer a 3 star 'paperclips' and 5 star 'humans in control' to a 5 star 'paperclips' and a 3 star 'humans in control', even though both would average 4 stars?

When alignment schemes fail to scale, I think it typically means that they work while the system is unable to overpower/outsmart the oversight process, and then break down when the system becomes able to do so. I think that this usually results in the AI shifting from behavior that is mostly constrained by the training process to behavior that is mostly unconstrained (once they effectively disempower humans).

I think the results are relatively unlikely to be good in virtue of "the AI internalized something about our values, just not everything", and I'm pretty skeptical of recognizable "near miss" scenarios rather than AI gradually careening in very hard-to-predict directions with minimal connection with the surface features of the training process. 

Overall I think that the most likely outcome is a universe that is orthogonal to anything we directly care about, maybe with a vaguely similar flavor owing to convergence depending on how AI motivations shake out. (But likely not close enough to feel great, and quite plausibly with almost no visible relation. Probably much more different from us than we are from aliens.)

I think it's fairly plausible that the results are OK just because of galaxy-brained considerations about cooperation and niceness, where we might have been in the AI's shoes and part of being a good cosmic citizen is not worrying too much about who gets to do what they want with the universe. That said, I'd put that at <50% chance, with uncertainty both over empirical questions of how the AI is trained and what kind of psychology that leads to, and very hard problems in moral philosophy.

It's also fairly plausible to me (maybe also ~50%) that such systems will care enough about humans to give them a tiny slice (e.g. 1e-30) of the universe, whether as part of complicated trade with other civilizations who didn't mess up alignment so much (or nicer AIs, or whatever other nice civilization is willing to spare 1e-30 of their endowment to help us out), or just because you don't have to care very much at all. But of course it's also plausible that destructive conflict between aggressive civilizations leads to horrifying outcomes for us, so it's not all roses (and even when everything is good those trades do come at real costs).

Don't have strong views on any of those questions; they seem important but not closely related to my day job so I haven't thought about them too much.

Basically agree with Paul, and I especially want to note that I've barely thought about it and so this would likely change a ton with more information. To put some numbers of my own:

• "No generalization": 65%
• "Some generalization": 5% (I don't actually have stories where this is an outcome; this is more like model uncertainty)
• "Lots of generalization, but something went wrong": 30%

These are from my own perspective of what these categories mean, which I expect are pretty different from yours -- e.g. maybe I'm at ~2% that upon reflection I'd decide that hedonium is great and so that's actually perfect generalization; in the last category I include lots of worlds that I wouldn't describe as "existing humans recognizably survive", e.g. we decide to become digital uploads, then get lots of cognitive enhancements, throw away a bunch of evolutionary baggage, but also we never expand to the stars because AI has taken control of it and given us only Earth.

I think the biggest avenues for improving the answers would be to reflect more on the kindness + cooperation and acausal trade stories Paul mentions, as well as the possibility that a few AIs end up generalizing close to correctly and working out a deal with other AIs that involves humanity getting, say, Earth.

Given that not every biological civilization solves the problem, what does the rest of the multiverse look like?

If we're imagining civilizations very similar to humanity, then the multiverse looks like ~100% of one of the options. Reality's true answer will be very overdetermined [LW · GW]; it is a failure of our map that we cannot determine the answer. I don't know much about quantum physics / many-worlds, but I'd be pretty surprised if small fluctuations to our world made a huge difference; you'll need a lot of fluctuations adding up to a lot of improbability before you affect a macro-level property like this, unless you just happen to already be on the knife-edge.

If the biological civilizations could be very different from ours, then I have no idea how to quickly reason about this question and don't have an answer, sorry.

No, I very strongly predict they did not do things like that. I expect they (perhaps implicitly) predicted with high confidence that GPT-3 would not have the capabilities needed to kill everyone.

I'm going to try and write a table of contents for the textbook, just because it seems like a fun exercise.

Epistemic status: unbridled speculation

Volume I: Foundation

• Preface [mentioning, ofc, the infamous incident of 2041]
• Chapter 0: Introduction

Part I: Statistical Learning Theory

• Chapter 1: Offline Learning [VC theory and Watanabe's singular learning theory are both special cases of what's in this chapter]
• Chapter 2: Online Learning [infra-Bayesianism is introduced here, Garrabrant induction too]
• Chapter 3: Reinforcement Learning
• Chapter 4: Lifelong Learning [this chapter deals with traps, unobservable rewards and long-term planning]

Part II: Computational Learning Theory

• Chapter 5: Algebraic Classes [the theory of SVMs is a special case of what's explained here]
• Chapter 6: Circuits [learning various class of circuits]
• Chapter 7: Neural Networks
• Chapter 8: ???
• Chapter 9: Reflective Learning [some version of Turing reinforcement learning comes here]

Part III: Universal Priors

• Chapter 10: Solomonoff's Prior [including regret analysis using algorithmic statistics]
• Chapter 11: Bounded Simplicity Priors
• Chapter 12: ??? [might involve: causality, time hierarchies, logical languages, noise-tolerant computation...]
• Chapter 13: Physicalism and the Bridge Transform
• Chapter 14: Intelligence Measures

Part IV: Multi-Agent Systems

• Chapter 15: Impatient Games
• Chapter 16: Population Games
• Chapter 17: Space Bounds and Superrationality
• Chapter 18: Language [cheap talk, agreement theorems...]

Part V: Alignment Protocols

• Chapter 19: Quantilization
• Chapter 20: Malign Capacity Bounds [about confidence thresholds and consensus algorithms]
• Chapter 21: Value Learning [using the intelligence measures from chapter 14]
• Chapter 22: ??? [debate and/or some version of IDA might be here, or not]
• Chapter 23: ???

Volume II: Algorithms [about efficient algorithms for practical models of computation, and various trade-offs.]

???

Volume III: Applications

???

I don't think there is an "AGI textbook" any more than there is an "industrialization textbook." There are lots of books about general principles and useful kinds of machines. That said, if I had to make wild guesses about roughly what that future understanding would look like:

1. There is a recognizable concept of "learning" meaning something like "search for policies that perform well in past or simulated situations." That plays a large role, comparably important to planning or Bayesian inference. Logical induction is likely an elaboration of Bayesian inference that receives relatively little airtime except in specialized discussions.
2. This one is tougher given that I don't know what "the textbook" is. And I guess in the story all other humans are magically disappeared? If I was stuck with a single AWS cluster from 2022 and given unlimited time, I'd wildly guess that it would take me something between 1e4 and 1e8 years to create an autopoetic AI that obsoleted my own contributions (mostly because serial time is extremely valuable and I have a lot of compute). Writing the textbook does not seem like very much work after having done the deed?
3. I'd roughly guess big sections on learning, inference, planning, alignment, and clever algorithms for all of the above. I'd guess maybe 50% of content is smart versions of stuff we know now and 50% is stuff we didn't figure out at the time, but it depends a lot on how you define this textbook.

I'm mostly going to answer assuming that there's not some incredibly different paradigm (i.e. something as different from ML as ML is from expert systems). I do think the probability of "incredibly different paradigm" is low.

I'm also going to answer about the textbook at, idk, the point at which GDP doubles every 8 years. (To avoid talking about the post-Singularity textbook that explains how to build a superintelligence with clearly understood "intelligence algorithms" that can run easily on one of today's laptops, which I know very little about.)

I think I roughly agree with Paul if you are talking about the textbook that tells us how to build the best systems for the tasks that we want to do. (Analogy: today's textbook for self-driving cars.) That being said, I think that much of the improvement over time will be driven by improvements specifically in ML. (Analogy: today's textbook for deep learning.) So we can talk about that textbook as well.

1. It's a textbook that's entirely about "finding good programs through a large, efficient search with a stringent goal", which we currently call ML. The content may be primarily some new approach for achieving this, with neural nets being a historical footnote, or it might be entirely about neural nets (though presumably with new architectures or other changes from today). Logical induction doesn't appear in the textbook.
2. Jeez, who knows. If I intuitively query my brain here, it mostly doesn't have an answer; a thousand vs. million vs. billion years don't really change my intuitive predictions about what I'd get done. So we can instead back it out from other estimates. Given timelines of 10^1 - 10^2 years, and, idk, ~10^6 humans working on the problem near the end, seems like I'm implicitly predicting ~10^7 human-years of effort in our actual world. Then you have to adjust for a ton of factors, e.g. my quality relative to the average, the importance of serial thinking time, the benefit that real-world humans get from AI products that I won't get, the difficulty of exploration in thought-space by 1 person vs. 10^6 people, etc. Maybe I end up at ~10^5 years as a median estimate with wide uncertainty (especially on the right tail).
3. Jeez, who knows. Probably chapters / sections on how to define search spaces of programs (currently, "architectures"), efficient search algorithms within those spaces (currently, "gradient descent" and "loss functions"), how to set a stringent goal (currently, "what dataset to use").

1. Where is ML in this textbook? Is it under a section called "god-forsaken approaches" or does it play a key role? Follow-up: Where is logical induction?

Key role, but most current ML is in the "applied" section, where the "theory" section instead explains the principles by which neural nets (or future architectures) work on the inside. Logical induction is a sidebar at some point explaining the theoretical ideal we're working towards, like I assume AIXI is in some textbooks.

1. Is there anything else you can share about this textbook? Do you know any of the other chapter names?

Planning, Abstraction, Reasoning, Self-awareness.

I don't have any such advice at the moment.  It's not clear to me what makes a difference at this point.

For sure. It's tricky to wipe out humanity entirely without optimizing for that in particular -- nuclear war, climate change, and extremely bad natural pandemics look to me like they're at most global catastrophes, rather than existential threats. It might in fact be easier to wipe out humanity by enginering a pandemic that's specifically optimized for this task (than it is to develop AGI), but we don't see vast resources flowing into humanity-killing-virus projects, the way that we see vast resources flowing into AGI projects. By my accounting, most other x-risks look like wild tail risks (what if there's a large, competent, state-funded successfully-secretive death-cult???), whereas the AI x-risk is what happens by default, on the mainline (humanity is storming ahead towards AGI as fast as they can, pouring billions of dollars into it per year, and by default what happens when they succeed is that they accidentally unleash an optimizer that optimizes for our extinction, as a convergent instrumental subgoal of whatever rando thing it's optimizing).

(Off the cuff answer including some random guesses and estimates I won't stand behind, focused on the kind of theoretical alignment work I'm spending most of my days thinking about right now.)

Over the long run I would guess that alignment is broadly similar to other research areas, where a large/healthy field could support lots of work from lots of people, where some kinds of contributions are very heavy-tailed but there is a lot of complementarity and many researchers are having large overall marginal impacts.

Right now I think difficulties (at least for growing the kind of alignment work I'm most excited about) are mostly related to trying to expand quickly, greatly exacerbated by not having a good idea what's going on / what we should be trying to do, and not having a straightforward motivating methodology/test case since you are trying to do things in advance motivated by altruistic impact. I'm still optimistic that we will be able to scale up reasonably quickly such that many more people are helpfully engaged in the future and eventually these difficulties will be resolved.

In the very short term, while other bottlenecks are severe, I think it's mostly a question of how to use complementary resources (like mentorship and discussion) rather than "who could do useful work in an absolute sense." My vague guess is that in the short term the bar will be kind of quantitatively similar to "would get a tenure-track role at a top university" though obviously our evaluations of the bar will be highly noisy and we are selecting on different properties and at an earlier career stage.

I think it's much easier for lots of people to work more independently and take swings at the problem and that this could also be quite valuable (though there are lots of valuable things to do). Unfortunately I think that's a somewhat harder task and there are fewer people who will have a good time doing it. But at least the hard parts depend on a somewhat different set of skills (e.g. more loaded on initiative and entrepreneurial spirit and being able to figure things out on your own) so may cover some people who wouldn't make sense as early hires, and also there may be a lot of people who would be great hires but where it's too hard to tell from an application process.

"Possibly produce useful alignment work" is a really low bar, such that the answer is ~100%. Lots of things are possible. I'm going to instead answer "for what fraction of people would I think that the Long-Term Future Fund should fund them on the current margin".

If you imagine that the people are motivated to work on AI safety, get good mentorship, and are working full-time, then I think on my views most people who could get into an ML PhD in any university would qualify, and a similar number of other people as well (e.g. strong coders who are less good at the random stuff that academia wants). Primarily this is because I think that the mentors have useful ideas that could progress faster with "normal science" work (rather than requiring "paradigm-defining" work).

In practice, there is not that much mentorship to go around, and so the mentors end up spending time with the strongest people from the previous category, and so the weakest people end up not having mentorship and so aren't worth funding on the current margin.

I'd hope that this changes in the next few years, with the field transitioning from "you can do 'normal science' if you are frequently talking to one of the people who have paradigms in their head" to "the paradigms are understandable from the online written material; one can do 'normal science' within a paradigm autonomously".

Anthropic says that they're looking for experienced engineers who are able to dive into an unfamiliar codebase and solve nasty bags and/or are able to handle interesting problems with distributed systems and parallel processing. I was personally surprised to get an internship offer from CHAI and expected the bar for getting an AI safety role to be much higher. I'd guess that the average person able to get a software engineering job at Facebook, Microsoft, Google, etc. (not that I've ever received an offer from any of those companies), or perhaps a broader category of people, could do useful direct work, especially if they committed time to gaining relevant skills if necessary. But I might be wrong. (This is all assuming that Anthropic, Redwood, CHAI, etc. are doing useful alignment work.)

If humans could state a theorem such that, if only a theorem-prover would give us a machine-verifiable proof of it for us, the world would thereby be saved, we would be a great deal closer to survival than we are in reality.

This isn't how anything works, alas, nor is it a way that anything can work.

I don't think this is the main crux -- disagreements about mechanisms of intelligence seem far more important -- but to answer the questions:

Do you think major AI orgs will realize that AI is potentially worldendingly dangerous, and have any kind of process at all to handle that?

Clearly yes? They have safety teams that are focused on x-risk? I suspect I have misunderstood your question.

(Maybe you mean the bigger tech companies like FAANG, in which case I'm still at > 95% on yes, but I suspect I am still misunderstanding your question.)

(I know less about Chinese orgs but I still think "probably yes" if they become major AGI orgs.)

Do you think government intervention on AI regulations or policies will be net-positive or net-negative, for purposes of preventing x-risk?

Net positive, though mostly because it seems kinda hard to be net negative relative to "no regulation at all", not because I think the regulations will be well thought out. The main tradeoff that companies face seems to be speed / capabilities vs safety; it seems unlikely that even "random" regulations increase the speed and capabilities that companies can achieve. (Though it's certainly possible, e.g. a regulation for openness of research / reducing companies' ability to keep trade secrets.)

Note I am not including "the military races to get AGI" since that doesn't seem within scope, but if we include that, I think I'm at net negative but my view here is really unstable.

How quickly do you think the AI ecosystem will update on new "promising" advances (either in the realm of capabilities or the realm of safety)

Not sure how to answer this. Intellectual thought leaders will have their own "grand theory" hobbyhorses, from which they are unlikely to update very much. (This includes the participants in these dialogues.) But the communities as a whole will switch between "grand theories"; currently the timescale is 2-10 years. That timescale will shorten over time, but the rate at which such switches happen will lengthen. Later on, once we start getting into takeoff (e.g. GDP growth rate doubles), both the timescale and the rates shorten (but also most of my predictions are now very low probability because the world has changed a bunch in ways I didn't foresee).

For more simple advances like "new auxiliary loss that leads to more efficient learning" or "a particular failure mode and how to avoid it by retraining your human raters", the speed at which they are incorporated depends on complexity of implementation, amount of publicity, economic value, etc, but typical numbers are 1 week - 1 year.

How many intelligent, sensible people do there seem to be in the world who are thinking about AGI? (order of magnitude. like is there 1, 10, 1000, 100,000?)

"Sensible" seems incredibly subjective so that predictions about it do not actually lead to communication of information between people, so I'm going to ignore that. In that case, I'd say currently 300 FTE-equivalents in or adjacent to AI safety, and 1000 FTE-equivalents on AGI more generally.

What's your vague emotional valence towards "civilization generally", "the AI ecosystem in particular", or "the parts of government that engage with AGI".

Civilization generally seems pretty pathetic at doing good things relative to what "could" be accomplished. Incentives are frequently crazy. Good things frequently don't happen because of a minority with veto powers. Decisionmakers frequently take knowably bad actions that would look good to the people judging them. I am pretty frustrated at it.

Also, the world still runs mostly smoothly, and most of the people involved seem to be doing reasonable things given the constraints on them (+ natural human inclinations like risk aversion).

Peer review is extremely annoying to navigate and one of the biggest benefits of DeepMind relative to academia is that I don't have to deal with it nearly as much. Reviewers seem unable to grasp basic conceptual arguments if they're at all different from the standard style of argument in papers. Less often but still way too frequently they display failures of basic reading comprehension. The specific people I meet in the AI ecosystem are better but still have frustrating amounts of Epistemic Learned Helplessness and other heuristics like "theory is useless without experiments" that effectively mean that they can't ever make a hard update.

Also, more senior people are more competent and have less of these properties, people can and do change their minds over time (albeit over the course of years rather than hours), and the things AI people say often seem about as reasonable as the things that AI safety people say (controlling for seniority).

I'm scared of the zero-sum thinking that I'm told goes on in the military. Outside of the military, governments feel like hugely influential players whose actions are (currently) very unpredictable. It feels incredibly high-stakes, but also has the potential to be incredibly good.

What new facts would change your mind about any of the above questions?

Though it isn't a "fact", one route is to show me an abstract theory that does well at predicting civilizational responses, especially the parts that are not high profile (e.g. I want a theory that doesn't just explain COVID response -- it also explains seatbelt regulations, how property taxes are set, and why knives aren't significantly regulated, etc).

What were the causal nodes in your history that led you to form your opinions about the above?

Those were a lot of pretty different opinions, each of which has lots of different causal nodes. I currently don't see a big category that influenced all of them, so I think I'm going to punt on this question.

If you imagine turning out to be wrong/confused in some deep way about any of the above, do you have a sense of why that would turn out to be?

I only looked at things that can be observed online, rather than looking at all the other different ways that humans interact with the world; those other ways would have demonstrated obvious flaws in my answers.

Yes, it was an intentional part of the goal.

If there were any possibility of surviving the first AGI built, then it would be nice to have AGI projects promising to do something that wouldn't look like trying to seize control of the Future for themselves, when, much later (subjectively?), they became able to do something like CEV.  I don't see much evidence that they're able to think on the level of abstraction that CEV was stated on, though, nor that they're able to understand the 'seizing control of the Future' failure mode that CEV is meant to prevent, and they would not understand why CEV was a solution to the problem while 'Apple pie and democracy for everyone forever!' was not a solution to that problem.  If at most one AGI project can understand the problem to which CEV is a solution, then it's not a solution to races between AGI projects.  I suppose it could still be a solution to letting one AGI project scale even when incorporating highly intelligent people with some object-level moral disagreements.

My Eliezer-model thinks pivotal acts are genuinely, for-real, actually important. Like, he's not being metaphorical or making a pedagogical point when he says (paraphrasing) 'we need to use the first AGI systems to execute a huge, disruptive, game-board-flipping action, or we're all dead'.

When my Eliezer-model says that the most plausible pivotal acts he's aware of involve capabilities roughly at the level of 'develop nanotech' or 'put two cellular-identical strawberries on a plate', he's being completely literal. If some significantly weaker capability level realistically suffices for a pivotal act, then my Eliezer-model wants us to switch to focusing on that (far safer) capability level instead.

If we can save the world before we get anywhere near AGI, then we don't necessarily have to sort out how consequentialist, dangerous, hardware-overhang-y, etc. the first AGI systems will be. We can just push the 'End The Acute Existential Risk Period' button, and punt most other questions to the non-time-pressured Reflection that follows.

The goal is to bring x-risk down to near-zero, aka "End the Acute Risk Period". My usual story for how we do this is roughly "we create a methodology for building AI systems that allows you to align them at low cost relative to the cost of gaining capabilities; everyone uses this method, we have some governance / regulations to catch any stragglers who aren't using it but still can make dangerous systems".

If I talk to Eliezer, I expect him to say "yes, in this story you have executed a pivotal act, via magical low-cost alignment that we definitely do not get before we all die". In other words, the crux is in whether you can get an alignment solution with the properties I mentioned (and maybe also in whether people will be sensible enough to use the method + do the right governance). So with Eliezer I end up talking about those cruxes, rather than talking about "pivotal acts" per se, but I'm always imagining the "get an alignment solution, have everyone use it" plan.

When I talk to people who are attempting to model Eliezer, or defer to Eliezer, or speaking out of their own model that's heavily Eliezer-based, and I present this plan to them, and then they start thinking about pivotal acts, they do not say the thing Eliezer says above. I get the sense that they see "pivotal act" as some discrete, powerful, gameboard-flipping action taken at a particular point in time that changes x-risk from non-trivial to trivial, rather than as a referent to the much broader thing of "whatever ends the acute risk period". My plan doesn't involve anything as discrete and powerful as "melt all the GPUs", so from their perspective, a pivotal act hasn't happened, and the cached belief is that if a pivotal act hasn't happened, then we all die, therefore my plan leads to us all dying. With those people I end up talking about how "pivotal act" is a referent to the goal of "End the Acute Risk Period" and if you achieve that you have won and there's nothing else left to do; it doesn't matter that it wasn't "discrete" or "gameboard-flipping".

To answer the original question: if by "pivotal act" you mean "anything that ends the acute risk period", then I think that pivotal acts are strategically important. If by "pivotal act" you mean "discrete, powerful, gameboard-flipping actions", then I'm not all that interested in it but it seems fine to use it as a referent to the kind of intelligence level where it really matters that AGI is safe.

In all cases, the real answer is "the actual impact will depend a ton on the underlying argument that led to the update; that argument will lead to tons of other updates across the board".

I imagine that the spirit of the questions is that I don't perform a Bayesian update and instead do more of a "causal intervention" on the relevant node and propagate downstream. In that case:

1. I'm confused by the question. If the peak rate of GWP growth ever is 25%, it seems like the singularity didn't happen? Nonetheless, to the extent this question is about updates on the quality or duration of the singularity (as opposed to the leadup to it), I don't think this affects my actions at all.
2. I'm often acting based on my 10%-timelines, so if you tell me that TAI comes at exactly the midway point between now and my current median, that can counterintuitively have the same effect as lengthening my timelines. (Also I start sketching out far more concrete plans given this implausibly precise knowledge of when TAI comes.) So I'm instead going to answer the question where we imagine that my entire probability distribution is compressed / stretched along the time axis by a factor of 2. If compressed (shorter timelines), probably not much changes; I care more about having influence on AGI actors but I'm already in a great place for that. If stretched (longer timelines), I maybe focus more on weirder alignment theory, e.g. perhaps I work at ARC.
3. Not much effect. In more unipolar worlds, I spend more time predicting which labs will develop AGI, so that I can be there at crunch time; in more multipolar worlds I spend less time doing that.
4. No effect. Averting 50% of an existential catastrophe is still really good.
5. Similar effects as (2), but with much smaller magnitude. (Lower cost => more focus on weird alignment theory, since influence becomes less useful.)
6. This one particularly feels like I would be making some big Bayesian update (e.g. I think Eliezer's view predicts this much more strongly?) but if I ignore that then no effect.
7. Similar effects as (2), similar magnitudes as well. (More risk-averse => getting the right solution becomes more important than having influence.)
8. More dissimilar => more focus on weirder but more general work (e.g. less language models, more ELK). (This isn't the same as 2, 5, and 7 because this doesn't change how much I care about influence.)

I would "destroy the world" from the perspective of natural selection in the sense that I would transform it in many ways, none of which were making lots of copies of my DNA, or the information in it, or even having tons of kids half resembling my old biological self.

From the perspective of my highly similar fellow humans with whom I evolved in context, they'd get nice stuff, because "my fellow humans get nice stuff" happens to be the weird unpredictable desire that I ended up with at the equilibrium of reflection on the weird unpredictable godshatter that ended up inside me, as the result of my being strictly outer-optimized over millions of generations for inclusive genetic fitness, which I now don't care about at all.

Paperclip-numbers do well out of paperclip-number maximization. The hapless outer creators of the thing that weirdly ends up a paperclip maximizer, not so much.

I guess there's some meta-level question here that I'm interested in, as a sort of elaboration, which is something like: how do you go about balancing which meta-levels of the world to satisfy and which to destroy? [I kind of have a sense that Eliezer's answer can be guessed as an extension of the meta-ethics sequence, and so am interested both in his actual answer and other people's answers.]

For example, one might imagine a mostly-upload situation like The Metamorphosis of Prime Intellect / Friendship is Optimal / Second Life / etc., wherein everyone gets a materially abundant digital life in their shard of the metaverse, with communication heavily constrained (if nothing else, by requiring mutual consent). This, of course, discards as no-longer-relevant entities that exist on higher meta-levels; nations will be mostly irrelevant in such a world, companies will mostly stop existing, and so on.

But one could also apply the same logic a level lower. If you take Internal Family Systems / mental modules seriously, humans don't look like atomic objects, they look like a collection of simpler subagents balanced together in a sort of precarious way. (One part of you wants to accumulate lots of fat to survive the winter, another part of you wants to not accumulate lots of fat to look attractive to mates, the thing the 'human' is doing is balancing between those parts.) And so you can imagine a superintelligent system out to do right by the mental modules 'splitting them apart' in order to satisfy them separately, with one part swimming in a vat of glucose and the other inhabiting a beautiful statue, and discarding the 'balancing between the parts' system as no-longer-relevant.

Of course, applying this logic a level higher--the things to preserve are communities/nations/corporations/etc.--seems like it can quite easily be terrible for the people involved, and feels like it's preserving problems in order to maintain the relevance of traditional solutions.

My nonexpert guess would be that the "superintelligent brain emulation" class of solutions has potential to work. But we'd still need to figure out how to prevent an AGI from being made until we're ready to actually implement that solution.

If you post questions here, there's a decent chance I'll respond, though I'm not promising to.

Changing one's mind on P(doom) can be useful for people comparing across cause areas (e.g. Open Phil), but it's not all that important for me and was not one of my goals.

Generally when people have big disagreements about some high-level question like P(doom), it means that they have very different underlying models that drive their reasoning within that domain. The main goal (for me) is to acquire underlying models that I can then use in the future.

Acquiring a new underlying model that I actually believe would probably be more important than the rest of my work in a full year combined. It would typically have significant implications on what sorts of proposals can and cannot work, and would influence what research I do for years to come. In the case of Eliezer's model specifically, it would completely change what research I do, since Eliezer's model specifically predicts that the research I do is useless (I think).

I didn't particularly expect to actually acquire a new model that I believed from these conversations, but there was some probability of that, and I did expect that I would learn at least a few new things I hadn't previously considered. I'm unfortunately quite bad at noticing my own "updates", so I can't easily point to examples. That being said, I'm confident that I would now be significantly better at Eliezer's ITT than before the conversations.

I mean I had an impression that pretty much everyone assigned >5% probability to "if we scale we all die" so it's already enough reason to work on global coordination on safety.

What specific actions do you have in mind when you say "global coordination on safety", and how much of the problem do you think these actions solve?

My own view is that 'caring about AI x-risk at all' is a pretty small (albeit indispensable) step. There are lots of decisions that hinge on things other than 'is AGI risky at all'.

I agree with Rohin that the useful thing is trying to understand each other's overall models of the world and try to converge on them, not p(doom) per se. I gave some examples here [LW(p) · GW(p)] of some important implications of having more Paul-ish models versus more Eliezer-ish models.

More broadly, examples of important questions people in the field seem to disagree a lot about:

• Alignment
  • How hard is alignment? What are the central obstacles? What kind of difficulty is it? (Is it hard like 'building a secure OS that works on the first try'? Hard like 'the engineering/logistics/implementation portion of the Manhattan Project'? Both? Some other option? Etc.)
  • What alignment research directions are potentially useful, and what plans for developing aligned AGI have a chance of working?
• Deployment
  • What should the first AGI systems be aligned to do?
  • To what extent should we be thinking of "large disruptive act that upends the gameboard", versus "slow moderate roll-out of regulations and agreements across a few large actors"?
• Information spread
  • How important is research closure and opsec for capabilities-synergistic ideas? (Now, later, in the endgame, etc.)
• Path to AGI
  • Is AGI just "current SotA systems like GPT-3, but scaled up", or are we missing key insights?
  • More broadly, what's the relationship between current approaches and AGI?
  • How software- and/or hardware-bottlenecked are we on AGI?
  • How compute- and/or data-efficient will AGI systems be?
  • How far off is AGI? How possible is it to time future tech developments? How continuous is progress likely to be?
  • How likely is it that AGI is in-paradigm for deep learning?
  • If AGI comes from a new paradigm, how likely is it that it arises late in the paradigm (when the relevant approach is deployed at scale in large corporations) versus early (when a few fringe people are playing with the idea)?
  • Should we expect warning shots? Would warning shots make a difference, and if so, would they be helpful or harmful?
  • To what extent are there meaningfully different paths to AGI, versus just one path? How possible (and how desirable) is it to change which path humanity follows to get to AGI?
• Actors
  • How likely is it that AGI is first developed by a large established org, versus a small startup-y org, versus an academic group, versus a government?
  • How likely is it that governments play a role at all? What role would be desirable, if any? How tractable is it to try to get governments to play a good role (rather than a bad role), and/or to try to get governments to play a role at all (rather than no role)?

Nope.  It's just as hard and harder than aligning on some more limited pivotal task.  This is Sacrifice to the Gods; you imagine accepting some big downside but the big downside doesn't actually buy you anything.

Yep, fixed!

A bunch of people work on a bunch of different aspects of AI. You might be interested in AI governance and/or FATE* research (Fairness, Accountability, Transparency, Ethics). You can see some examples of the former in this spreadsheet under the AI governance category.

I don't know of anyone who works specifically on the questions you listed (but as mentioned above there are many who work on subquestions).

2030, 20%

For practical purposes, I'd say the pandemic is already over. MIRI isn't doing much hiring, though it's doing a little. The two big things we feel bottlenecked on are:

• (1) people who can generate promising new alignment ideas. (By far the top priority, but seems empirically rare.)
• (2) competent executives who are unusually good at understanding the kinds of things MIRI is trying to do, and who can run their own large alignment projects mostly-independently.

For 2, I think the best way to get hired by MIRI is to prove your abilities via the Visible Thoughts Project [LW · GW]. The post there says a bit more about the kind of skills we're looking for:

Eliezer has a handful of ideas that seem to me worth pursuing, but for all of them to be pursued, we need people who can not only lead those projects themselves, but who can understand the hope-containing heart of the idea with relatively little Eliezer-interaction, and develop a vision around it that retains the shred of hope and doesn’t require constant interaction and course-correction on our part. (This is, as far as I can tell, a version of the Hard Problem of finding good founders, but with an additional constraint of filtering for people who have affinity for a particular project, rather than people who have affinity for some project of their own devising.)

For 1, I suggest initially posting your research ideas to LessWrong, in line with John Wentworth's advice [LW · GW]. New ideas and approaches are desperately needed, and we would consider it crazy to not fund anyone whose ideas or ways-of-thinking-about-the-problem we think have a shred of hope in them. We may fund them via working at MIRI, or via putting them in touch with external funders; the important thing is just that the research happens.

If you want to work on alignment but you don't fall under category 1 or 2, you might consider applying to work at Redwood Research (https://www.redwoodresearch.org/jobs), which is a group doing alignment research we like. They're much more hungry for engineers right now than we are.

Suppose there are four possibilities you're uncertain between: A, B, C, and D. A implies X, B implies X, C implies X, and D implies X. So if A/B/C/D genuinely exhaust the options, then we can conclude X without needing to know which of the four is correct.

A simple concrete example might be: 'When I'm in wishful-thinking Abstract Mode, I imagine that one of my siblings will take out the trash today. But when I actually think about each individual sibling, I remember that Alice is sleeping over at a friend's house today, and Bob never takes out the trash, and Carol is really busy with her class project, and...'

I wrote a quick draft about this a month ago, working through the simple math of this kind of update. I've just put it up [LW · GW].

Idk, 95%? Probably I should push that down a bit because I haven't thought about it very hard.

It's a bit fuzzy what "deployed" means, but for now I'm going to assume that we mean that we put inputs into the AI system for the primary purpose of getting useful outputs, rather than for seeing what the AI did so that we can make it better.

Any existential catastrophe that didn't involve a failure of alignment seems like it had to involve a deployed system.

For failures of alignment, I'd expect that before you get an AI system that can break out of the training process and kill you, you get an AI system that can break out of deployment and kill you, because there's (probably) less monitoring during deployment. You're also just running much longer during deployment -- if an AI system is waiting for the right opportunity, then even if it is equally likely to happen for a training vs deployment input (i.e. ignoring the greater monitoring during training), you'd still expect to see it happen at deployment since >99% of the inputs happen at deployment.

Can we prove glider alignment by demonstrating an optimization process that will generate a Game of Life starting position where the inferred values of gliders are respected and fulfilled throughout the evolution of the game?

Here's a scheme for glider alignment. Train your AGI using deep RL in an episodic Game of Life. At each step, it gets +1 reward every time the glider moves forward, and -1 reward if not (i.e. it was disrupted somehow).

I assume you are not happy with this scheme because the we arbitrarily picked out values for gliders?

Good news! While gliders can't provide the rewards to the AGI (in order to define their values themselves), humans can provide rewards to the AI systems they are training (in order to define their values themselves). This is the basic (relevant) difference between humans and gliders, and is why I think primarily about alignment to humans and not alignment to simpler systems.

(This doesn't get into potential problems with inner alignment and informed oversight but I don't think those change the basic point.)

Hm, upon more thought I actually kind of endorse this as a demo. I think we should be able to run an alignment scheme on c. elegans and get out a universe full of well-fed worms, and that's a decent sign that we didn't screw up, despite the fact that it doesn't engage with several key problems that arise in humans because we're more complicated, have preferences about pur preferences, etc. No weird worm-stimulation should be needed. But we do have to accept that we're not getting some notion of values independent of an act of interpretation.

I feel a bit confused by these questions, but to take a stab at them anyway:

Are there projects (or even megaprojects [EA · GW]) that we can undertake, that can increase probability of noticeable GDP impact before we run AI in dangerous regimes? 

Sure. Encourage people to use the apply the best current AI techniques in actual applications. This leads to more applications before the dangerous regime. It also plausibly hastens the dangerous regime (because the dangerous thing gets deployed as an application more quickly).

Does getting noticeable GDP impact before we run AI in dangerous regimes, help to reduce actual likelihood of us running AI in dangerous regime?

One useful effect is that it increases the probability of a warning shot, and the severity of the warning shots that we do get. One harmful effect (of the action I suggested, not necessarily of the overall goal) is that it normalizes "deploy AI as fast as possible", including risky / dangerous AI.

I'd guess that the harms exceed the benefits.