The Rocket Alignment Problem

I expect that some otherwise convinceable readers are not going to realize that in this fictional world, people haven't discovered Newton's physics or calculus, and those readers are therefore going to miss the analogy of "this is how MIRI would talk about the situation if they didn't already know the fundamental concepts but had reasons for searching in the right direction". (I'm not thinking of readers incapable of handling that counterfactual, but of readers who aren't great at inferring implicit background facts from a written dialogue. Such readers might get very confused at the unexpected turns of the dialogue and quit rather than figure out what they're baffled by.)

I'd suggest adding to the preamble something like "In a weird world where people had figured out workable aeronautics and basic rocket propulsion by trial and error, but hadn't figured out Newton's laws or calculus".

I've curated this post. Here's the reasons that were salient to me:

• The post clearly communicated (to me) some subtle intuitions about what useful research looks like when you're deeply confused about basic principles.
• The question of "How would I invent calculus if I didn't know what calculus was and I was trying to build a plane" and finding some very specific and basic problems that cut to the heart of what we'd be confused about in that situation like "how to fire a cannonball such that it forever orbits the earth", point to a particular type of thinking that I can see being applied to understanding intelligence, and potentially coming up with incredibly valuable insights.
• I had previously been much more confused about why folks around here have been asking many questions about logical uncertainty and such, and this makes it much clearer to me.
• As usual, well-written dialogues like this are very easy and fun to read (which for me trades off massively with the length of the post).

Quick thoughts on further work that could be useful:

The post gives me a category of 'is missing some fundamental math like calculus' that applies to rocket alignment and AI alignment. I would be interested in some examples of how to look at a problem and write down a simple story [LW · GW] of how it works, including

• some positive and negative examples - situations where it works, situations where it doesn't
• what steps in particular seem to fail on the latter
• what signs say that the math should exist (e.g. some things are just complicated with few simple models predicting it - I expect a bunch of microbiology looks more like this than, say, physics).

I would also be interested to read historical case studies of this thing happening and what work lead to progress here - Newton, Shannon, others.

Meta: Eliezer previously made a quick attempt to summarise similar points in this SSC comment 2 years ago.

[Disclaimer: I'm reading this post for the first time now, as of 1/11/2020. I also already have a broad understanding of the importance of AI safety. While I am skeptical about MIRI's approach to things, I am also a fan of MIRI. Where this puts me relative to the target demographic of this post, I cannot say.]

Overall Summary

I think this post is pretty good. It's a solid and well-written introduction to some of the intuitions behind AI alignment and the fundamental research that MIRI does. At the same time, the use of analogy made the post more difficult for me to parse and hid some important considerations about AI alignment from view. Though it may be good (but not optimal) for introducing some people to the problem of AI alignment and a subset of MIRI's work, it did not raise or lower my opinion of MIRI as someone who already understood AGI safety to be important.

To be clear, I do not consider any of these weaknesses serious because I believe them to be partially irrelevant to the audience of people who don't appreciate the importance of AI-Safety. Still, they are relevant to the audience of people who give AI-Safety the appropriate scrutiny but remain skeptical of MIRI. And I think this latter audience is important enough to assign this article a "pretty good" instead of a "great".

I hope a future post directly explores the merit of MIRI's work on the context AI alignment without use of analogy.

Below is an overview of my likes and dislikes in this post. I will go into more detail about them in the next section, "Evaluating Analogies."

Things I liked:

• It's a solid introduction to AI-alignment, covering a broad range of topics including:
• Why we shouldn't expect aligned AGI by default
• How modern conversation about AGI behavior is problematically underspecified
• Why fundamental deconfusion research is necessary for solving AI-alignment
• It directly explains the value/motivation of particular pieces of MIRI work via analogy -- which is especially nice given that it's hard for the layman to actually appreciate the mathematically complex stuff MIRI is doing
• On the whole, the analogy is elegant

Things I disliked:

• Analogizing AI alignment to rocket alignment created a framing that hid important aspects of AI alignment from view and (unintentionally) stacked the deck in favor of MIRI.
• A criticism of rocket alignment research with a plausible AI alignment analog was neglected (and could only be addressed by breaking the analogy).
• An argument in favor of MIRI for rocket alignment had an AI analog that was much less convincing when considered in the context of AI alignment unique facts.
• The cognitive effort I spent mapping the rocket alignment problem to the AI alignment problem took more cognitive effort than just directly reading justifications of AI alignment and MIRI
• The world-building wasn't great
• The actual world of the dialogue is counterintuitive -- imagine a situation where planes and rockets exist (or don't exist, but are being theorized about), but no one knows calculus (despite modeling cannonballs pretty well) or how centripetal force+gravity works. It's hard for me to parse the exact epistemic meaning of any given statement relative to the world
• The world-building wasn't particularly clear -- it took me a while to completely parse that calculus hadn't been invented.
• There's a lot of asides where Beth (a stand-in for a member of MIRI) makes nontrivial scientific claims that we know to be true. While this is technically justified (MIRI does math and is unlikely to make claims that are wrong; and Eliezer has been right about about a lot of stuff and does deserve credit), it probably just feels smug and irritating to people who are MIRI-skeptics, aka this post's probable target.

Evaluating Analogies

Since this post is intended as an analogy to AI alignment, evaluating its insights requires two steps. First, one must re-interpret the post in the context of AI alignment. Second, one must take that re-interpretation and see whether it holds up. This means that, if I criticize the content of this post -- my criticism might be directly in error or my interpretation could be in error.

1. The Alignment Problem Analogy:

Overall, I think the analogy between the Rocket Alignment Problem and the AI Alignment Problem is pretty good. Structurally speaking, they're identical and I can convert one to the other by swapping words around:

Rocket Alignment: "We know the conditions rockets fly under on Earth but, as we make our rockets fly higher and higher, we have reasons to expect those conditions to break down. Things like wind and weather conditions will stop being relevant and other weird conditions (like whatever keeps the Earth moving around the sun) will take hold! If we don't understand those, we'll never get to the moon!"

AI Alignment: "We know the conditions that modern AI performs under right now, but as we make our AI solve more and more complex problems, we have reason to expect those conditions to break down. Things like model overfitting and sample-size limitations will stop being relevant and other weird conditions (like noticing problems so subtle and possible decisions so clever that you as a human can't reason about them) will take hold! If we don't understand those, we'll never make an AI that does what we want!"

1a. Flaws In the Alignment Problem Analogy:

While the alignment problem is pretty good, it leaves out the key and fundamentally important fact that failed AI Alignment will end the world. While it's often not a big deal when an analogy isn't completely accurate, missing this fact leaves MIRI-skeptics with a pretty strong counter-argument that can only exist outside of the analogy:

In Rocket Alignment terms -- "Why bother thinking about all this stuff now? If conditions are different in space, we'll learn that when we start launching things into space and see things happen to them? This sounds more efficient than worrying about cannonballs."

In AI Alignment terms -- "Why bother thinking about all this stuff now? If conditions are different when AI start getting clever, we'll learn about those differences once we start making actual AI that are clever enough to behave like agents. This sounds more efficient than navel-gazing about mathematical constructs."

If you explore this counter-argument and its counter-counter-argument deeper, the conversation gets pretty interesting:

MIRI-Skeptic: Fine okay. The analogy breaks down there. We can't empirically study a superintelligent AI safely. But we can make AI that are slightly smarter than us but put security mechanisms around them that only AI extremely smarter than us would be expected to break. Then we can learn experimentally from the behavior of those AI about how to make clever AI safe. Again, easier than navel-gazing about mathematical constructs and we might expect this to happen because slow take-off.

MIRI-Defender: First of all, there's no theoretical reason we would expect to be able to extrapolate the behavior of slightly clever AI to the behavior of extremely clever AI. Second, we have empirical reasons for thinking your empirical approach won't work. We already did a test-run of your experiment proposal with a slightly clever being; we put Eliezer Yudkowsky in an inescapable box armed with only a communication tool and the guard let him out (twice!).

MIRI-Skeptic: Fair enough but... [Author's Note: There are further replies to MIRI-Defender but this is a dialogue for another day]

Given that this post is supposed to address MIRI skeptics and that the aforementioned conversation is extremely relevant to judging the benefits of MIRI, I consider the inabillity to address this argument to be a flaw -- despite it being an understandable flaw in the context of the analogy used.

2. The Understanding Intractably Complicated Things with Simple Things Analogy:

I think that this is a cool insight (with parallels to inverse-inverse problems [LW · GW]) and the above post captures it very well. Explicitly, the analogy is this: "Rocket Alignment to Cannonballs is like AI Alignment to tiling agents." Structurally speaking, they're identical and I can convert one to the other by swapping words around:

Rocket Modeling: "We can't think about rocket trajectories using actual real rockets under actual real conditions because there are so many factors and complications that can affect them. But, per the rocket alignment problem, we need to understand the weird conditions that rockets need to deal with when they're really high up and these conditions should apply to a lot of things that are way simpler than rockets. So instead of dealing with the incredibly hard problem of modeling rockets, let's try really simple problems using other high-up fast-moving objects like cannonballs."

AI Alignment: "We can't think about AI behavior using actual AI under actual real conditions because there are so many factors and complications that can affect them. But, per the AI alignment problem, we need to understand the weird conditions that AI need to deal with when they're extremely intelligent and these conditions should apply to a lot of things that are way simpler than modern AI. So instead of dealing with the incredibly hard problem of modeling AI, let's try the really simple problem of using other intelligent decision-making things like Tiling Agents."

3. The "We Need Better Mathematics to Know What We're Talking About" Analogy

I really like just how perfect this analogy is. The way that AI "trajectory" and literal physical rocket trajectory line-up feels nice.

Rocket Alignment: "There's a lot of trouble figuring out exactly where a rocket will go at any given moment as it's going higher and higher. We need calculus to make claims about this."

AI alignment: "There's a lot of trouble figuring out exactly what an AI will do at any given moment as it gets smarter and smarter (ie self-modification but also just in general). We need to understand how to model logical uncertainty to even say anything about its decisions."

4. The "Mathematics Won't Give Us Accurate Models But It Will Give Us the Ability to Talk Intelligently" Analogy

This analogy basically works...

Rocket Alignment: "We can't use math to accurately predict rockets in real life but we need some of if so we can even reason about what rockets might do. Also we expect our math to get more accurate when the rockets get higher up."

AI alignment: "We can't use math to accurately predict AGI in real life but we need some of if so we can even reason about what AGI might do. Also we expect our math to get more accurate when the AGI gets way smarter."

I also enjoy the way this discussion lightly captures the frustration that the AI Safety community has felt. Many skeptics have claimed their AGIs won't become misaligned but then never specify the details of why that wouldn't have it. And when AI Safety proponents produce situations where the AGI does become misaligned, the skeptics move the goal posts.

4a. Flaws in the "Mathematics Won't Give Us Accurate Models But It Will Give Us the Ability to Talk Intelligently" Analogy

On a cursory glance, the above analogy seems to make sense. But, again, this analogy breaks down on the object level. I'd expect being able to talk precisely about what conditions affect movement in space to help us make better claims about how a rocket would go to the moon because that is just moving in space in a particular way. The research (if successful) completes the set of knowledge needed to reach the goal.

But being able to talk precisely about the trajectory of an AGI doesn't really help us talk precisely about getting to the "destination" of friendly AGI for a couple reasons:

• For rocket trajectories, there are clear control parameters that can be used to exploit the predictions made by a good understanding of how trajectories work. But for AI alignment, I'm not sure what would constitute a control parameter that would exploit a hypothetical good understanding of what strategies superintelligent beings use to make decisions.
• For rocket trajectories, the knowledge set of how to get a rocket into a point in outer-space and how to predict the trajectories of objects in outer-space basically encompass the things one would need to know to get that rocket to the moon. For AGI trajectories, the trajectories depend on three things: it's decision theory (a la logical uncertainty, tiling agents, decision theory...), the actual state of the world that the AGI perceives (which is fundamentally unknowable to us humans, since the AGI will be much more perceptive than us), and its goals (which are well-known to be orthogonal to the AGI's actual strategy algorithms).
• Given the above, we know scenarios where we understand agent foundations but not the goals of our agents won't work. But, if we do figure out the goals of our agents, it's not obvious that controlling those superintelligent agents' rationality skills will be a good use of our time. After all, they'll come up with better strategies than we would.
• Like I guess you could argue that we can view our goals as the initial conditions and then use our agent foundations to reason about the AGI behavior given those goals and decide if we like its choices... But again, the AGI is more perceptive than us. I'm not sure if we could capably design toy circumstances for an AGI to behave under that would reflect the circumstances of reality in a meaningful way
• Also, to be fair, MIRI does work on goal-oriented stuff in addition to agent-oriented stuff. Corrigibility ,which the post later links to, is an example of this. But, frankly, my expectation that this kind of thing will pan out is pretty low.

In principle, the rocket alignment analogy could've written in a way that captured the above concerns. For instance, instead of asking the question "How do we get this rocket to the moon when we don't understand how things move in outer-space?", we could ask "How do we get this rocket to the moon when we don't understand how things move in outer-space, we have a high amount of uncertainty about what exactly is up there in outer-space, and we don't have specifics about what exactly the moon is?"

But that would make this a much different, and much more epistemologically labyrinthian post.

Minor Comments

1. I appreciate the analogizing of an awesome thing (landing on the moon) to another awesome thing (making a friendly AGI). The AI safety community is quite rationally focused mostly on how bad a misaligned AI would be but I always enjoy spending some time thinking about the positives.

2. I noticed that Alfonso keeps using the term "spaceplanes" and Beth never does. I might be reading into it but my understanding is that this is done to capture how deeply frustrating when people talk about the thing you're studying (AGI) like it's something superficially similar but fundamentally different (modern machine-learning but like, with better data).

However, coming into this dialogue without any background on the world involved, the apparent interchangeability of spaceplane and rocket just felt confusing.

3.

As an example of work we’re presently doing that’s aimed at improving our understanding, there’s what we call the “tiling positions” problem. The tiling positions problem is how to fire a cannonball from a cannon in such a way that the cannonball circumnavigates the earth over and over again, “tiling” its initial coordinates like repeating tiles on a tessellated floor –

Because of the deliberate choice to analogize tiling agents and tiling positions, I spent probably five minutes trying to figure out exactly what the relationship between tiling positions and rocket alignment meant about tiling agents and AI alignment. It seems to me tiling isn't clearly necessary in the former (understanding any kind of trajectory should do the job) while it is in the latter (understanding how AI can guarantee similar behavior in agents it creates seems fundamentally important).

My impression now is that this was just a conceptual pun on the idea of tiling. I appreciate that but I'm not sure it's good for this post. The reason I thought so hard about this was also because the Logical Discreteness/Logical Uncertainty analogy seemed deeper.

Next time I see somebody say "shoot for the moon so if you miss you'll land among the stars" I'm going to link them here.

Ouch.

This post is very frightening to me in that I had not viscerally understood just how far off we are from solving FAI. Are we really (probably) in the area of before solving the problem, invent Calculus? Is that Hyperbole? Or worse? Also, the post also gives me hope that the problem of FAI is solvable (hopefully tractably!). When i was at first shown the problems that arise from just trying something, it felt like no one had any attack whatsoever. Now i think i understand better that no one really knows anything, but they do have some sort of an attack on it. I'd feel pretty safe if one could formalise a solution in math without paradox.

This might constitute an artists complaint, but since i believe your goal is to effectively persuade/argue/show i think that the post suffered from stewing on the exact same material all the way through. Maybe contrast Beth/Alonso with a Dunce/Alonso interaction? Show how immediatly jumping on a solution is rather silly directly, not only indirectly?

Anyways, my sympathies if conversations like this are part of your daily (weekly) work. It hurt just to read.

I liked this at first, but about 6-7 paragraphs of back-and-forth in I got really tired, started skimming, and then realized it went on for way more pages.

I get that your internal experience of trying to talk to people about AI risk feels like going on and on in this sort of "humble and correct person argues with smugly incorrect person forever" way but I don't think that's a good format for actually convincing people.

Is it safe for me to assume that all the relevant sub-analogies within the dialogue have been linked already? By that I mean, wherever the specific example is an analogy for something actually from MIRI, that information is linked?

This made me want a State of MIRI Research post for the layperson, in the style of "the way hard problems are handled" instead of situating projects in their respective fields or within the objective of alignment. I feel like this post crystallized the meta-research level in a way I have not encountered before, and that might be an important contribution in its own right.

ALFONSO:  So the first few rockets might not hit the Moon, I get that. Might fall back down and hurt someone even. But that's how technology advances in general, doesn't it? The first planes crashed often, and it took a long time for all the details to be worked out both theoretically and experimentally, both helping each other. Wouldn't Moon Rocketry follow the standard pattern?

BETH:  We are very much worried that this is not what would happen at all! Moon Rocketry is special! The very first rocket that goes very high but misses the Moon will have enough power to destroy both the Earth and the Moon. And the Sun. And there is nothing we can do about it once it's fired. So we better get the first rocket as right as we possibly can, the stakes are just too high.

ALFONSO:  This sounds alarmist and super far fetched! But humor me, explain what reasons do you have for your suspicion.

BETH: Well, we suspect that to get really high up, a rocket will need to keep gaining power, the higher up, the more power, and if it misses the Moon, the fallout from its engines will be so bad, it may end up causing worldwide destruction.

ALFONSO: I think your analogy between propulsion and information is getting a bit strained.

I have a sense of déjà vu when reading this. Has Eliezer used the rocket-alignment analogy before somewhere?

... Aha, found it (or at least something that might be it): this LW article from 2015 [LW · GW].

Translation into Russian by me

Good for loading intuitions about complexity of philosophical progress disagreements.

It's been very helpful for understanding the motivations behind MIRI's "deconfusion" research, in particular through linking it to another hard technical problem.

This is a very important post. It provides a justification for why agent foundations research might be important which was always unclear to me.

Big fan of this but, like most of us, I knew all this already. What I want to know is, how effective is/was this when not preaching to the choir? What happens when someone who doesn't understand MIRI's mission starts to read this? I'd like to think it helps them grok what is going on reasonably often, but I could be fooling myself, and that question is ultimately the test of how vital this really is.

Clarified what MIRI is about.

Used for research for my EA/rationality novel project, this is a really good analogy!

I think this post is a good and memorable explanation-by-analogy of what kind of research MIRI is doing to solve AI risk and a good and memorable response to some common criticism of or confusion about the former.

I just wanted to flag that this post hasn't been reviewed [LW · GW] yet, despite being one of the most nominated posts. (And mosts of the nominations here are quite short).

The most obvious sort of review that'd be good to see is from people who were in this post's target demographic (i.e. people who hadn't understood or been unpersuaded about what sort of problem MIRI is trying to solve), about whether this post actually helped them understand that. 

I'd also be interested in reviews that grapple a bit more with "how well exactly does this metaphor hold up?", although I'm not sure how to go about that.

The funny thing is, once you are in orbit, pointing at Earth's moon as it comes up over the horizon DOES get you there. This fact was used by the Apollo astronauts to eyeball their maneuver.

In a world in which somehow we don't have Calculus, Gravity, or a Heliocentric model of the solar system, but somehow the US goverment, Google, and North Korea are all valid references and have the same connotations they do in our universe.

I hate to say this but I'm taking the side of the Spaceplane designers. Perhaps it's because it's what I know.

That is, I can't really extrapolate to fully sentient, free to act AI. What I see as plausible is we build a vast set of shared libraries used in automated systems all over the economy. And each of these is essentially a hierarchical stack of agents. So a robotic arm might have agents that use reinforcement learning to classify what's in range of the arm. And then internally there's an agent that evaluates possible paths and the estimated reward from moving the arm to do something. And then there's control algorithms that generate the exact motor control sequences of commands. And a higher level system above this that is commanding various robotic systems to do things, with a goal of making paperclips or whatever.

And meta-systems that can try possible architectural changes to pieces of this system and determine if the changes are 'better', using Baysian calculations of the distribution of performance.

So, ok, what stops the script that orders the robotic systems to do things to make paperclips from ordering up killbots so it can kill people who are blocking the system from making more paperclips?

Well the obvious thing to do would be to add more heuristics to your paperclip maker. Instead of just having a goal to make the number of paperclips bigger, add some more goals. Add a goal to keep energy consumption under a certain level. And factory space taken to a certain level. And put a value on each person you statistically kill so you consider killing a person (by, saying, having your robotic actuators impinge on the probability cloud where you think a human worker is) pretty bad. (but not infinitely bad or your actuators won't move at all because the probability a human is in the way is never exactly zero)

So that's the "guidance fins" of this analogy. And the truth is, once you start getting spaceplanes into orbit and notice that heading right for the moon isn't making progress, you could probably get together some mathematicians and scrum together a rough model of orbital mechanics in time for the next launch.

And making a spaceplane so powerful it wrecks the planet if it crashes into it, when you don't know what you are doing...seems implausible to me. I agree we can make an AI that powerful but I think we would need to know what we are doing. Nobody made fission bombs work by slamming radioactive rocks together, it took a set of millions of deliberate actions in a row, by an army of people, to get to the first nuclear weapon.

A nuclear weapon is a system of tightly interconnected components. Some parts of it are extremely pure substances that took enormous energy (and human effort) to obtain.

I think a sentient AI is the same way. Nobody's inventing one at a terminal in their basement. You'll need libraries, trained on millions of working robot systems in the real world. Probably some new programming languages meant for the purpose that we haven't invented yet. A vast network of server-clouds that provide the compute power to test and iterate on your models. Your actual underlying machine learning techniques will most likely been machine optimized.

Once you have all these pieces available to parties with sufficient budgets, it would be like having a way to order highly enriched plutonium from Granger. Then it would be possible to build a closed-loop, self improving system.