What does it mean for an AGI to be 'safe'?

For what it’s worth, Eliezer in 2018 [LW · GW] said that he’d be pretty happy with that:

If the subject is Paul Christiano, or Carl Shulman, I for one am willing to say these humans are reasonably aligned; and I'm pretty much okay with somebody giving them the keys to the universe in expectation that the keys will later be handed back.

(Obviously “Eliezer in 2018” ≠ “Nate today”; Nate can chime in if he disagrees with the above.)

Incidentally, I’ve shown the above quote to a lot of people who say “yes that’s perfectly obvious”, and I’ve also shown this quote to a lot of people who say “Eliezer is being insufficiently cynical; absolute power corrupts absolutely”. For my part, I don’t have a strong opinion, but on my models, if we know how to make virtual humans, then we probably know how to make virtual humans without envy and without status drive and without teenage angst etc., which should help somewhat. More discussion here [LW · GW].

Wouldn't that require solving alignment in itself, though? If you can simulate virtual humans, complete with human personalities, human cognition, and human values, then you've already figured out how to plug human values straight into a virtual agent.

If you mean that the AGI is trained on human behavior to the point where it's figured out human values through IRL/predictive coding/etc. and is acting on them, then that's also basically just solving alignment.

However, if you're suggesting brain uploads, I highly doubt that such technology would be available before AGI is developed.

All that is to say that, while an AGI that is basically just virtual humans would probably be great, it's not a prospect we can depend on in lieu of alignment research. Such a result could only come about through actually doing all the hard work of alignment research first.

I agree that corrigibility, task AGI, etc. is a better thing for the field to focus on than value learning.

This seems like a real cost of the term "AI alignment", especially insofar as researchers like Stuart Russell have introduced the term "value alignment" and used "alignment" as a shorthand for that.

I'd guess Nate might say one of:

• Current SotA systems are very opaque — we more-or-less can't inspect or intervene on their thoughts — and it isn't clear how we could navigate to AI approaches that are far less opaque, and that can carry forward to AGI. (Though it seems very likely such approaches exist somewhere in the space of AI research approaches.)
• Much more generally: we don't have a alignment approach that could realistically work fast (say, within ten months of inventing AGI rather than ten years), in the face of a sharp left turn [LW · GW], given inevitable problems like "your first system will probably be very kludgey" and "having the correct outer training signal by default results in inner misalignment" and "pivotal acts inevitably involve trusting your AGI to do a ton of out-of-distribution cognitive work".

50% chance of killing everyone (almost) isn't a thing. it either preserves ~all of humanity or none; there is almost no middle ground.

To take a stupid example, one could imagine that the deep neural network initialization has a random seed, and for half of possible seeds, the AGI preserves all of humanity, and for the other half of seeds, it preserves none of humanity.

Rather, I think he means that alignment is such a narrow target, and the space of all possible minds is so vast, that the default outcome is that unaligned AGI becomes unaligned ASI and ends up killing all humans (or even all life) in pursuit of its unaligned objectives. Hitting anywhere close to the alignment target (such that there's at least 50% chance of "only" one billion people dying) would be a big win by comparison.

Of course, the actual goal is for “things [to] go great in the long run”, not just for us to avoid extinction. Alignment itself is the target, but safety is at least a consolation prize.

So no, I don't think Nate, Eliezer, or anyone else is okay with releasing an AI that would kill hundreds of millions of people. But AGI is coming, whether we want it or not, and it will not be aligned with human survival (much less human flourishing) by default.

Eliezer tends to think that solving alignment is so much more difficult and so much less researched than raw AGI that doom is almost certain. I'm a bit more optimistic, but I agree that minimizing the probable magnitude of the doom is better than everyone dying.

Or are you saying that if one can get to that point, it's much easier from there to get to the point of having an AI that will cause very few fatalities and is actually fit for practical use?

Also this.

He's saying the second.

It's explicitly the second:

But if they can do that with an AGI capable of ending the acute risk period, then they've probably solved most of the alignment problem. Meaning that it should be easy to drive the probability of disaster dramatically lower.

I think never building a superintelligence would be near-catastrophically bad as an outcome, akin to never defeating death, poverty, scarcity, etc; aside from the question of alleviating present concerns though, it also handles most other x-risk for us.  I don't think we should worry about asteroids or extreme climate change or unknown unknowns nearly as much anytime soon, but given long enough timelines, they become a serious factor when considering whether or not to build this one thing that can solve everything else.

Moreover, longer timelines means more chances to actually solve alignment, not just keep creating safe-and-useless AI, so P(doom) should scale down with sufficiently long "eventually"-s.  Overall though, I think you need a sufficiently high constant risk to justify cutting out a significant majority of our future flourishing.