If we solve alignment, do we die anyway?

Yes; except that a successful act can still be quite difficult.

You could reframe the concern to be that pivotal acts in a slow takeoff are prone to be bloody and dangerous. And because they are, and humans are likely to retain control, a pivotal act may be put off until it's even more bloody - like a nuclear conflict or sending the sun nova.

Worse yet, the "pivotal act" may be performed by the worst (human) actor, not the best.

This is an excellent point. I do not want to give up on value alignment. And I will endeavor to not make it seem impossible or not worth working on.

However, we also need to be realistic if we are going to succeed.

We need specific plans to achieve value alignment. I have written about alignment plans for likely AGI designs. They look to me like they can achieve personal intent alignment, but are much less likely to achieve value alignment. Those plans are linked here. Having people, you or others, work out how those or other alignment plans could lead to robust value alignment would be a step in having them implemented.

One route to value alignment is having a good person or people in charge of an intent aligned AGI, having them perform a pivotal act, and using that AGI to help design working stable value alignment. That is the best long term success scenario I see.

There we go!

This type of scheme to split a rapidly-growing pie semi fairly will definitely help reduce the urge to strike first.

If proliferation continues unchecked, we'll have RSI-capable AGI in the hands of teenagers and other malcontents eventually. And they often have irrational urges to strike first :)

But this type of scheme might stabilize the situation amongst a few AGIs in different hands, allowing them to collectively enforce not creating more and proliferating further.

Yes to all of the first paragraph. A caveat is that there's a big difference between humans remaining nominally in charge of an AGI-driven economy and not. If we're still technically in charge, we will retire (however many of us those in charge care to support; hopefull eventually quadrillions or so); if not, we'll be either entirely extinct or have a few of us maintained for historical interest by the new AGI overlords.

I see no way to meaningfully pause AI in time. We could possibly pause US progress with adequate fearmongering, but that would just make China get there first. That could be a good thing if they're more cautious, which it now seems they might very well be [LW(p) · GW(p)]. That would be only if Xi or whoever winds up in charge is not a sociopath. Which I have no idea about.

Absolutely. I mentioned getting advice briefly in this short article and a little more in Instruction-following AGI is easier... [LW · GW]

The problem in that case is that I'm not sure your b) is true. I certainly hope it is. I agree that it's unclear. That's why I'd like to get more analysis of a multipolar human-controlled ASI scenario. I don't think people have thought about this very seriously yet.

Edit: I very much agree with your arguments against sleepwalking and against the continuation of normality. I think the "inattentive world" hypothesis is all but disproven, and it still plays an outsized role in alignment thinking.

I don't think the arguments in that section depend on any assumption of normality or sleepwalking. And the multipolar scenario is the problem, so it can't be part of a solution. They do depend on people making nonoptimal decisions, which people do constantly.

So I think the arguments in that section are more general than you're hoping.

If those don't hold, what is the alternate scenario in which a multipolar world remains safe?

I agree entirely with the points made in that post. AGI will only "transform" the economy temporarily. It will very soon replace the economy. That is an entirely separate concern.

If you don't think a multipolar scenario is as offense-advantaged as I've described, where do you think the argument breaks down? What defensive technologies are you envisioning that could counter the types of offensive strategies I've mentioned?

Great point. I definitely mean fully capable of recursive self-improvement - that is, needing no humans in the loop. This lengthens the timelines to at least when we have roughly human-level robotics that are commercially available- but I expect that to be ten years or less.

The hardware requirements for early AGI are another factor in the timeline before this RSI-catastrophe is possible. Let's remember that algorithmic progress is roughly as fast as hardware progress to date, so that will also cease to be a large limitation all too soon.

The problem is that not having that scenario be immediately a risk may make people complacent about allowing lots of parahuman AGI before it becomes superhuman and fully RSI capable.

capable of RSI in the weak sense of being able to do capabilities research and help plan training runs

The speed at which this kind of thing is possible is crucial, even if capabilities are not above human level. This speed can make planning of training runs less central to the bulk of worthwhile activities. With very high speed, much more theoretical research that doesn't require waiting for currently plannable training runs becomes useful, as well as things like rewriting all the software, even if models themselves can't be "manually" retrained as part of this process. Plausibly at some point in the theoretical research you unlock online learning, even the kind that involves gradually shifting to a different architecture, and the inconvenience of distinct training runs disappears.

So this weak RSI would either need to involve AIs that can't autonomously research, but can help the researchers or engineers, or the AIs need to be sufficiently slow and non-superintelligent that they can't run through decades of research in months.

Good idea, done.

Thanks for reading, and responding! It's very helpful to know where my arguments cease being convincing or understandable.

I fully agree that just having AI do the work of solving alignment is not a good or convincing plan. You need to know that AI is aligned to trust it.

Perhaps the missing piece is that I think alignment is already solved for LLM agents. They don't work well, but they are quite eager to follow instructions. Adding more alignment methods as they improve makes good odds that our first capable/dangerous agents are also aligned. I listed some of the obvious and easy techniques we'll probably use in Internal independent review for language model agent alignment [AF · GW]. I'm not happy with the clarity of that post, though, so I'm currently working on two followups that might be clearer.

Or perhaps the missing link is going from aligned AI systems to aligned "Real AGI" [LW · GW]. I do think there's a discontinuity in alignment once a system starts to learn continuously and reflect on its beliefs (which change how its values/goals are interpreted). However, I think the techniques most likely to be used are probably adequate to make those systems aligned - IF that alignment is for following instructions, and the humans wisely instruct it to be honest about ways its alignment could fail.

So that's how I get to the first aligned AGI at roughly human level or below.

From there it seems easier, although still possible to fail.

If you have an agent that's aligned and smarter than you, you can trust it to work on further alignment schemes. It's wiser to spot-check it, but the humans' job becomes making sure the existing AGI is truly aligned, and letting it do the work to align its successor, or keep itself aligned as it learns.

I usually think about the progression from AGI to superintelligence as one system/entity learning, being improved, and self-improving. But there's a good chance that progression will look more generational, with several distinct systems/entities as successors with greater intelligence, designed by the previous system and/or humans. Those discontinuities seem to present more danger of getting alignment wrong

Great question. I was thinking of adding an edit to the end of the post with conclusions based on the comments/discussion. Here's a draft:

None of the suggestions in the comments seemed to me like workable ways to solve the problem.

I think we could survive an n-way multipolar scenario if n is small - like a handful of ASIs controlled by a few different governments. But not indefinitely - unless those ASIs come up with coordination strategies no human has yet thought of (or argued convincingly enough that I've heard of it - this isn't really my area, but nobody has pointed to any strong possibilities in the comments).

So my conclusion was more on the side that it's going to be so obviously such a bad/dangerous scenario that it won't be allowed to happen.

Basically, the hope is that this all becomes viscerally obvious to the first people who speak with a superhuman AGI and who think about global politics. They'll pull their shit together, as humans sometimes do when they're motivated to actually solve hard problems.

Here's one scenario in which multipolarity is stopped. Similar scenarios apply if the number of AGIs is small and people coordinate well enough to use their small group of AGIs similarly to what I'll describe below.

The people who speak to the first AGIi(s) and realize what must be done will include people in the government, because of course they'll be demanding to be included in decisions about using AGI. They'll talk sense to leadership, and the government will declare that this shit is deathly dangerous, and that nobody else should be building AGI.

They'll call for a voluntary global moratorium on AGI projects. Realizing that this will be hugely unpopular, they'll promise that the existing AGI will be used to benefit the whole world. They'll then immediately deploy that AGI to identify and sabotage projects in other countries. If that's not adequate, they'll use minimal force. False-flag operations framing anti-AGI groups might be used to destroy infrastructure and assassinate key people involved in foreign projects. Or who knows.

The promise to benefit the whole world will be halfway kept. The AGI will be used to develop military technology and production facilities for the government that controls it; but it will simultaneously be used to develop useful technologies that aid the problems most pressing for other governments. That could be useful tool AI, climate geoengineering, food production, etc.

The government controlling AGI keeps their shit together enough that no enterprising sociopath seizes personal control and anoints themselves god-emperor for eternity. They realize that this will happen eventually if their now-ASI keeps following human orders. They use its now-well-superhuman intelligence to solve value alignment sufficiently well to launch it or a successor as a fully autonomous sovereign ASI.

Humanity prospers under their sole demigod until the heat death of the universe, or an unaligned expansion AGI crosses our lightcone and turns everyone into paperclips. It will be a hell of a party for a subjectively very long time indeed. The one unbreakable rule will be that thou shalt worship no other god. All of humanity everywhere is monitored by copies of the sovereign AGI to prevent them building new AGI that aren't certified-aligned by the servant-god ASI. But since it's aligned and smart, it's pretty cool about the whole thing. So nobody minds that one rule a lot, given how much fun they're building everything and having every experience imaginable within the consent of all sentient entities involved.

I'd love to get more help thinking about how likely the central premise, that people get their shit together once they're staring real AGI in the face is. And what we can do now to encourage that.

I actually completely agree with this call to action. 

Unfortunately, I suspect that it's impossible to make value alignment easier than personal intent alignment. I can't think of a technical alignment approach that couldn't be used both ways equally well. And worse than that, I think that intent aligned AGI is easier than value aligned AGI for reasons I outline in that post, and Max Harms has elaborated in much more detail in Corrigibility as Singular Target sequence (as well as Paul Christiano and many others' arguments.

But I still agree with your call to action: we should be working now to make value alignment as safe as possible. That requires deciding what we align to. The concept of humanity is not well-defined in the future, when upgrades and digital copies of human minds become possible. Roger Dearnaley's sequence AI, alignment, and ethics [? · GW] lays out these problems and more; for instance, if we stick to baseline humans, the future will be largely controlled by whatever values are held by the most humans, in a competition for memes and reproduction. So there's conceptual as well as technical/mind-design work to be done on technical alignment.

And that work should be done. In multipolar scenarios with, someone may well decide to "launch" their AGI to be autonomous with value alignment, out of magnanimity or desperation. We'd better make their odds of success as high as we can manage.

I don't think refusing to work on intent alignment is a helpful option. It will likely be tried, with or without our help. Following instructions is the most obvious alignment target for any agent that's even approaching autonomy and therefore usefulness. Thinking about how to make those attempts successful will also increase our odds of surviving the first competent autonomous AGIs.

WRT definitions: alignment doesn't specify alignment with whom. I think this ambiguity is causing important confusions in the field.

I was trying to draw a distinction between two importantly different alignment goals, which I'm terming personal intent alignment and value alignment until better terminology comes along. More on that in an upcoming post.

If you did have an AGI that follows instructions and you told it "do the right thing", you'd have to specify right for who.

And during the critical risk period, that AGI wouldn't know for sure what the right thing was. We don't expect godlike intelligence right out of the gate. It won't know whether a risky takeover/pivotal act is the right move. If the situation is multipolar, it won't know even as it becomes truly superintelligent, because it will have to guess at the plans, technologies, and capabilities of other superintelligent AGI.

My call to action is this: help me understand and make or break the argument that a multipolar scenario is very bad, so that the people in charge of the first really successful AGI project know the stakes when they make their calls.

The problem is that "do the right thing" makes no sense without a reference to what values, or more formally what utility functions the human in question has, so there's no way to do what you propose to do even in theory, at least without strong assumptions on their values/utility functions.

Also, it breaks corrigiblity, and in many applications like military AI, this is a dangerous property to break, because you probably want to change their orders/actions, and this sort of anti-corrigiblity is usually bad unless you're very confident value learning works, which I don't share.

Perhaps seemingly obvious, but given some of the reactions around Apple putting "Do not hallucinate" into the system prompt of its AI ...

If you do get an instruction-following AI that you can simply give the instruction, "Do the right thing", and it would just do the right thing:

Remember to give the instruction.

That's a good point that the nuclear detente might become stronger with more actors, because the certainty of mutual destruction goes up with more parties that might start shooting if you do.

I don't think the coalition and treaties for counter-aggression are important with nukes; anyone can destroy everyone, they're just guaranteed to be mostly destroyed in response. The numbers don't matter much. And I think they'll matter even less with AGI than nukes - without the guarantee of mutually assured destruction, since AGI might allow for modes of attack that are more subtle.

Re-introducing mutually assured destruction could actually be a workable strategy. I haven't thought of this before, so thanks for pushing my thoughts in that direction.

I fully agree that non-iterated prisoner's dilemmas don't exist in the world as we know it now. And it's not a perfect fit for the scenario I'm describing- but it's frighteningly close. I use the term because it invokes the right intuitions among LWers, and it's not far off for the particular scenario I'm describing.

That's because, unlike the nuclear standoff or any other historical scenario, the people in charge of powerful AGI could be reasonably certain they'd survive and prosper if they're the first to "defect".

I'm pretty conscious of the benefits of cooperation in our modern world; they are huge. That type of nonzero sum game is the basis of the world we now experience. I'm worried that changes with RSI-capable AGI.

My point is that AGI will not need cooperation once it passes a certain level of capability. AGI capable of fully autonomous recursive self-improvement and exponential production (factories that build new factories and other stuff) doesn't need allies because it can become arbitrarily smart and materially effective on its own. A human in charge of this force would be tempted to use it. (Such an AGI would still benefit from cooperation on the margin, but it would be vastly less dependent on it than humans are).

So a human or small group of humans would be tempted to tell their AGI to hide, self-improve, and come up with a strategy that will leave them alive while destroying all rival AGIs before they are ordered to do the same. They might arrive at a strategy that produces a little collatoral damage or a lot - like most of mankind and the earth being destroyed. But if you've got a superintelligent inventor on your side and a few resources, you can be pretty sure you and some immediate loved ones can survive and live in material comfort, while rebuilding a new society according to your preferences.

Whether the nefarious team can conceal such a push for exponential progress is the question. I'm afraid surviving a multipolar human-controlled AGI scenario will necessitate ubiquitous surveillance. If that is handled ethically, that might be okay. It's just one of the more visible signs of the disturbing fact that anyone in charge of a powerful AGI will be able to take over the world if they want to - if they're willing to accept the collatoral damage. That is not the case with nukes or any historical scenarios - yet human leaders have made many decisions leading to staggering amounts of death and destruction. That's why I think we need AGI to be in trustworthy hands. That's a tall order but not an impossible one.