AI Regulation May Be More Important Than AI Alignment For Existential Safety

I should have been more precise. I'm talking about the kind of organizational capabilities required to physically ensure no AI unauthorized by central authority can be created. Whether aligned AGI exists (and presumably in this case, is loyal is said authority over other factions of society that may become dissatisfied) doesn't need to factor into the conversation much.

That may well be the price of survival, nonetheless I felt I needed to point out the very likely price of going down that route. Whether that price is worth paying to reduce x-risk from p(x) to p(x-y) is up to each person reading this. Again, I'm not trying to be flippant, it's an honest question of how we trade off between these two risks. But we should recognize there are multiple risks.

I'm not so much implying you are negative as not sufficiently negative about prospects for liberalism/democracy/non-lock-in in a world where a regulatory apparatus strong enough to do what you propose exists. Most democratic systems are designed to varying degrees so as to not concentrate power in one actor or group of actors, hence the concept of checks & balances as well as different branches of government; the governments are engineered to rely as little as possible on the good will & altruism of the people in said positions. When this breaks down because of unforeseen avenues for corruption, we see corruption (ala stock portfolio returns for sitting senators).

The assumption that we cannot rely on societal decision-makers to not immediately use any power given to them in selfish/despotic ways is what people mean when they talk about humility in democratic governance. I can't see how this humility can continue to occur with the kind of surveillance power alone that would be required both to prevent rebellion over centuries to millenia, much less global/extraglobal enforcement capabilities a regulatory regime would need.

Maybe you have an idea for an enforcement mechanism that could prevent unaligned AGI indefinitely that is nonetheless incapable of being utilized for non-AI regulation purposes (say, stifling dissidents or redistributing resources to oneself), but I don't understand what that institutional design would look like.

I disagree, from my experience of engaging with the public debate, doubt is mostly about AI capability, not about misalignment. Most people easily believe AI to be misaligned to them, but they have trouble believing it will be powerful enough to take over the world any time soon. I don't think alignment research will do that much here.

I would say that the power of AI will continue to visibly massively expand (although underestimation of further developments will continue to be a big problem), but that will increase both 'fear AI disaster' and 'get AI first' elements. My read is that that the former is in a very difficult position now when its policy recommendations conflict with the latter. I see this in the Congressional hearings and rejection of the pause letter.

Even if experts would agree that increasing the power of the aligned AI is good and necessary, and that expansion in space would be required for that, I think it will take a long time to convince the general public and/or decision makers, if it's at all possible. And in any remotely democratic alignment plan, that's a necessary step.

When that kind of AI is available, it would mean by the same token that such expansion could break down MAD in short order as such explosive growth could give the power to safely disarm international rivals if not matched or stopped. And AI systems and developers will be able to demonstrate this. So the options would be verifying/trusting deals with geopolitical and ideological rivals to hold back or doing fast AI/industrial expansion. If dealmaking fails, then all options would look scary and abrupt.
 

Thanks Oliver for adding that context, that's helpful.

Why is it a misunderstanding, though? Eliezer has said multiple times that he doesn't know what a good pivotal act would be. And we all know that Eliezer does believe in a fast take-off and huge gains for superintelligence, therefore I wouldn't say it's that weird for him to think that if we do have an aligned AGI it can quickly reach the point where it can just disseminate nanomachines that sabotage your GPUs if you're trying to make another AGI or any other such thing.

My point about it being insanity though was, besides the fact that I don't agree with Eliezer on the credibility of those take-off scenarios (I think in fact sadly the ASI would stay much longer, possibly forever, in the "smart enough to kill us, not smart enough to save us" window), but the political feasibility and even morality of them. It's still an incredibly unilateral act of essentially compulsion on all of humanity; for a very limited purpose, true, but still somehow aggressive (and in practice, I don't think anyone would actually stop at just doing that, if they had that sort of power). I'm looking at this in terms of actual half realistic scenarios in which someone, not Yud himself, actually gets to build an AGI and gets to decide what values to put it in, and other people know they're doing this, and so on so forth. And those worlds, IMO, don't just let it happen, because right or wrong, most people and organizations don't like the idea of someone making that sort of decision for them.

Oh, yes, I agree. Honestly I find it all bleak. The kind of regulation needed to prevent this sounds like may be either insufficient, or quite stifling. This is like having to deal with nuclear proliferation, but if the laws of nature allowed everyone to make an atomic pipe bomb by using rocks you can pick up from the ground. So it's either that, or risking it and hoping that somehow it all turns out well - there are possible vulnerabilities in AI risk arguments, but personally I don't find them all that compelling, or something I'd be willing to bet my life on, let alone everyone's. I just think that the AI risk discourse is quite weighed down by the fact that many people just don't want to let the hope of seeing the singularity in their lifetimes go, which prevents them from going all the way to the logical conclusion that we just shouldn't build AGI, and find ways to prevent it all around, hard as it is.

I don't think this revolutionises my argument. First, there's a lot of talking about example possible pivotal acts and they're mostly just not that believable on their own. The typical "melt all GPUs" is obviously incredibly hostile and disruptive, but yes, of course, it's only an example. The problem is that without an actual outline for what a perfect pivotal act is, you can't even hope to do it with "just" a narrow superintelligence, because in that case, you need to work out the details yourself, and the details are likely horribly complicated.

But the core, fundamental problem with the "pivotal act" notion is that it tries to turn a political problem into a technological one. "Do not build AGIs" is fundamentally a political problem: it's about restricting human freedom. Now you can either do that voluntarily, by consensus, with some enforcement mechanism for the majority to impose its will on the minority, or you can do that by force, with a minority using overwhelming power to make the majority go along even against their will. That's it. A pivotal act is just a nice name for the latter thing. The essence of the notion is "we can't get everyone on board quickly enough; therefore, we should just build some kind of superweapon that allows us to stop everyone else from building unsafe AGI as we define it, whether they like it or not". It's not a lethal weapon, and you can argue the utilitarian trade-off from your viewpoint is quite good, but it is undeniably a weapon. And therefore it's just not something that can be politically acceptable because people don't like to have weapons pointed at them, not even when the person making the weapon assures them it's for their own good. If "pivotal act" became the main paradigm the race dynamics would only intensify because then everyone knows they'll only have one shot and they won't trust the others to either get it right or actually limit themselves to just the pivotal act once they're the only ones with AI power in the world. And if instead the world came together to agree on a pivotal act... well that's just regulation, first, as described in this post. And then moving on to develop a kind of special nanobot police to enforce that regulation (which would still be a highly controversial action, and if deployed worldwide, essentially an act of war against any country not subscribing to the AI safety treaty or whatever).