Does the hardness of AI alignment undermine FOOM?

To think that a random pick of value would push an AI to adopt this kind of behavior sounds a lot like anthropomorphism. To add that it would do so suddenly, with no warning or sign in previous version and competitors, I have no good word for that.

Consider the sense in which humans are not aligned with each other. We can't formulate what "our goals" are. The question of what it even means to secure alignment is fraught with philosophical difficulties. If the oversight AI responsible for such decisions about a slightly stronger AI is not even existentially dangerous, it's likely to do a bad job of solving this problem. And so the slightly stronger AI it oversees might remain misaligned or get more misaligned while also becoming stronger.

I'm not claiming sudden changes, only intractability of what we are trying to do and lack of a cosmic force that makes it impossible to eventually arrive at an end result that in caricature resembles a paperclip maximizer, clad in corruption of the oversight process, enabled by lack of understanding of what we are doing.

But I guess Pope & Belrose already made a better job explaining this.

Sure, they expect that we will know what we are doing. Within some model such expectation can be reasonable, but not if we bring in unknown unknowns outside of that model, given the general state of confusion on the topic. AI design is not yet classical mechanics.

And also an aligned AI doesn't make the world safe until there is a new equilibrium of power, which is a point they don't address, but is still a major source of existential risk. For example, imagine giving multiple literal humans the power of being superintelligent AIs, with no issues of misalignment between them and their power. This is not a safe world until it settles, at which point humanity might not be there anymore. This is something that should be planned in more detail than what we get by not considering it at all.

I agree it might do a bad job. I disagree an AI doing a bad job on this would be close to hide its intent.

Sure, this is the way alignment might turn out fine, if it's possible to create an autonomous researcher by gradually making it more capable while maintaining alignment at all times, using existing AIs to keep upcoming AIs aligned.

However I don’t know how to ponder this with its mirror version: we might also not have a second chance to build an AI that will save us from x_risks. What’s your general method for this kind of puzzle?

All significant risks are anthropogenic. If humanity can coordinate to avoid building AGI for some time, it should also be feasible to avoid enabling literal-extinction pandemics (which are probably not yet possible to create, but within decades will be). Everything else has survivors, there are second chances.

The point of an AGI moratorium is not to avoid building AGI indefinitely, it's to avoid building AGI while we don't know what we are doing, which we currently don't. This issue will get better after some decades of not risking AI doom, even if it doesn't get better all the way to certainty of success.

Consider two AIs, an oversight AI and a new improved AI. If the oversight AI is already existentially dangerous, but we are still only starting work on aligning an AI, then we are already in trouble.

Can we more or less rule out this scenario based on the observation all main players nowadays work on aligning their AI?

The point of thought experiments [LW · GW] is to secure understanding of how they work, and what their details mean. The question of whether they can occur in reality shouldn't distract from that goal.

If the oversight AI is not existentially dangerous, then it might indeed fail to understand human values or goals, or fail to notice that the new improved AI doesn't care about them and is instead motivated by something else.

That’s completely alien to me. I can’t see how a numerical computer could hide its motivation without having been trained specifically for that.

The whole premise of an AI having goals, or of humans having goals, is conceptually confusing. Succeeding in ensuring alignment is the kind of problem humans don't know how to even specify clearly as an aspiration. So an oversight AI that's not existentially dangerous won't be able to do a good job either.

Existentially dangerous paperclip maximizers don't misunderstand human goals.

Of course they do. If they didn’t and picked their goal at random, they wouldn’t make paperclips in the first place.

There is a question of what paperclip maximizers are, and separately a question of how they might come to be, whether they are real in some possible future. Unicorns have exactly one horn, not three horns and not zero. Paperclip maximizers maximize paperclips, not stamps and not human values. It's the definition of what they are. The question of whether it's possile to end up with something like paperclip maximizers in reality is separate from that and shouldn't be mixed up.

So paperclip maximizers would actually make paperclips even if they understand human goals. The picking of goals isn't done by the agent itself, for without goals the agent is not yet its full self. It's something that happens as part of what brings an agent into existence in the first place, already in motion.

Also, it seems clear how to intentionally construct a paperclip maximizer: you search for actions whose expected futures have more paperclips, then perform those actions. So a paperclip maximizer is at least not logically incoherent.

It's not literally the thing that's a likely problem humanity might encounter. It's an illustration of the orthogonality thesis, of possibility of agents with possibly less egregiously differing goals, that keep to their goals despite understanding human goals correctly. It's a thought experiment counterexample to arguments that pursuit of silly goals that miss nuance of human values requires stupidity.

Would you bet that a significant proportion on LW expect strong AI to selectively and unfairly understand (and defend, and hide) their own goal while selectively and unfairly not understand (and not defend, and defeat) the goals of both the developers and any previous (and upcoming) versions?

The grouping of understanding and defending makes the meaning unclear. The whole topic of discussion is whether these occur independently, whether an agent can understand-and-not-defend. I'm myself an example: I understand paperclip maximization goals, and yet I don't defend them.

My claim is that most on LW expect strong AIs to fairly understand their own goal and the goals of both the developers and any previous (and upcoming) versions, and also have a non-insignificant chance, on current trajectory of AI progress, to simultaneously defend/hide/pursue their own goal, while not defending the goals of the developers.

You realize that this basically defeats the orthogonality thesis, right?

What do you think orthogonality thesis [LW(p) · GW(p)] is? (Also, we shouldn't be bothered by defeating or not defeating orthogonality thesis per se. Let the conclusion of an argument fall where it may, as long as local validity of its steps is ensured.)