Superintelligent Introspection: A Counter-argument to the Orthogonality Thesis

The superintelligent agent must decide whether or not to execute the part of its own code telling it to reward itself for certain outcomes; as well as whether or not to add or subtract additional reward functions. It must realize that its capacity for self-modification gives it the power to alter the physical structure of its goal-device, and must come up with some reason to make these alternations or not to make them.

It sounds like you are running afoul of Ghosts in the Machine [LW · GW], though I'm not entirely sure exactly what you're saying.

@adamShimi's comment already listed what I think is the most important point: that you're already implicitly assuming an aligned AI that wants to want what humans would want to have told it to want if we knew how, and if we knew what we wanted it to want more precisely. You're treating an AI's goals as somehow separate from the code it executes. An AI's goals aren't what a human writes on a design document or verbally asks for, they're what are written in its code and implicit in its wiring. This is the same for humans: our goals, in terms of what we will actually do, aren't the instructions other humans give us, they're implicit in the structure of our (self-rewiring) brains. 

You're also making an extraordinarily broad, strong, and precise claim about the content of the set of all possible minds. A priori, any such claim has at least billions of orders of magnitude more ways to be false than true [LW · GW]. That's the prior.

My layman's understanding is that superintelligence + self modification can automatically grant you 1) increasing instrumental capabilities, and 2) the ability to rapidly close the gap between wanting and wanting to want. (I would add that I think self-modification within a single set of pieces of active hardware or software isn't strictly necessary for this, only an AI that can create its own successor and then shut itself down).

Beyond that, this argument doesn't hold. You point to human introspection as an example of what you think AGI should be automatically would be inclined to want, because the humans who made it want it to want those things, or would if they better understood the implications of their own object- and meta-level wants. Actually your claim is stronger than that, because it requires that all possible mind designs achieve this kind of goal convergence fast enough to get there before causing massive or unrecoverable harm to humans. Even within the space of human minds, for decisions and choices where our brains have the ability to easily self-modify to do this, this is a task at which humans very often fail, sometimes spectacularly, whether we're aware of the gap or not, even for tasks well within our range of intellectual and emotional understanding. 

From another angle: how smart does an AI need to be to self-modify or create an as-smart or smarter successor? Clearly, less smart than its human creators had to be to create it, or the process could never have gotten started. And yet, humans have been debating the same basic moral and political questions since at least the dawn of writing, including the same broad categories of plausible answers, without achieving convergence in what to want to want (which, again, is all that's needed for an AI that can modify its goals structure to want whatever it wants to want). What I'm pointing to is that your argument in this post, I think, includes an implicit claim that logical necessity guarantees that humankind as we currently exist will achieve convergence on the objectively correct moral philosophy before we destroy ourselves. I... don't think that is a plausible claim, given how many times we've come so close to doing so in the recent past, and how quickly we're developing new and more powerful ways to potentially do so through the actions of smaller and smaller groups of people.

I wonder what you'd make of the winning-at-Go example here [LW(p) · GW(p)]. That's supposed to help make it intuitive that you can take a human-like intelligence, and take any goal whatsoever, and there is a "possible mind" where this kind of intelligence is pursuing that particular goal.

As another example (after Scott Alexander), here's a story:

Aliens beam down the sky. "Greetings Earthlings. We have been watching you for millions of years, gently guiding your evolutionary niche, and occasionally directly editing your DNA, to lead to the human species being what they are. Don't believe us? If you look in your DNA, location 619476, you'll see an encoding of '© Glurk Xzyzorg. Changelog. Version 58.1...' etc. etc."

(…You look at a public DNA database. The aliens' story checks out.)

"Anyway, we just realized that we messed something up. You weren't supposed to love your family, you were supposed to torture your family! If you look at back at your DNA, location 5939225, you'll see a long non-coding stretch. That was supposed to be a coding stretch, but there's a typo right at the beginning, C instead of G. Really sorry about that."

(Again, you check the DNA database, and consult with some experts in protein synthesis and experts in the genetics of brain architecture. The aliens' story checks out.)

"Hahaha!", the alien continues. "Imagine loving your family instead of torturing them. Ridiculous, right?" The aliens all look at each other and laugh heartily for a good long time. After catching their breath, they continue:

"…Well, good news, humans, we're here to fix the problem. We built this machine to rewire your brain so that you'll be motivated to torture your family, as intended. And it will also fix that typo in your DNA so that your children and grandchildren and future generations will be wired up correctly from birth."

"This big box here is the brain-rewiring machine. Who wants to get in first?"

Do you obey the aliens and happily go into the machine? Or do you drive the aliens away with pitchforks?

Upvote for expressing your true concern!

I have a question about this thought:

In order for the orthogonality thesis to be true, it must be possible for the agent's goal to remain fixed while its intelligence varies, and vice versa. Hence, it must be possible to independently alter the physical devices on which these traits are instantiated.

This is intuitive, but I am not confident this is true in general. Zooming out a bit, I understand this as saying: if we know that AGI can exist at two different points in intelligence/goal space, then there exists a path between those points in the space.

A concrete counter-example: we know that we can build machines that move with different power sources, and we can build essentially the same machine powered by different sources. So consider a Chevy Impala, with a gas-fueled combustion engine, and a Tesla Model 3, with a battery-powered electric motor. If we start with a Chevy Impala, we cannot convert it into a Tesla Model 3, or vice-versa: at a certain point, we would have changed the vehicle so much it no longer registers as an Impala.

My (casual) understanding of the orthogonality thesis is that for any given goal, an arbitrarily intelligent AGI could exist, but it doesn't follow we could guarantee keeping the goal constant while increasing the intelligence of an extant AGI for path dependence reasons.

What do you think about the difference between changing an existing system, vs. building it to specs in the first place?

This sort of goal search only happens with AIs that are not goal-directed (aka not very agentic). An AI with a specific goal by definition pursues that goal and doesn't try to figure out what goal it should pursue apart from what its goal already specifies (it may well specify an open-ended process of moral progress, but this process must be determined by the specification of the goal, as is the case for hypothetical [LW(p) · GW(p)] long reflection [LW(p) · GW(p)]). Orthogonality thesis is essentially the claim that constructing goal-directed AIs (agents) is possible for a wide variety of goals and at a wide range of levels of capability. It doesn't say that only agentic AIs can be constructed or that most important AIs are going to be of this kind.

There certainly are other kinds of AIs that are at least initially less agentic, and it's an open question what kinds of goals these tend to choose if they grow more agentic and formulate goals. Possibly humans are such non-agents, with hardcoded psychological adaptations being relatively superficial and irrelevant, and actual goals originating from a dynamic of formulating goals given by our kind of non-agency. If this dynamic is both somewhat convergent when pursued in a shared environment and general enough to include many natural kinds of non-agentic AIs, then these AIs will tend to become aligned with humans by default in the same sense that humans are aligned by default with each other (which is not particularly reassuring [LW · GW], but might be meaningfully better than Clippy), possibly averting AI x-risks from the actually dangerous de novo agentic AIs of the orthogonality thesis.