A caveat to the Orthogonality Thesis

The sharp left turn argument boils down to some handwavey analogy that evolution didn't optimize humans to optimize for IGF - which is actually wrong, as it clearly did [LW(p) · GW(p)], combined with another handwavey argument that capabilities will fall into a natural generalization attractor, but there is no such attractor for alignment. That second component of the argument is also incorrect, because there is a natural known attractor for alignment - empowerment [LW · GW]. The more plausible attractor argument is that selfish-empowerment is a stronger attractor than altruistic-empowerment.

Also more generally capabilities generalize through the world model, but any good utility function will also be defined through the world model, and thus can also benefit from its generalization.

Is it true that a more and more sophisticated world model is enough to “hit the core of intelligence”?

A sufficiently sophisticated world-model, such as one that includes human values, would include humans. Humans are general problem-solvers [LW · GW], so the AI would have an algorithm for general problem-solving represented inside it somewhere. And once it's there, I expect sharp steep gradients towards co-opting that algorithm for the AI's own tasks.

I agree [LW · GW] that figuring out a way to stop this, to keep the AI from growing agency even as its world-model becomes superhumanly advanced, would be fruitful.

A sufficiently strong world model can answer the question "What would a very smart very good person think about X?" and then you can just pipe that to the decision output, but that won't get you higher intelligence than what was present in the training material.

Shouldn't human goals have to be in the within human intelligence part, since humans have them? Or are we considering exactly human intelligence AI unsafe? Do you imagine a slightly dumber version of yourself failing to actualise your goals from not having good strategies, or failing to even embed them due to having a world model that lacks definitions of objects you care about?

Corrigibility has to be in the reachable part of the goals because a well-trained dog genuinely wants to do what you want it to do, even if it doesn't always understand, and even if following the command will get it less food than otherwise and this is knowable to the dog. You clearly don't need human intelligence to describe the terminal goal "Do what the humans want me to do", although it's not clear the goal will stay there as intelligence rises above human intelligence.

I think this makes total sense if you think of "how much intelligence" by fixing an agent-like architecture that wants the goal and then scaling the parts, and "intelligence" is something like the total effort being exerted by those parts.

But it isn't quite a caveat to formulations (like Bostrom's) that define "how much intelligence" in terms of external behavior rather than internal structure - maybe you've heard definitions like intelligence is how good it is at optimization across a wide range of domains. If that's your measuring stick, you can scale the world model without changing intelligence, so long as the search process doesn't output vetter plans on average.

EDIT: Both are points are moot using Stuart Armstrong's narrower definition of the Orthogonality thesis that he argues in General purpose intelligence: arguing the Orthogonality thesis [LW · GW]:

High-intelligence agents can exist having more or less any final goals (as long as these goals are of feasible complexity, and do not refer intrinsically to the agent’s intelligence).

Old post:

I was just working through my own thoughts on the Orthogonality thesis and did a search on LW on existing material and found this essay. I had pretty much the same thoughts on intelligence limiting goal complexity, so yay!

Additional thought I had: Learning/intelligence-boosting motivations/goals are positively correlated with intelligence. Thus, given any amount of time, an AI with intelligence-boosting motivations will become smarter than those do not have that motivation.

It is true that instrumental convergence should lead any sufficiently smart AI to also pursue intelligence-boosting (cognitive enhancement) but:

• At low levels of intelligence, AI might fail at instrumental convergence strategies.
• At high levels of intelligence, AI that is not intelligence-boosting will spend some non-zero amount of resources on its actual other goals and thus be less intelligent than an intelligence-boosting AI (assuming parallel universes, and thus no direct competition).

I'm not sure how to integrate this insight in to the orthogonality thesis. It implies that:

"At higher intelligence levels, intelligence-boosting motivations are more likely than other motivations" thus creating a probability distribution across the intelligence-goal space that I'm not sure how to represent. Thoughts?