Reframing misaligned AGI's: well-intentioned non-neurotypical assistants

I think this is the wrong way of looking at it. Because in this analogy, the PA is "genuinely trying their hardest to optimize for your values", it's just poor at understanding these values. That problem is basically ignorance, and so by making the PA smarter or more aware, we can solve the problem.

But an AGI that fully understood your values, would still not optimise for them if it had a bad goal. The AGI is not well-intentioned-but-weird-in-implementation; its intentions themselves are alien/weird to us.

I've curated this post for these reasons:

• You put solid work into understanding central ideas in alignment, and published 4 posts simultaneously (1 [LW · GW], 2 [LW · GW], 3 [LW · GW]) communicating some key intuitions you've picked up. Doing this communicative work is really valuable and something I want to celebrate.
• The posts helped me develop clear intuitions about corrigibility, approval-directed agents, and agent-foundations.
• The posts are short, concrete, and easy to understand, which is the opposite of basically all the rest of the writing about AGI alignment.

My biggest hesitation(s) with curating this post:

• I am somewhat hesitant to share simple intuition pumps about important topics, in case those intuition pumps are misleading.
• I had a real hard time which of the four to pick, and have gone with the one that I thought people would find the clearest to read.

I was really excited that you wrote these posts, and learned a lot from them (plus the ensuing discussion in the comments).

In Superintelligence, Nick Bostrom talks about various "AI superpowers". One of these is "Social manipulation", which he summarizes as

Social and psychological modeling, manipulation, rhetoric persuasion

Strategic relevance:

• Leverage external resources by recruiting human support
• Enable a “boxed” AI to persuade its gatekeepers to let it out
• Persuade states and organizations to adopt some course of action
• AI can expropriate computational resources over the Internet

And Eliezer Yudkowsky writes:

There’s a popular concept of “intelligence” as book smarts, like calculus or chess, as opposed to say social skills. So people say that “it takes more than intelligence to succeed in human society”. But social skills reside in the brain, not the kidneys. When you think of intelligence, don’t think of a college professor, think of human beings; as opposed to chimpanzees. If you don’t have human intelligence, you’re not even in the game.

In order to have elite social skills, you need to be able to form accurate models about the thoughts & intentions of others. But being able to form accurate models about the thoughts & intentions of an overseer is exactly the ability we'd like to see in a corrigible AI.

If we can build AI systems that form those models without being goal-driven agents, maybe it's possible to have the benefits of elite social skills without the costs. I'm optimistic that this is the case--many of our most powerful model-building techniques don't really behave as though they have some kind of goal they are trying to achieve in the world.

Minor point: You changed the title! The title used to be better.

I think this is a clever new way of phrasing the problem.

When you said 'friend that is more powerful than you', that also made me think of a parenting relationship. We can look at whether this well-intentioned personification of AGI would be a good parent to a human child. They might be able to give the child a lot of attention, a expensive education, and a lot of material resources, but they might take unorthodox actions in the course of pursuing human goals.

So if not Human Approval, are we not saying it has to have certain levels of Self-Faith? I.e. it has to improve based off it own set of Ideals, that in the long run are what we actually want from a benevolent God?

I think this might be the Case, although of course, who am I to think something is unilaterally true...

So would not then, a set of Ideals need to be created which, for Us as Humans, makes us Better Humans; and based off that try to converge to a set of Ideals that an AGI can then inherit and start Self Improving on It's Own Ideas about Improvements.

We can't birth a Child, and then expect it to stay a Child. At some point, it must be Independent, Self-Sufficient and Sufficiently Rational.