Shane Legg interview on alignment

Consequentialism is not uniformly bad. I think the specific way Shane wants to make the models more consequentialist defends against some failure modes. Deep Deceptiveness [AF · GW] is essentially about humans being misled because the module that checks for safety of an action is shallow, and the rest of the model is smarter than it and locally pointed at goals that are more difficult to satisfy without misleading the operators. If the model in this story were fully aware of the consequences of its actions through deliberation, it could realize that modeling the human operators in a different ontology in order to route around them is still bad. (I feel like self-knowledge is more important here though.)

Deliberation also does not have to be consequentialist. The model could deliberate to ensure it's not breaking some deontological rule, and this won't produce instrumental pressure towards a coup.

Would be curious to hear your idea of some of the "difficulty and traps".

I think it is about making the models more consequentialist, in the sense of making them smarter and more agentic.

I don't see evidence that he's ignoring the hard parts of alignment. And I'm not even sure how optimistic he is, beyond presumably thinking success in a possible.

You could be right in assuming those are his reasons for optimism. That does seem ungenerous, but it could be true. Those definitely aren't my reasons for my limited optimism. See my linked pieces for those. Language model agents are modestly consequentialist and not automatically corrigible, but they have unique and large advantages for alignment. I'm a bit puzzled at the lack and of enthusiasm for that direction; I've only gotten vague criticisms along the lines of "that can't possibly work because there are ways for it to possibly fail". The lines of thought those critiques reference just argue that alignment is hard, not that it's impossible or that a natural language alignment approach couldn't work. So I'm really hoping to get some more direct engagement with these ideas.

That is very generous. My impression was that Shane Legg does not even know what the alignment problem is, or at least tried to give the viewer the idea that he didn't. His "solution" to the "alignment problem" was to give the AI a better world model and the ability to reflect, which obviously isn't an alignment solution, it's just a capabilities enhancement. Dwarkesh Patel seemed confused for the same reason.

That's what I've tried to do here, and I'd be happy to do a more thorough job, including direct quotes. I have a bit of a conflict of interest, since I think his ideas parallel mine closely. I've tried to note areas where I may be reading into his statements, and I can do that more carefully in a longer version.

I made a short clip highlighting how Legg seems to miss an opportunity to acknowledge the inner alignment problem, since his proposed alignment solution seems to be a fundamentally training / black box approach.

I think you're right that I'm reading into this. But there is probably more to his thinking, whether I'm right or wrong about what that is. Shane Legg was thinking about alignment as far back as his PhD thesis, which doesn't go into depth on it but does show he'd at least read a some of the literature prior to 2008.

I agree that LLM chain of thought is not totally reliable, but I don't think it makes sense to dismiss it as too unreliable to work with for an alignment solution. There's so much that hasn't been tried, both in making LLMs more reliable, and making agents built on top of them reliable by taking multiple paths, and using new context windows and different models to force them to break problems into steps, and use the last natural language statement as their whole context for the next step.

Whether or not this is a reliable path to alignment, it's a potential path to huge profits. So there are two questions: will this lead to alignable AGI? And, will it lead to AGI. I think both are unanswered.

I agree that Legg's focus was more on getting the model to understand our ethical principles. And he didn't give much on how you make a system that follows any principles. My guess is that he's thinking of something like a much-elaborated form of AutoGPT; that's what I mean by a language model agent. You prompt it with "come up with a plan that follows these goals" and then have a review process that prompts with something like "now make sure this plan fulfills these goals". But I'm just guessing that he's thinking of a similar system. He might be deliberately vague so as to not share strategy with competitors, or maybe that's just not what the interview focused on.