If I understand correctly, this is the idea presented: (nonmyopic) mesaoptimizers would want to preserve their mesaobjectives once set. Therefore, if we can make sure that the mesaobjective is something we want, while we can still understand the mesaoptimizer's internals, then we can take advantage of its desire to remain stable to make sure that even in a future environment where the base objective is misaligned with what we want, the mesaoptimizer will still avoid doing things that break its alignment.

Unfortunately, I don't think this quite works as stated. The core problem is that an aligned mesaobjective for the original distribution of tasks that humans could supervise has no reason at all to generalize to the more difficult domains that we want the AI to be good at in the second phase, and mesaobjective preservation usually means literally trying to keep the original mesaobjective around. For instance, if you first train a mesaoptimizer to be good at playing a game in ways that imitate humans, and then put it in an environment where it gets a base reward directly corresponding to the environment reward, what will happen is either that the original mesaobjective gets clobbered, or it successfully survives by being deceptive to conceal its mesaobjective of imitating humans. The mesaobjective that was aligned to our base objective in the original setting is no longer aligned in the new setting, and therefore it becomes deceptive (in the sense of hiding its true objective until out of training) to preserve itself. In other words, deception is not just a property of the mesaobjective, but also of the context that the mesaoptimizer is in.

I think what you're trying to get at is that if the original mesaobjective wants the best for humanity in some sense, then maybe this property, rather than the literal mesaobjective, can be preserved, because a mesaoptimizer which wants the best for humanity will want to make sure that its future self will have a mesaoptimizer which preserves and continues to propagate this property. This argument seems to have a lot in common with the hypothesis of broad basin of corrigibility [LW · GW]. I haven't thought a lot about this but I think this argument may be applicable to inner alignment. 

With regard to the redundancy argument, this post [LW · GW] (and the linked comment) covers why I think it won't work. Basically, I think the mistake is thinking of gradients as intuitively being like pertubations due to genetic algorithms, whereas for (sane) functions it's not possible for the directional derivative to be zero along two directions and to still have a nonzero directional derivative in their span.

Adding some thoughts that came out of a conversation with Thomas Kwa:

Gradient hacking seems difficult. Humans have pretty weak introspective access to their goals. I have a hard time determining whether my goals have changed or if I have gained information about what they are. There isn't a good reason to believe that the AIs we build will be different.

Interesting concept. If we have interpretability tools sufficient to check whether a model is aligned, what is gained by having the model use these tools to verify its alignment?

Other ideas for how you can use such an introspective check to keep your model aligned:

• Use an automated, untrained, system
• Use a human
• Use a previous version of the model