Imitation learning considered unsafe?

I find it one of the more troubling outstanding issues with a number of proposals for AI alignment.

Which proposals? AFAIK Paul's latest proposal no longer calls for imitating humans in a broad sense (i.e., including behavior that requires planning), but only imitating a small subset of the human policy which hopefully can be learned exactly correctly. See this comment [LW(p) · GW(p)] where he wrote:

Are you perhaps assuming that we can max out regret during training for the agents that have to be trained with human involvement, but not necessarily for the higher level agents?

Yeah, I’m relatively optimistic that it’s possible to learn enough from humans that the lower level agent remains universal (+ aligned etc.) on arbitrary distributions. This would probably be the case if you managed to consistently break queries down into simpler pieces until arriving at a very simple queries.

ETA: Oh, but the same kind of problems you're point out here would still apply at the higher level distillation steps. I think the idea there is for an (amplified) overseer to look inside the imitator / distilled agent during training to push it away from doing anything malign/incorrigible (as Jessica also mentioned). Here is a post where Paul talked about this.

This sounds to me like an argument that inner optimizers are particularly likely to arise in imitation learning, because humans are pretty close to optimizers. Does that seem right?

In trying to argue for the safety of imitation learning, the key property of the imitation might be something like "the imitation is indistinguishable from the actual human on this distribution, where the distinguishers may be from this model class and have this information". GANs get a property like this pretty directly, although there is an issue in that they don't have access to the imitator's internal states, so e.g. they can't tell whether the imitator is using pseudorandomness or the input-based randomness imitators have access to.

Suppose we have an imitation system with a property like this. Then the argument for safety is going to look something like: "The human being imitated is corrigible. If the imitator weren't corrigible, then the discriminator could tell, because this would result in distinguishable actions."

For example, let's say that the imitated human is able to violently take control of the AI lab. In fact the imitated human doesn't do this (in any of the training data), and, arguably, it doesn't take a smart discriminator to tell that the human probably wouldn't do this in a new situation. So, if the imitator did do this (because e.g. it incorrectly inferred the human's decision theory), the discriminator could tell it apart. Of course, arguing that the discriminator generalizes this well would require some robustness argument; this particular problem seems easy (if the method for taking control involves taking really obvious actions like using weapons) but there might be more subtle ways of taking control. In those cases we would want some argument that, if the imitator comes up with a malign/incorrigible plan, then a discriminator with access to the imitator's internal states can notice this and notice that the imitated human wouldn't do this, because this isn't harder than coming up with the plan in the first place, and the discriminator is at least as capable as the imitator.

In general, while there are potential problems, I expect them to be more subtle than "the imitator incorrectly infers the human's decision theory and pursues convergent instrumental goals".

(Worth noting other problems with imitation learning, discussed in this post and this post)

If I'm taking your point correctly, it seems you're concerned about Goodharting in imitation learning. I agree, this seems a major issue, and I think people are aware of it and thinking about ways to address it.

Suppose you take a terabyte of data on human decisions and actions. You search for the shortest program that outputs the data, then see what gets outputted afterwards. The shortest program that outputs the data might look like a simulation of the universe with an arrow pointing to a particular hard drive. The "imitator" will guess at what file is next on the disk.

One problem for imitation learning is the difficulty in pointing out the human and separating them from the environment. The details of the humans decision might depend on what they had for lunch. (Of course, multiple different decisions might be good enough. But this illustrates that "imitate a human" isn't a clear cut procedure. And you have to be sure that the virtual lunch doesn't contain virtual mind control nanobots. ;-)

You could put a load of data about humans into a search for short programs that produce the same data. Hopefully the model produced will be some approximation of the universe. Hopefully, you have some way of cutting a human out of the model and putting them into a virtual box.

Alternatively you could use nanotech for mind uploading, and get a virtual human in a box.

If we have lots of compute and not much time, then uploading a team of AI researchers to really solve friendly AI is a good idea.

If we have a good enough understanding of "imitation learning", and no nanotech, we might be able to get an AI to guess the researchers mental states given observational data.

An imitation of a human might be a super-fast intelligence, with a lot of compute, but it won't be qualitatively super-intelligent.