Just Imitate Humans?

Thanks for all the links.

Wow, when I click these links from greaterwrong.com, they go to arbital.greaterwrong.com which loads instantly. Thanks to Said for the nice work!

Here are some of my thoughts on these posts. Thank you again for linking them.

Against mimicry:

Humans and machines have very different capabilities. Even a machine which is superhuman in many respects may be completely unable to match human performance in others. In particular, most realistic systems will be unable to exactly mimic human performance in any rich domain.

In light of this, it’s not clear what mimicry even means, and a naive definition won’t do what we want.

I don’t understand why an approximation of optimal sequence prediction doesn’t do what we want. That makes the objective minimizing the KL-divergence from the human policy to the imitation policy, but I think it is easier to think of this as just proper Bayesian updates (approximately). When there are too few samples, or using a bad approximation of optimal prediction, the imitator could fail, as the blocks examples describes. But a) it will learn to do everything that a human can do that it “can” learn, and b) the complaint that what we really want is for the imitator to just solve the task is just a wish for safe AGI. Yes, if better, more capable options than imitation can be resolved as safe, they will be superior.

Mimicry and Meeting Halfway:

It would be great if there were some way to get the best of both worlds [between approval directed agents (good for weak reasoners) and imitation (good from strong reasoners)]

We’ll be able to teach Arthur to achieve the task X if it can be achieved by the “intersection” of Arthur and Hugh 

If I’m understanding correctly, this seems more like getting the worst of both worlds. (Or at least doing no better than imitation).

Also, the generator (i.e. the agent) has an incentive to take over the world to shut off the discriminator.

Edit: I was ascribing too much agent-ness to the generator, which might be relevant for future GAN-inspired stuff, but for current versions of GANs, its only conception of the discriminator is its gradient update, and it doesn't believe the output of the discriminator depends on the state of the world. Depending on the internals of the discriminator, this incentive might reappear, but I'm not sure.

Reliable prediction:

I think this is a question of confidence calibration. I don’t know how to tractably approximate ideal reasoning, but I don’t this really jeopardizes imitation learning.

Safe training procedure for human-imitators:

How do we train a reinforcement learning system to imitate a human producing complex outputs such as strings?

Supervised learning suffices: tractably approximate ideal reasoning. I know this is a non-answer, but I don’t know the details of how to do this. This most naturally falls under the retrodiction category in the article. The “tractable approximations” which computational complexity problems threaten nonetheless seem attainable to me given the existence of humans.

Selective similarity metrics for imitation:

These are some interesting ideas on the problem that I am abstracting away regarding tractably approximating optimal sequence prediction.

Whole brain emulation and DL imitation learning:

Seems reasonable. Worth stressing something I think Gwern would agree with: a WBE inspired DL architecture for an artificial agent is definitely not going to make it safe by default.

Imitation Learning Considered Unsafe?:

1) Training a flexible model with a reasonable simplicity prior to imitate (e.g.) human decisions (e.g. via behavioral cloning) should presumably yield a good approximation of the process by which human judgments arise, which involves a planning process.

2) We shouldn't expect to learn exactly the correct process, though.

3) Therefore imitation learning might produce an AI which implements an unaligned planning process, which seems likely to have instrumental goals, and be dangerous.

If I’m understanding correctly, the concern is that the imitator learns how humans plan before learning what humans want, so then it plans like a human toward the achievement of some inhuman goal. I don’t think this causes an existential catastrophe. Human-like planning in the service of very roughly human-like goals just doesn’t seem to me to be similar at all to take-over-the-world behavior.

The AI that Pretends to be Human:

This seems quantilization-like, but without the formal guarantees of quantilization. I like quantilization a lot. I intend to think more about whether it could be extended to a multi-action instead of single-action definition, and whether one could use an approximate human policy rather than a perfect one.

Elaborations on Apprenticeship Learning:

Rather than imitating human behavior, the AI system imitates the behavior of a human who has access to a collection of AI assistants. These assistants can also be trained using AL-with-bootstrapping. In principle, such a process could scale well past human level.

This seems to be HCH (the prediction version). One reason why I think HSIFAUH might be superior is that if you have a bunch of copies of HSIFAUH that are in a flexible management hierarchy, intelligent agents can be in charge of allocating resources effectively between instances, and restructuring communication protocols, whereas with HCH, there is the fixed tree hierarchy. More critically, if I’m understanding HCH correctly, it is trained by having an actual human with access to the freshest version of HCH, and then HCH gets trained on the human’s output. If the real is human is the “manager”, or if the human eventually assumes that role, there is never any more training on the subtasks, like making a good spreadsheet for a manager to look at. A training regimen for the human could be designed ad hoc around when to query the actual human with different subtasks, or you could use the approach of HSIFAUH to query a human when there is sufficient expected information gain. But I think the capabilities of vanilla HCH depend a lot on how you design the set of tasks it is trained on.

Counterfactual human-in-the-loop:

The situation that this proposal is designed for, if I’m understanding correctly, is that we have an otherwise unaligned and otherwise dangerous AGI, but if it attempted to take over the world, a human would recognize its behavior as dangerous, and can step in. This proposal is to replace the human with a human imitation to make it all more efficient. If we are in that situation, I agree this is a good proposal for a speedup. I don’t think we will find ourselves in that situation.

A reinforcement learner has to model the world. A policy imitator has to model a household of humans. But the world can’t be modeled without modeling humans, and a household of humans can’t be modeled with modeling (their model of) the world. So on the face of it, these two things seem about equally difficult to me, in terms of both sample complexity and the “intelligence” required of the model.

Also, if an AGI is a world-model and a planner, and the world-model part is about as hard as the policy imitator, then any time spent planning slows down the AGI. Heuristics are powerful, and planning doesn’t have to be optimal, but in adversarial contexts, in general, planning is in the realm of PSPACE complete.

current ML methods would get us mesa-optimizers

Maybe you mean the methods you expect we will use? I don't think current ML methods make mesa-optimizers.

I know you're not disputing this, but I think it's worth having this formal result in the background: for a maximum a posteriori predictor that assigns positive prior probability to the truth, for all $\epsilon$, for sufficiently large $T$, the predictor will be within $\epsilon$ of the truth when assigning probability to all events (even regarding events well into the future). But yes, I think mesa-optimizers are something to keep in mind, especially if we use good heuristics to pick a model to see if it is maximum a posteriori (since in reality, we wouldn't be comparing all possible models).

Side note: I was just thinking about what a mesa-optimizer designed to be robust to gradient updates might look like. Could it try to ensure that small changes to its "values" would be relatively inconsequential to its behavior? For the decision at every timestep between "blend in" and "treacherous turn", it seems like gradient updates would shift its probability of toward "blend in". Could it avoid this?

Also, compared to my fears about other areas of alignment, I feel pretty decent about the possibility of weeding out mesa-optimizers by biasing toward fast or memory-lite functions.

Yeah I agree that this might secretly be the same as a question about uploads.

If you're only trying to copy human behavior in a coarse-grained way, you immediately run into a huge generalization problem because your human-imitation is going to have to make plans where it can copy itself, think faster as it adds more computing power, can't get a hug, etc, and this is all outside of the domain it was trained on.

So if people aren't being very specific about human imitations, I kind of assume they're really talking and thinking about basically-uploads (i.e. imitations that generalize to this novel context by having a model of human cognition that attempts to be realistic, not merely predictive).

Wouldn't this take an enormous amount of observation time to generate enough data to learn a human-imitating policy?

Yes, although we could start now.

Also, I just wanted to give the simplest possible proposal. More reasonably, data like this could probably be gathered in many ways.

Also, there is an underlying genetically determined structure upon which the morals are built by learning. An AGI without that structure, even if exposed to same situations, would not develop the same morals.