Agents That Learn From Human Behavior Can't Learn Human Values That Humans Haven't Learned Yet

If Alice received further information (from a moral philosopher, maybe), she'd start maximizing a specific one of those utility functions instead.

This is the key fact about Alice's behavior, which distinguishes it from Bob's behavior, so the question is whether an AI can learn that fact.

Of course the AI could if it ever observed Alice in a situation where she learned anything about morality.

Or any case that has any mutual information with how Alice would respond to moral facts. (For a sufficiently smart reasoner that includes everything---e.g. watching Alice eat breakfast gives you lots of general information about her brain, which in turn lets you make better predictions about how she would behave in other cases.)

And of course the AI would tend to create situations where Alice learned moral facts, since that's a very natural response to uncertainty about how she'd respond to moral facts.

So overall it seems like you'd have to restrict the behavior of the IRL agent quite far before this becomes a problem.

Yeah. In general, if we want a machine to learn from data as well as we can, we need to give it a prior that's as good as ours. And there's no guarantee that such a prior can itself be learned from data, because we didn't learn it from data (a lot of it came with our brain structure). We can try giving the machine more data, but we don't know how much or what kind of data would be enough.

I think that this might not end up being a problem if the value learning agent can communicate with Alice (e.g. in the context of CIRL). If they don't get any info from moral philosophers, then they should probably maximise something like the expectation of her utility function for the same reason that Alice does. If they do get info, they can just give Alice that info, see what she does, and act accordingly. I think the real problem comes in in the realistic case where Alice isn't handling moral uncertainty perfectly, so the value learning agent shouldn't actually maximise the weighted sum of the utility functions she's uncertain over.

Huh, not sure why I didn't say this when I first read this post, but there is a difference between Alice and Bob -- Alice will seek out information about her utility function, while Bob will not.

Certainly any value learning method will have to account for the fact that humans do not in fact know their own values, but it's not the case that such behavior is indistinguishable from behavior that maximizes a utility function.

There seems to be an assumption in this post that a value learner will be learning utility functions directly; and since utility functions are something which is associated with behavior, this framing leads to a focus on learning utility functions from behavior, and hence this post.

It seems to me that a value learner shouldn't try to learn any given individual's utility functions directly; rather it should first learn the psychological content corresponding to values, and then construct utility functions out of that. Among other positive features, this would allow a value learner to predict how the human would behave in a situation which the human hadn't been exposed to yet (or even one which was totally alien to the human's current conceptual landscape [LW · GW]).

Here's how I'd summarize my disagreement with the main claim: Alice is not acting rationally in your thought experiment if she acts like Bob (under some reasonable assumptions). In particular, she is doing pure exploitation and zero (value-)exploration by just maximizing her current weighted sum. For example, she should be reading philosophy papers.

There is one more fact that is ignored in IRL: that humans often have contradicting values. For example, if I want a cake very much, but also have a strong inclination for dieting, I will do nothing. So I have two values, which exactly compensate each other and have zero effects on behaviour. Observing only behaviour will not give a clue about them. More complex examples are possible, where contradicting values create inconsistent behaviour, and it is very typical for biological humans.