I agree with the sentiment that there are philosophical difficulties that AI needs to take into account, but that very likely take far too long to formulate. Simpler kinds of indirect normativity that involve prediction of uploads allow delaying that work to after AI.

So this issue doesn't block all actionable work, as its straightforward form would suggest. There might be no need for the activities to be in this order in physical time. Instead it motivates work on the simpler kinds of indirect normativity that would allow such philosophical investigations to take place inside AI's values. In particular, it motivates figuring out what kind of thing AI's values are, in sufficient generality so that it would be able to represent the results of unexpected future philosophical progress.

If we could model humans as having well-defined values but irrational in predictable ways (e.g., due to computational constraints or having a limited repertoire of heuristics), then some variant of CIRL might be sufficient (along with solving certain other technical problems such as corrigibility and preventing bugs) for creating aligned AIs. I was (and still am) worried that some researchers think this is actually true, or by not mentioning further difficulties, give the wrong impression to policymakers and other researchers.

If you are already aware of the philosophical/metaphilosophical problems mentioned here, and have an approach that you think can work despite them, then it's not my intention to dampen your enthusiasm. We may differ on how much expected value we think your approach can deliver, but I don't really know another approach that you can more productively spend your time on.

In order to do value learning we need to specify what the AI is supposed to infer from some observations. The usual approach is to specify how the observations depend on the human's preferences, and then have the AI do bayesian updating. If we are already in the business of explicitly specifying a causal model that links latent preferences to observations, we will presumably specify a model where latent preferences are upstream of observations and not downstream of the AI's actions.

At some points it seems like you are expressing concerns about model misspecification, but I don't see how this would cause the problem either.

For example, suppose that I incorrectly specify a model where the human is perfectly reliable, such that if at any time they say they like death, then they really do. And suppose that the AI can easily intervene to cause the human to say they like death. You seem to imply that the AI would take the action to cause the human to say they like death, if death is easier to achieve. But I don't yet see why this would happen.

If the AI updates from the human saying that they like death, then it's because the AI doesn't recognize the impact of its own actions on the human's utterances. And if the AI doesn't recognize the impact of its own action on the human's utterances, then it won't bother to change its actions in order to influence the human's utterances.

I don't see any in-between regime where the AI will engage in this kind of manipulation, even if the model is completely misspecified. That is, I literally cannot construct any Bayesian agent that exhibits this behavior.

It seems like the only way it can appear is if we either (1) directly specify how the AI ought to update on observations, rather than specifying a model, or (2) specify a model in which the user's preferences are causally downstream of the AI's actions. But neither of those seems like things we would do.

because I haven’t seen a decent Bayesian approach proposed.

In some sense I agree with this. Specifying a model of how observations relate to preferences is very difficult! But both IRL and your writing seem to take as given such a model, and people who work on IRL in fact believe that we'll be able to construct good-enough models. So if you are objecting to this leg of the proposal, that would be a much more direct criticism of IRL on its own terms. (And this is what I meant by saying "give up on the Bayesian approach.")

For example, if you assume "Anything humans say about their preferences is true," that's basically giving up on the Bayesian approach as usually imagined (which would be to directly specify a model that relates preferences to utterances, and then to update on utterances) and replacing it with an ad-hoc algorithm for making inferences from human utterances (namely, accept them at face value). In the usual Bayesian setting, "humans are perfectly reliable" corresponds to believing that human utterances correctly track (fixed) human preferences, i.e. believing that it is impossible to influence those utterances.

Ok, here's my reconstruction of what model you're using:

The AI will take an action $A_{AI}$; then the human will take action $A_H$; then a world state $W$ will result.

The human utility function $U$ is taken to be a function from $W$ to $R$.

The AI has a "prior" over utility functions, $Q\left(U\right)$, and a "rationality model", $Q(A_H|U,A_{AI})$, saying what action a human would take given that they have a given utility function and given the action the AI took (in this case, let's say the human directly observes the AI's action). For example, $Q$ could say that the human takes the optimal action 80% of the time and otherwise takes a suboptimal action.

Separately, the AI has a predictive model $P\left(A_H|A_{AI}\right)$, saying what action the human is actually going to take. The predictive model is "good" is the sense that, for example, it should predict that the human will say they want heroin if they are given it. The AI also has a predictive model for the world $P(W|A_{AI},A_H)$.

Notably, $Q$ is inconsistent with $P$. If we define $Q\left(A_H|A_{AI}\right):={\sum }_{U}Q\left(U\right)Q(A_H|U,A_{AI})$, then in general $P\left(A_H|A_{AI}\right)\ne Q\left(A_H|A_{AI}\right)$. For example, $P$ correctly predicts that the human will say they like heroin if the AI administers it; while $Q$ says that the human will probably not (since they're correct 80% of the time, and dislike heroin with 60% probability).

The AI's "estimate" for the human's utility function, given action $A_H$, is

$$Q(U|A_H,A_{AI}):=\frac{Q\left(U\right)Q(A_H|U,A_{AI})}{{\sum }_{U^{\prime }}Q\left(U\right)Q(A_H|U,A_{AI})}$$.

The AI's objective is to optimize its "estimate" of expected utility:

$$U_{AI}:=\sum _{U}Q(U|A_H,A_{AI})U\left(W\right)$$

And it scores actions by taking the expectation of this, using $P$:

$$score\left(A_{AI}\right):=E_P\left[U_{AI}|A_{AI}\right]:=\sum _{A_H}P\left(A_H|A_{AI}\right)\sum _{W}P(W|A_H,A_{AI})\sum _{U}Q(U|A_H,A_{AI})U\left(W\right)$$

Correct me if I'm wrong about the model you're using.

If this is the model you're using, then it is (a) inconsistent with the IRL/CIRL literature, and (b) looks really weird (as Paul points out). In this case you should be clear that you're not criticizing IRL/CIRL, you're criticizing a different model which, as far as I know, no one has advocated as a good way of learning human values.