Oh, I see. The reason my argument is wrong is because while for a specific $s_0,\text{prior},u$, the optimal policy is independent of the evaluator, you don't get to choose a separate policy for each $s_0,\text{prior},u$: you have to use the evaluator to distinguish which case you are in, and then specialize your policy to that case.

It looks closer to the Value Learning Agent in that paper to me and maybe can be considered an implementation / specific instance of that?

I think I intuitively agree but I also haven't checked it formally. But the point about no-deception seems to be similar to the point about observation-utility maximizers not wanting to wirehead. This agent also ends up learning which utility function is the right one, and in that sense is like the Value Learning agent.

Something that confuses me is that since the evaluator sees everything the agent sees/does, it's not clear how the agent can deceive the evaluator at all. Can someone provide an example in which the agent has an opportunity to deceive in some sense and declines to do that in the optimal policy?

(Copying a comment I just made elsewhere)

This setup still allows the agent to take actions that lead to observations that make the evaluator believe they are in a state that it assigns high utility to, if the agent identifies a few weird convictions the prior. That's what would happen if it were maximizing the sum of the rewards, if it had the same beliefs about how rewards were generated. But it's maximizing the utility of the true state, not the state that the evaluator believes they're in.

(Expanding on it)

So suppose the evaluator was human. The human's lifetime of observations in the past give it a posterior belief distribution which looks to the agent like a weird prior, with certain domains that involve oddly specific convictions. The agent could steer the world toward those domains, and steer towards observations that will make the evaluator believe they are in a state with very high utility. But it won't be particularly interested in this, and it might even be particularly disinterested, because the information it gets about what the evaluator values may less relevant to the actual states it finds itself in a position to navigate between, if the agent believes the evaluator believes they are in a different region of the state space. I can work on a toy example if that isn't satisfying.

ETA: One such "oddly specific conviction", e.g., might be the relative implausibility of being placed in a delusion box where all the observations are manufactured.

It looks closer to the Value Learning Agent in that paper to me and maybe can be considered an implementation / specific instance of that?

Yes. What the value learning agent doesn't specify is what constitutes observational evidence of the utility function, or in this notation, how to calculate $P_{s_0,\text{prior},u}^{\pi }$ and thereby calculate $w\left(u|h_{<t}\right)$. So this construction makes a choice about how to specify how the true utility function becomes manifest in the agent's observations. A number of simpler choices don't seem to work.

I may be missing something, but it looks to me like specifying an observation-utility maximizer requires writing down a correct utility function? We don't need to do that for this agent.

I was more looking for the simplest example of "no deception". My claim was that an observation-utility maximizer is not incentivized to deceive its utility function. But now I see what you meant by "deceive" so we can ignore that point.

I can't really do this toy-example-style, because the key feature is that the AI has a model of a deceived agent, and I can't see how spin up such a thing in an MDP with a few dozen states.

Luckily, most of the machinery of the setup isn't needed to illustrate this. Abstracting away some of the details, the agent is learning a function from strings of states to utility, which it observes in a roundabout way. I don't have a mathematical formulation of a function mapping state sequences to the real numbers that can be described by the phrase "the value of the reward that a certain human would provide upon observing the observations produced by the given sequence of states", but suffice it to say that this function exists. (Really we're dealing with distributions/stochastic functions, but that doesn't really change the fundamentals; it just makes it more cumbersome). While I can't give that function in simple mathematical form, hopefully it's a legible enough mathematical object.

If the evaluator has this utility function, she will always provide reward equal to the utility of the state, because even if she is uncertain about the state, this utility function only depends on the observations, which she has access to. (Again, the stochastic case is more complicated, but the conclusion is the same.) And indeed, if a human is playing the role of evaluator when this program is run, the rewards will mach the function in question, by definition. Therefore, no observed reward will contradict the belief that this function is the true utility function. Technically, the infimum of the posterior on this utility function is strictly positive with probability 1.

Sorry, this isn't really any more illustrative an example, but hopefully it's a clearer explanation.

Ok! That's very useful to know.

It seems pretty related to the Inverse Reward Design paper. I guess it's a variation. Your setup seems to be more specific about how the evaluator acts, but more general about the environment.