Updatelessness and Son of X

This does seem to be the "obvious" next step in the UDT approach. I proposed something similar as "UDT2" in a 2011 post to the "decision theory workshop" mailing list, and others have made similar proposals.

But there is a problem with having to choose how much time/computing resources to give to the initial decision process. If you give it too little then its logical probabilities might be very noisy and you could end up with a terrible decision, but if you give it too much then it could update on too many logical facts and lose on acausal bargaining problems. With multiple AI builders, UDT2 seems to imply a costly arms-race situation where each has an incentive to give their initial decision process less time (than would otherwise be optimal) so that their AI could commit faster (and hopefully be logically updated upon by other AIs) and also avoid updating on other AI's commitments.

I'd like to avoid this but don't know how. I'm also sympathetic to Nesov's (and others such as Gary Drescher's) sentiment that maybe there is a better approach to the problems that UDT is trying to solve, but I don't know what that is either.

Prior to working more on decision theory or thin priors, it seems worth clearly fleshing out the desiderata that a naive account (say a task/act-based AI that uses CDT) fails to satisfy.

You say: "[Son of X] is very opaque. There is an extra level of indirection, where you can't just directly reason about what agent X will do. Instead have to reason about what agent X will modify into, which gives you a new agent, which you probably understand much less than you understand agent X, and then you have to reason about what that new agent will do. Second, it is unmotivated. If you had a good reason to like Son of X, you would probably not be calling it Son of X"

But if we trust X's decisions, then shouldn't we trust its decision to replace itself with Son of X? And if we don't trust X's decisions, aren't we in trouble anyway? Why do we need to reason about Son of X, any more than we need to reason about the other ways in which the agent will self-modify, or even other decisions the agent will make?

I agree this makes thinking about Son of X unsatisfying, we should just be thinking about X. But I don't see why this makes building X problematic. I'm not sure if you are claiming it does, but other people seem to believe something like that, and I thought I'd respond here.

I agree that there is a certain perspective on which this situation is unsatisfying. And using a suboptimal decision theory / too thick a logical prior in the short term will certainly will involve some cost (this is a special case of my ongoing debate with Wei Dai about the urgency of philosophical problems). But it seems to me that it is way less pressing than other possible problems---e.g. how to do aligned search or aligned induction in the regime with limited unlabelled data.

These other problems: (a) seem like they kill us by default in a much stronger sense, (b) seem necessary on both MIRI agendas as well as my agenda, and I suspect are generally going to be necessary, (c) seem pretty approachable, (d) don't really seem to be made easier by progress on decision theory, logical induction, on the construction of a thin prior, etc.

(Actually I can see how a good thin prior would help with the induction problem, but it leads to a quite different perspective and a different set of desiderata.)

On the flip side, I think there is a reasonably good chance that problems like decision theory will be obsoleted by a better understanding of how to build task/act-based AI (and I feel like for the most part people haven't convincingly engaged with those arguments).

UDT, in its global policy form, is trying to solve two problems: (1) coordination between the instances of an agent faced with alternative environments; and (2) not losing interest in counterfactuals as soon as observations contradict them. I think that in practice, UDT is a wrong approach to problem (1), and the way in which it solves problem (2) obscures the nature of that problem.

Coordination, achieved with UDT, is like using identical agents to get cooperation in PD. Already in simple use cases we have different amounts of computational resources for instances of the UDT agent that could make the decision processes different, hence workarounds with keeping track of how much computation to give the decision processes, so that coordination doesn't break, or hierarchies of decision processes that can access more and more resources. Even worse, the instances could be working on different problems and don't need to coordinate at the level of computational resources needed to work on these problems. But we know that cooperation is possible in much greater generality, even between unrelated agents, and I think this is the right way of handling the differences between the instances of an agent.

It's useful to restate the problem of not ignoring counterfactuals, as a problem of preserving values. It's not quite reflective stability, as it's stability under external observations rather than reflection, but when an agent plans for future observations it can change itself to preserve its values when the observations happen (hence "Son of CDT" that one-boxes). One issue is that the resulting values are still not right, they ignore counterfactuals that are not in the future of where the self-modification took place, and it's even less clear how self-modification addresses computational uncertainty. So the problem is not just preserving values, but formulating them in the first place so that they can already talk about counterfactuals and computational resources.

I think that in the first approximation, the thing in common between instances of an agent (within a world, between alternative worlds, and at different times) should be a fixed definition of values, while the decision algorithms should be allowed to be different and to coordinate with each other as unrelated agents would. This requires an explanation of what kind of thing values are, their semantics, so that the same values (1) can be interpreted in unrelated situations to guide decisions, including worlds that don't have our physical laws, and by agents that don't know the physical laws of the situations they inhabit, but (2) retain valuation of all the other situations, which should in particular motivate acausal coordination as an instrumental drive. Each one of these points is relatively straightforward to address, but not together. I'm utterly confused about this problem, and I think it deserves more attention.

This is more or less what I was talking about here (see last paragraph). This should also give us superrationality, provided that instead of allowing an arbitrary "future version", we constrain the future version to be a limited agent with access to a powerful "oracle" for queries of the form $E[U\mid \pi ]$ for all possible policies $\pi$ (which might involve constructing another, even more powerful, agent). If we don't impose this constraint, we run into the problem of "self-stabilizing mutually detrimental blackmail" in multi-agent scenarios.