Should VS Would and Newcomb's Paradox

Yes! It's interesting how the concepts of agency and choice seem so natural and ingrained for us, humans, that we are often tempted to think that they describe reality deeper than they really do. We seem to see agents, preferences, goals, and utilities everywhere, but what if these concepts are not particularly relevant for the actual mechanism of decision-making even from the first-person view?

What if much of the feeling of choice and agency is actually a social adaptation, a storytelling and explanatory device that allows us to communicate and cooperate with other humans (and, perhaps more peculiarly, with our future selves)? While it feels like we are making a choice due to reasons, there are numerous experiments that point to the explanatory, after-the-fact role of reasoning in decision-making. Yes, abstract reasoning also allows us to model conceptually a great many things, but those models just serve as additional data inputs to the true hidden mechanism of decision-making.

It shouldn't be surprising then if for other minds [such as AGI], not having this primal adaptation to being modeled by others, decision-making would feel nothing like choice or agency. We already see it in our simpler AIs -- choice and reasoning mean nothing to a health diagnostic system, it simply computes, it is only for us, humans, to feel like we understand "why" it made a particular choice, we have to add an explanatory module that gives us "reasons" but is completely unnecessary for the decision-making itself!

I talked about it a few years back [LW · GW]. If you think of the world as a deterministic or non-deterministic evolution of some initial conditions, you can potentially separate small parts of it as "agents" and study what they would internally consider a better or worse outcome, which is the "should", vs what they actually do, which is the "would". You don't have to internally run a God's eye view algorithm, some agents can still notice themselves "make decisions" (the internal feeling as an artifact of the agency algorithm evolving in time), while understanding that this is only a feeling, and the reality is nothing more than learning what kind of an agent you are, for example whether you one-box or two-box. Or maybe it's what you mean by an outside view.

Notice, however, that the view you have re-discovered is anti-memetic: it contradicts the extremely strong "free choice" output of a subroutine that converts multiple potential maps of the observed world into an action that has a certain degree of optimality, and so is almost instantly internally rejected in most cases. In fact, most agents pretend to find a loophole, often under the guise of compatibilism, that lets them claim that choices matter and that they can affect the outcome by making them, not just passively watch themselves think and act and discover what they would actually do.

There is a problem, but it is not quite an inconsistency. The problem is the assumption that Omega is a perfect predictor. That is: for the system Omega + Agent + everything else, Omega can always find a fixed point such that Omega's prediction is always correct for the state of the system some time after making that prediction and subsequent evolution of the system based on that. Even in a completely deterministic universe, this is asking too much. Some systems just don't have fixed points.

The problem becomes much more reasonable when you consider an almost perfect predictor. It ruins the purity of the original question, but the space of possibilities becomes much broader.

It becomes more like a bluffing game. Omega can observe tells with superhuman acuity, you don't know what those tells are, and certainly don't know how to mask them. If your psychology was such that you'd take both boxes, you were already showing signs of it before you went into the room. Note that this is not supernatural, retro-causal, nor full of anthropic paradoxes but something people already do. Omega is just very much better at it than other humans are.

In this viewpoint, the obvious solution is to be so good at hiding your tells that Omega thinks you're a one-boxer while actually being a two-boxer. But you don't know how to do that, nobody else who tried succeeded, and you have to think you've got a 99.9% chance of success for it to be more worthwhile than obviously taking one box.

From this first-person perspective, we consider all possible actions, imagine their respective outcomes, then choose the one deemed optimal according to some objective.

vs.

When the two boxes with predetermined content are right in front of me, obviously I should take them both.

These seem to conflict. What do you expect the outcome to be if you take one box? What if you take two?