A Problem About Bargaining and Logical Uncertainty

To make this more relevant to real life, consider two humans negotiating over the goal system of an AI they're jointly building.

"To give a practical down to earth example, ..."

Is Omega even necessary to this problem?

I would consider transferring control to staply if and only if I were sure that staply would make the same decision were our positions reversed (in this way it's reminiscent of the prisoner's dilemma). If I were so convinced, then shouldn't I consider staply's argument even in a situation without Omega?

If staply is in fact using the same decision algorithms I am, then he shouldn't even have to voice the offer. I should arrive at the conclusion that he should control the universe as soon as I find out that it can produce more staples than paperclips, whether it's a revelation from Omega or the result of cosmological research.

My intuition rebels at this conclusion, but I think it's being misled by heuristics. A human could not convince me of this proposal, but that's because I can't know we share decision algorithms (i.e. that s/he would definitely do the same in my place).

This looks to me like a prisoner's dilemma problem where expected utility depends on a logical uncertainty. I think I would cooperate with prisoners who have different utility functions as long as they share my decision theory.

(Disclaimers: I have read most of the relevant LW posts on these topics, but have never jumped into discussion on them and claim no expertise. I would appreciate corrections if I misunderstand anything.)

(I'll review some motivations for decision theories in the context of Counterfactual Mugging, leading to the answer.)

Precommitment in the past, where it's allowed, was a CDT-style solution to problems like this. You'd try making the most general possible precommitment as far in the past as possible that would respond to any possible future observations. This had two severe problems: it's not always possible to be far enough in the past to make precommitments that would coordinate all relevant future events, and you have to plan every possible detail of future events in advance.

TDT partially resolves such problems by implementing coordinated decisions among the instances of the agent within agent's current worlds (permitted by observations so far) that share the same epistemic state (or its aspects relevant to the decision) and decide for all of themselves together, so arrive at the same decision. (It makes sense for the decision to be a strategy that then can take into account additional information differentiating the instances of the agent.) This is enough for Newcomb's problem and (some versions of) Prisoner's Dilemma, but where coordination of agents in mutually exclusive counterfactuals are concerned, some of the tools break down.

Counterfactual Mugging both concerns agents located in mutually exclusive counterfactuals, and explicitly forbids the agent to be present in the past to make a precommitment, so TDT fails to apply. In this case, UDT (not relying on causal graphs) can define a common decision problem shared by the agents from different counterfactuals, if these agents can be first reduced to a shared epistemic state, so that all of them would arrive at the same decision (which takes the form of a strategy), which is then given each agent's particular additional knowledge that differentiates it from the other agents within the group that makes the coordinated decision.

In the most general case, where we attempt to coordinate among all UDT agents, these agents arrive, without using any knowledge other than what can be generated by pure inference (assumed common among these agents), at a single global strategy that specifies the moves of all agents (depending on each agent's particular knowledge and observations). However, when applied to a simple situation like Counterfactual Mugging, an agent only needs to purge itself of one bit of knowledge (identifying an agent) and select a simple coordinated strategy (for both agents) that takes that bit back as input to produce a concrete action.

So this takes us the whole circle, from deciding in a moment, to deciding (on a precommitment) in advance, and to deciding (on a coordinated strategy) in the present (of each instance). However, the condition for producing a coordinated strategy in the present is different from that for producing a precommitment in the past: all we need is shared state of knowledge among the to-be-coordinated agents, and not the state of knowledge they could've shared in the past, if they were to attempt a precommitment.

So for this problem, in coordinating with the other player (which let's assume abstractly exists, even if with measure 0), you can use your knowledge of the millionth digit of pi, since both players share it. And using this shared knowledge, the strategy you both arrive at would favor the world that's permitted by that value, in this case the paperclip world, the other world doesn't matter, contrary to what would be the case with a coin toss instead of the accessible abstract fact. And since the other player has nothing of value to offer, you take the whole pie.

Here's an argument I made in a chat with Wei. (The problem is equivalent to Counterfactual Mugging with a logical coin, so I talk about that instead.)

1) A good decision theory should always do what it would have precommitted to doing.

2) Precommitment can be modeled as a decision problem where an AI is asked to write a successor AI.

3) Imagine the AI is asked to write a program P that will be faced with Counterfactual Mugging with a logical coin (e.g. parity of the millionth digit of pi). The resulting utility goes to the AI. The AI writing P doesn't have enough resources to compute the coin's outcome, but P is allowed to use as much resources as needed.

4) Writing P is equivalent to supplying only one bit: should P pay up if asked?

5) Supplying that bit is equivalent to accepting or declining the bet "win $10000 if the millionth digit of pi is even, lose $100 if it's odd".

6) So if your AI can make bets about the digits of pi (which means it represents logical uncertainty as probabilities), it should also pay up in Counterfactual Mugging with a logical coin, even if it already has enough resources to compute the coin's outcome. The AI's initlal state of logical uncertainty should be "frozen" into its utility function, just like all other kinds of uncertainty (the U in UDT means "updateless").

Maybe this argument only shows that representing logical uncertainty as probabilities is weird. Everyone is welcome to try and figure out a better way :-)

In the old counterfactual mugging problem, agents who precommit are trading utilities across possible worlds, each world having a utility-gain called a prior that expresses how much the agent wants its utilities to lie in those worlds instead of silly ones. From that perspective, it's true that nothing in reality will be different as a result of the agent's decision, just because of determinism, but the agent is still deciding what reality (across all possible worlds) will look like, just like in Newcomb's problem.

So when I read in Nesov's post that "Direct prediction of your actions can't include the part where you observe that the digit is even, because the digit is odd", what I'm really seeing is someone saying, "I give zero weight to possible worlds in which math doesn't work sensibly, and tiny weights to worlds in which math does work, but my confusion or the conspiring of a malicious / improbable / senseless / invaluable universe cause me to think it does not."

One of the reasons why I think possible worlds of the first kind (different causal / programmatic histories but the same underlying ontology-stuff) are valuable / real, is that we sort of know how to calculate their properties using causal networks or timeless networks or whatever kind of networks you get when you combine the not-quite specified mathematical machinery in TDT with UDT. Our ability to calculate their properties reifies them, opens them up to interacting with this world even more directly via simulation.

The next step seems to be to ask, "for agents that do care about those impossible possible worlds, how would they act?" If omega is choosing in a way that can be computed in our world, using our math (and somehow that other universe and our calculations don't explode when it gets to the contradiction (or it does! I suppose you can care about worlds where math explodes, even if I can't visualize them)), then we can simulate his reasoning in all respects save the identify of the logical fact in question, and use that to calculate which behaviour maximizes the utility across possible worlds via their dependence on our decision.

So in the example problem, if a valuer of contradictory worlds has roughly equal priors for both the world we're examining and the other world in which she find herself where the digit was even (the impossible one, which isn't impossible for her, because it wasn't assigned zero prior weight), then sure, she can go ahead and give up control. That's of course assuming that she has an expectation the staple maximizer will reciprocate in the impossible world, which you didn't spell out in your post, but that dependence on decisions is standard for counterfactual mugging problems. Please correct me if that's not the intended setup.

As as aside, this comment feels silly and wrong; an example of diseased thoughts unconnected with reality. It reminds me a bit of Greg Egan's short story Dark Integers. I would really love to see a more sensible interpretation that this.

That is a really, really weird dilemma to be in.

By the way, you can abbreviate paperclip/staple maximizer as clippy/staply (uncapitalized).

What deal would you have made if you didn't know that the first digit of pi was odd, or if you didn't know that 1+1=2?

It seems, the important factor is how does Omega makes its choice of which digit of pi (or other logical fact) to check. If Omega uses a quantum coin to select a number N between 1 and 84, and then checks the N-th digit of pi, then you should cooperate. If Omega searches through a roundish number N such that N-th digit of pi is odd, then the answer appears to further depend on Omega's motivation. If Omega made the choice of "seeking for 'odd' roundish numbers" vs. "'even' roundish numbers" by tossing a quantum coin, then you should cooperate. Otherwise... etc.

By the way, I notice that the thought experiment, as phrased, doesn't quite require the knowledge of the digit of pi, if Omega indeed states that the situation is predicated on its oddness, it even states that it's odd, and there is no counterfactual version of yourself, so computation of the digit becomes a ritual of cognition without purpose. There is likely a restatement that retains everyone's interpretation, but in the current phrasing the role of the logical uncertainty in it seems to be lost.

Do I know that Staply would have decided as I decide, had he been given control of the universe and been told by Omaga and his calculator that the millionth digit of pi is even?

Just thinking out loud, no complete solution yet.

1) Maybe different approximations to logical omniscience work well on different problems. It seems natural to guess that approximations work better overall as they get closer to logical omniscience. We already have a tentative decision theory that works assuming logical omniscience. Can we invent a decision theory that works assuming complete logical ignorance in some sense?

2) We could look at the moment when you decide to make the most general precommitment or self-modification for the future, while still having some logical uncertainty. If you know the first digit of pi but not the millionth, you should precommit accordingly. The first such moment seems to correspond to the AI programmer's state of logical uncertainty, so if you're unsure about the millionth digit of pi, then it's okay for your AI to be unsure.

Difficulty level 2: replace "millionth digit of pi" with "0.5*10^20".