LDT (and everything else) can be irrational

post by Christopher King (christopher-king) · 2024-11-06T04:05:36.932Z · LW · GW · 1 comments

Contents

  Symmetrical Ultimatum Game
  You can't become $9 rock
  No agent is rational in every problem
  Implications
None
1 comment

you should not reject the 'offer' of a field that yields an 'unfair' amount of grain! - Ultimatum Game (Arbital)

 

In this post, I demonstrate a problem in which there is an agent that outperforms Logical Decision Theory, and show how for any agent you can construct a problem and competing agent that outperforms it. Defining rationality as winning, this means that no agent is rational in every problem.

Symmetrical Ultimatum Game

We consider a slight variation on the ultimatum game to make it completely symmetrical. The symmetrical ultimatum game is a two-player game in which each players says how much money they want. The amount if a positive integer number of dollars. If the sum is ≤$10, both players get the amount of money they choose. Otherwise, they both get nothing.

Now consider the decision problem of playing the symmetrical ultimatum game against a logical decision theorist.

A casual decision theorist does particularly poorly in this problem, since the LDT agent always chooses $9 leaving the casual decision theorist with $1.

How does a LDT agent fare? Well, logical decision theory is still a bit underspecified. However, notice that this question reduces to "how does a LDT agent do against a LDT agent in a symmetrical game?". Without knowing any details about LDT, we must conclude that the expected value is at most $5.

What about a rock with $9 painted on it? The LDT agent in the problem reasons that the best action is to choose $1, so the rock gets $9. 

Thus, $9 rock is more rational than LDT in this problem. □

You can't become $9 rock

Now, what makes this problem particularly difficult is how picky the LDT agent in the problem is. If based on the previous you decide to "become $9 rock", the LDT agent will defect against you. If based on the previous section you build a robot that always chooses $9, the LDT agent will defect against that robot. Only a truly natural $9 rock can win.

No agent is rational in every problem

Consider an agent X. There are two cases:

1. Against $9 rock, X always chooses $1. Consider the problem "symmetrical ultimatum game against X". By symmetry, X on average can get at most $5. But $9 rock always gets $9. So $9 rock is more rational than X.
2. Against $9 rock, X sometimes chooses more than $1 (thus getting nothing). Consider the problem "symmetrical ultimatum game against $9 rock". X on average gets less than $1. But an agent that always picks $1 (that is, a $1 rock) always gets $1. So $1 rock is more rational than X.

□

Implications

I still have an intuition that LDT is the "best" decision theory so far. See 
Integrity for consequentialists [EA · GW] for practical benefits of a LDT style of decision making.

However, there can be no theorem that LDT is always rational, since it isn't. And replacing LDT with a different agent can not fix the problem. Notice that, as a special case, humans can never be rational.

This seems to suggest some sort of reformulation of rationality is needed. For example, given LDT's reasonableness, one option is to violate the thesis of Newcomb's Problem and Regret of Rationality [LW · GW] and simply define rationality to be LDT.

1 comments

Comments sorted by top scores.