A Decision Theory Can Be Rational or Computable, but Not Both

post by StrivingForLegibility · 2023-12-21T21:02:45.366Z · LW · GW · 4 comments

Contents

4 comments

In classical game theory, rational agents always choose options which are optimal according to their preferences. Even when such a choice implies the ability to evaluate functions that are provably uncomputable. In other words, rationality implies hypercomputation.

We can reduce any rational agent into an oracle for any decision problem by asking it to choose between YES and NO, and giving the agent a higher payoff for choosing the correct answer. ^[1] If we choose the halting problem, a properly motivated rational agent will act as a halting oracle. ^[2] If we could look at the gears [LW · GW] of a rational agent, we'd be able to find some subsystem that was performing hypercomputation.

Any computable implementation of any decision theory will necessarily fail to choose rationally in some contexts. For any undecidable problem, it is provably impossible for any algorithm to choose the correct answer in all cases.

This begs the question: if a rational agent had to delegate their decision to a computer program, what sort of program would they choose?

1. ^{^}

   Or at least we could if rational agents weren't hypercomputational horrors from beyond spacetime. 

2. ^{^}

   Aligning the behavior of an infinitely intelligent alien creature, using externally supplied payoffs, is left as an exercise to the reader.

4 comments

Comments sorted by top scores.