preferences:decision theory :: data:code

post by ArthurB · 2011-02-19T07:45:22.119Z · LW · GW · Legacy · 23 comments

 

I'd like to present a couple thoughts. While I am somewhat confident in my reasonning, my conclusions strongly contradict what I perceive (possibly incorrectly) to be the concensus around decision theory on LessWrong. This consensus has been formed by people who have spent more time than me thinking about it, and are more intelligent than I am. I am aware of that, this is strong evidence that I am mistaken or obvious. I believe nonetheless the argument I'm about to make is valuable and should be heard. 

It is argued that the key difference between Newcomb's problem and Solomon's problem is that precommitment is useful in the former and useless in the latter. I agree that the problems are indeed different, but I do not think that is the fundamental reason. The devil is in the details.

Solomon's problem states that

 - There is a gene that causes people to chew gum and to develop throat cancer
 - Chewing gum benefits everyone

It is generally claimed that EDT would decide not to chew gum, because doing so would place the agent in a state where its expected utility is reduced. This seems incorrect to me. The ambiguity is in what is meant by "causes people to chew gum". If the gene really causes people to chew gum, then that gene by definition affects that agent's decision theory, and the hypothesis that it is also following EDT is contradictory. What is generally meant is that having this gene induces a preference to chew gum, which is generally acted upon by whatever decision algorithm is used. An EDT agent must be fully aware of its own preferences, otherwise it could not calculate its own utility, therefore, the expected utility of chewing gum must be calculated conditional on having a preexisting or non preexisting taste for gum. In a nutshell, an EDT agent updates not on his action to chew gum, but on his desire to do so.

I've established here a distinction between preferences and decision theory. In fact, the two are interchangeable. It is always possible to hard code preferences in the decision theory, and vice versa. The distinction is very similar to the one drawn between code and data. It is an arbitrary but useful distinction. Intuitively, I believe hard coding preferences in the decision algorithm is poor design, though I do not have a clear argument why that is.

If we insist on preferences being part of the decision algorithm, the best decision algorithm for solomon's problem is the one that doesn't have a cancer causing gene. If the algorithm is EDT, then liking gum is a preference, and EDT makes the same decision as CDT.

Let's now look at Newcomb's problem. Omega's decision is clearly not based on a subjective preference for one box or two box (let's say an aesthetic preference for example). Omega's decision is based on our decision algorithm itself. This is the key difference between the two problems, and this is why precommitment works for Newcomb's and not Solomon's.

Solomon's problem is equivalent to this problem, which is not Newcomb's

- If Omega thinks you were born loving Beige, he puts $1,000 in box Beige and nothing in box Aquamarine.
- Otherwise, he puts $1,000 in box Beige and nothing in box Aquamarine.

In this problem, both CDT and EDT (correctly) two box. Again, this is because EDT knows that it loves beige.

Now the real Newcomb's problem. I argue that an EDT agent should integrate his own decision as evidence. 

 - If EDT's decision is to two-box, then Omega's prediction is that EDT two boxes and EDT should indeed two-box.
 - If EDT's decision is to one-box, then Omega's prediction is that EDT one box, and EDT should two-box. 

Since EDT reflects on his own decision, it can only be the only fixed point which is to two box.

Both CDT and EDT decide to chew gum and to two box.

If we're out shopping for decision algorithms (TDT, UDT...), we might as well shop for a set of preferences, since they can be interchangeable. It is clear that some preferences allow winning, when variable sum games are involved. This has been implemented by evolution as moral preferences, not as decision algorithms. One useful preference is the preference to keep one's word. Such a preference allows to pay Parfit's hitchiker without involving any preference reversal. Once you're safe, you do not try not to pay, because you genuinely prefer not breaking your promise than keeping the money. Yes, you could have preferences to two box, but there is no reason why you should catter in advance to crazy cosmic entities rewarding certain algorithms or preferences. Omega is no more likely than the TDT and UDT minimizer, evil entities known for torturing TDT and UDT practionners.

 

Edit: meant to write EDT two-boxes, which is the only fixed point.

 

23 comments

Comments sorted by top scores.