What Program Are You?

Methodological remark: One should write at some point on a very debilitating effect that I've noticed in decision theory, philosophy generally, and Artificial Intelligence, which one might call Complete Theory Bias. This is the academic version of Need for Closure, the desire to have a complete theory with all the loose ends sewn up for the sake of appearing finished and elegant. When you're trying to eat Big Confusing Problems, like anything having to do with AI, then Complete Theory Bias torpedoes your ability to get work done by preventing you from navigating the space of partial solutions in which you can clearly say what you're trying to solve or not solve at a given time.

This is very much on display in classical causal decision theory; if you look at Joyce's Foundations of Causal Decision Theory, for example, it has the entire counterfactual distribution falling as mana from heaven. This is partially excusable because Pearl's book on how to compute counterfactual distributions had only been published, and hence only really started to be popularized, one year earlier. But even so, the book (and any other causal decision theories that did the same thing) should have carried a big sign saying, "This counterfactual distribution, where all the interesting work of the theory gets carried out, falls on it as manna from heaven - though we do consider it obvious that a correct counterfactual for Newcomb ought to say that if-counterfactual you one-box, it has no effect on box B." But this would actually get less credit in academia, if I understand the real rules of academia correctly. You do not earn humility points for acknowledging a problem unless it is a convention of the field to acknowledge that particular problem - otherwise you're just being a bother, and upsetting the comfortable pretense that nothing is wrong.

Marcello and I have all sorts of tricks for avoiding this when we navigate the space of fragmentary solutions in our own work, such as calling things "magic" to make sure we remember we don't understand them.

TDT is very much a partial solution, a solution-fragment rather than anything complete. After all, if you had the complete decision process, you could run it as an AI, and I'd be coding it up right now.

TDT does say that you ought to use Pearl's formalism for computing counterfactuals, which is progress over classical causal decision theory; but it doesn't say how you get the specific causal graph... since factoring the causal environment is a very open and very large AI problem.

Just like the entire problem of factoring the environment into a causal graph, there's a whole entire problem of reasoning under logical uncertainty using limited computing power. Which is another huge unsolved open problem of AI. Human mathematicians had this whole elaborate way of believing that the Taniyama Conjecture implied Fermat's Last Theorem at a time when they didn't know whether the Taniyama Conjecture was true or false; and we seem to treat this sort of implication in a rather different way than "2=1 implies FLT", even though the material implication is equally valid.

TDT assumes there's a magic module bolted on that does reasoning over impossible possible worlds. TDT requires this magic module to behave in certain ways. For the most part, my methodology is to show that the magic module has to behave this way anyway in order to get commonsense logical reasoning done - i.e., TDT is nothing special, even though the whole business of reasoning over impossible possible worlds is an unsolved problem.

To answer Robin's particular objection, what we want to do is drop out of TDT and show that an analogous class of reasoning problems apply to, say, pocket calculators. Let's say I know the transistor diagram for a pocket calculator. I type in 3 + 3, not knowing the answer; and upon the screen flashes the LED structure for "6". I can interpret this as meaning 3 + 3 = 6, or I can interpret it as a fact about the output of this sort of transistor diagram, or I can interpret it as saying that 3 + 3 is an even number, or that 2 3 is 6. And these may all tell me different things, at first, about the output of another, similar calculator. But all these different interpretations should generally give me compatible logical deductions about the other calculator and the rest of the universe. If I arrive at contradictory* implications by forming different abstractions about the calculator, then my magic logic module must not be sound.

The idea that you want to regard "all computations similar to yourself as having the same output" is just a gloss on the real structure. In the real version, there's a single canonical mathematical fact of which you are presently logically uncertain, the output of the Godelian diagonal:

Argmax[A in Actions] in Sum[O in Outcomes](Utility(O)P(*this computation yields A []-> O|rest of universe))

The this computation above is not a reference to your entire brain. It is a reference to that one equation above, the canonical diagonal form. It's assumed, in TDT, that you're implementing that particular equation - that TDT is how you make your decisions.

Then you assume that particular equation has a particular output, and update your view of the rest of the physical universe accordingly. In "almost" the same way you would update your view of the universe when you saw the calculator output "6". It might indeed depend on your logical reasoning engine. There might be things similar to yourself that you did not know were similar to yourself. If so, then you'll (all) do worse, because your logical reasoning engine is weaker. But you should at least not arrive at a contradiction, if your logical reasoning engine is at least sound.

What if you can only approximate that equation instead of computing it directly, so that it's possible that you and the equation will have different outputs? Should the equation be about your approximation of it, or should you just try to approximate the original equation? This is an open problem in TDT, which reflects the underlying open problem in AI; I just assumed there was enough computing power to do the above finite well-ordered computation directly. If you could show me a particular approximation, I might be able to answer better. Or someone could deliver a decisive argument for why any approximation ought to be treated a particular way, and that would make the problem less open in TDT, even though which approximation to use would still be open in AI.

(I also note at this point that the only way your counterfactual can apparently control the laws of physics, is if you know that the laws of physics imply that at least one answer is not compatible with physics, in which case you already know that option is not the output of the TDT computation, in which case you know it is not the best thing to do, in which case you are done considering it. So long as all answers seem not-visibly-incompatible with physics relative to your current state of logical knowledge, supposing a particular output should not tell you anything about physics.)

An example of a much more unsolved problem within TDT, which is harder to dispose of by appeal to normal non-TDT logical reasoning, is something that I only realized existed after reading Drescher; you actually can't update on the subjunctive / counterfactual output of TDT in exactly the same way you can update on the actually observed output of a calculator. In particular, if you actually observed something isomorphic to your decision mechanism output action A2, you could infer that A2 had higher expected utility than A1, including any background facts about the world or one's beliefs about it, that this would require; but if we only suppose that the mechanism is outputting A2, we don't want to presume we've just calculated that A2 > A1, but we do want to suppose that other decision mechanisms will output A2.

The two ways that have occurred to me for resolving this situation would be to (1) stratify the deductions into the physical and the logical, so that we can deduce within the counterfactual that other physical mechanisms will output "A2", but not deduce within our own logic internal to the decision process that A2 > A1. Or (2) to introduce something akin to a causal order within logical deductions, so that "A2 > A1" is a parent of "output = A2" and we can perform counterfactual surgery on "output = A2" without affecting the parent node.

Does it bother anyone else that the world doesn't even decompose uniquely into physical objects?

Robin, until we solve this problem (and I do agree that you've identified a problem that needs to be solved), is there anything wrong with taking the decomposition of an agent into program and data as an external input to the decision theory, much like how priors and utility functions are external inputs to evidential decision theory, and causal relationships are an additional input to causal decision theory?

It seems that in most decision problems there are intuitively obvious decompositions, even if we can't yet formalize the criteria that we use to to do this, so this doesn't seem to pose a practical problem as far as using TDT/UDT to make everyday decisions. Do you have an example where the decomposition is not intuitively obvious?

The advice to "choose as though controlling the logical output of the abstract computation you implement" might have you choose as if you controlled the actions of all physical objects, if you viewed the laws of physics as your program, or choose as if you only controlled the actions of the particular physical state that you are, if every distinct physical state is a different program.

This seems to be about identity. First case being where one thinks that they are the Universe itself, and latter being complete denialist of time passing in any sense at all, and our experience being non-connected solid state, thus refusing to accept that he is in any meaningful sense the guy he was 5 minutes earlier. I don't think it's wrong that this kind of differences in views of self-identity should change our decisions.

Don't these kinds of considerations apply to any decision theory? Don't they all suppose that you're given some kind of carving-up of the world into various things with various manipulable properties? Don't they all suppose that you have some kind of identity criteria for saying when things are "the same", and for partitioning up events to assign payoffs to them? Is any decision theory responsible for dictating what your initial carving-up of the world should be?

I think that TDT and UDT assume that the agent, for whatever reason, starts out with a given decomposition of itself into program and data. If it had started with a different decomposition, it would have been a different agent, and so, unsurprisingly, might have made different decisions.

To reason sensibly under abnormal conditions like mind copying or dealing with perfect predictors, we first separate the things that we control and which can be kept constant through all the situations in the problem (the program) from the things which we can't control or which vary in the course of the problem (the state/data).

Which things we want to hold constant and which things vary depend on the problem we're considering. If we're trying to choose a strategy for a game - the Iterated Prisoner's Dilemma, for example - then the program is a complete strategy which we assume is memorized before the beginning and followed perfectly, and the state is all the observations made between the start of the game and some decision point within it. Problems may force us to move things that are normally part of the program into the state, by taking them out of our control. For example, when reasoning about how a company should act in relation to a market, we treat everything that decides what the corporation does as a black box program, and the observations it makes of the market as its input data. If internal politics matter, then we have to narrow the black-boxing boundary to only ourselves. If we're worried about procrastination, then we draw the boundary inside our own mind.

Whether something is Program or Data is not a property of the object itself, but rather of how we reason about it. If it can be fully modeled as a black box function, then it's part of the program; otherwise it's data.

While I agree that humans can be decomposed into a number of program/data systems, I think there a decomposition that preserves our 'self's.

This composition is where our conscious minds is the program and our memories and sensory input is the data.

In other words, we can be copied into another substrate that has different program/data layers as long as this is maintained we will still think that our 'self' is preserved.

Note that I am not suggesting that we have a preference for preserving the 'self' in this fashion, but that we will have a subjective experience of continuity.

Disposition-Based Decision Theory has the feature that it requires a conceptual split between an initial state and subsequent activity as well. However, exactly where the initial state snapshot is taken turn out not to be critical - provided it is "early enough" - and the paper introducing DBDT describes what "early enough" means.