Posts
Comments
The normal VNM approach is to start with an agent whose behavior satisfies some common sense conditions: can't be money pumped and so on.
Nitpicks: (1) the vNM theorem is about preference, not choice and behavior; and (2) "can't be money pumped" is not one of the conditions in the theorem.
and Silvia's work
Typo: it's Sylvia.
I wrote "I'm really not sure at this point whether UDT is even on the right track" in UDT shows that decision theory is more puzzling than ever which I think you've read? Did you perhaps miss that part?
Yes, missed or forgot about that sentence, sorry.
(BTW this issue/doubt about whether UDT / paying CM is normative for humans is item 1 in the above linked post. Thought I'd point that out since it may not be obvious at first glance.)
Thanks.
Do you have more examples where making such distinctions would be helpful?
I was mostly thinking about discussions surrounding what the "correct" decision theory, is whether you should pay in CM, and so on.
Here's a related idea that is maybe clearer: Suppose an agent has the ability to self-modify to use any decision theory, would they decide to stick with their current decision theory? (I'm actually not sure what term has been explicitly defined to mean this, so I'll just call it "self-endorsement" for now.)
This sounds similar to what's called "self-recommendation"—see e.g. Skyrms (1982, pp. 707-709), Meacham (2010, §3.3) and Pettigrew (2023). In the abstract Pettigrew writes: "A decision theory is self-recommending if, when you ask it which decision theory you should use, it considers itself to be among the permissible options.".
I have actually been thinking about ways of extending Pettigrew's work to theories of dynamic choice. That is: is sophistication/resoluteness self-recommending? I don't think it is immediately clear what the answers are, and it might depend on the interpretations of sophistication and resoluteness one adopts, but yeah, I do agree that it seems like sophistication might be self-undermining.
Thanks for the clarification!
I do understand from the SEP, like Wei, that sophisticated means "backwards planning", and resolute means "being able to commit to a policy" (correct me if I'm wrong).
That seems roughly correct, but note that there are different interpretations of resolute choice floating around[1], and I think McClennen's (1990) presentation is somewhat unclear at times. Sometimes resoluteness seems to be about the ability to make internal commitments, and other times it seems to be about being sensitive to the dynamic context in a particular way, and I think these can come apart. You might be interested in these notes I wrote while reading McClennen's book.
My usage of "dynamic instability" (which might be contrary to academic usage) was indeed what Wei mentions: "not needing commitments". Or equivalently, I say a decision theory is dynamically stable if itself and its resolute version always act the same.
Then that sounds a bit question-begging. Do you think dynamic instability is a problem (normatively speaking)?
- ^
See e.g. Gauthier (1997) and Buchak (2013, §6).
I think Sami's comment is entirely fair given the language and framing of the original post. It is of course fine to forget about references, but e.g. "I find it curious that none of my ideas have a following in academia or have been reinvented/rediscovered by academia" and "Clearly academia has some blind spots, but how big?" reads like you don't consider it a possilbity that you might have re-invented something yourself, and that academics are at fault for not taking up your ideas.
I don't think cohesive decision theory is being discussed much, but I'm not sure. Perhaps because the theory is mainly used to argue against the claim that "every decision rule will lead agents who can’t bind themselves to disaster" (p. 20, footnote 34) in the paper, and discussion of its independent interest is relegated to a footnote (footnote 34).
It would be interesting to get an overview of what these are. Or if that's too hard to write down, and there are no ready references, what are your own interests in decision theory?
Yeah, that would be too hard. You might want to look at these SEP entries: Decision Theory, Normative Theories of Rational Choice: Expected Utility, Normative Theories of Rational Choice: Rivals to Expected Utility and Causal Decision Theory. To give an example of what I'm interested in, I think it is really important to take into account unawareness and awareness growth (see §5.3 of the first entry listed above) when thinking about how ordinary agents should make decisions. (Also see this post.)
I'm not sure I wouldn't pay either. I see it as more of an interesting puzzle than having a definitive answer. ETA: Although I'm more certain that we should build AIs that do pay. Is that also unclear to you? (If so why might we not want to build such AIs?)
Okay, interesting! I thought UDT was meant to pay in CM, and that you were convinced of (some version of) UDT.
On the point about AI (not directly responding to your question, to which I don't have an answer): I think it's really important to be clear about whether we are discussing normative, constructive or descriptive decision theory (using Elliott Thornley's distinction here). For example, the answers to "is updatelessness normatively compelling?", "should we build an updateless AI?" and "will some agents (e.g. advanced AIs) commit to being updateless?" will most likely come apart (it seems to me). And I think that discussions on LW about decision theory are often muddled due to not making clear what is being discussed.
There are many many interesting questions in decision theory, and "dimensions" along which decision theories can vary, not just the three usually discussed on LessWrong. It's not clear to me why (i) philosophers should focus on the dimensions you primarily seem to be interested in, and (ii) what is so special about the particular combination you mention (is there some interesting interaction I don't know about maybe?). Furthermore, note that most philosophers probably do not share your intuitions: I'm pretty sure most of them would e.g. pay in counterfactual mugging. (And I have not seen a good case for why it would be rational to pay.) I don't mean to be snarky, but you could just be wrong about what the open problems are.
I haven't looked into academic DT literature in years, so you're probably more familiar with it. Do you know if they're puzzled/confused by the same problems that we are?
I wouldn't say so, no. But I'm not entirely sure if I understand what the open problems are. Reading your list of seven issues, I either (i) don't understand what you are asking, (ii) disagree with the framing/think the question is misguided, or (iii) think there is an obvious answer (which makes me think that I'm missing something). With that said, I haven't read all the posts you reference, so perhaps I should read those first.
This is indeed what happens to the best-known decision theories (CDT and EDT): they want to commit to paying, but if they don’t, by the time they get to the Heads world they don’t pay. We call this dynamic instability, because different (temporal) versions of the agent seem to be working against each other.
Unless you are using "dynamic stability" to mean something other than "dynamic consistency", I don't think this is quite right. The standard philosophical theory of dynamic choice, sophisticated choice (see e.g. the SEP entry on decision theory), would not pay but is still dynamically consistent.
The reason for the former is that I (and others) have been unable to find a rigorous formulation of it that doesn't have serious open problems. (I and I guess other decision theory researchers in this community currently think that UDT is more of a relatively promising direction to explore, rather than a good decision theory per se.)
That's fair. But what is it then that you expect academics to engage with? How would you describe this research direction, and why do you think it's interesting and/or important?
Could you perhaps say something about what a Kripkean semantics would look like for your logic?
On your first point: as Sami writes, resolute choice is mentioned in the introductory SEP article on dynamic choice (it even has its own section!), as well as in the SEP article on decision theory. And SEP is the first place you go when you want to learn about philosophical topics and find references.
On your second point: as I wrote in my comment above, (i) academics have produced seemingly similar ideas to e.g. updatelessness (well before they were written up on LW) so it is unclear why academics should engage with less rigorous, unpublished proposals that appear to be similar (in other words, I don't think the phrase "blind spots" is warranted), and (ii) when academics have commented on or engaged with LW DT ideas, they have to my knowledge largely been critical (e.g. see the post by Wolfgang Schwarz I linked above, as well as the quote from Greaves)[1].
- ^
Cheating Death in Damascus getting published in the Journal of Philosophy is a notable exception though!
I think the main reason why UDT is not discussed in academia is that it is not a sufficiently rigorous proposal, as well as there not being a published paper on it. Hilary Greaves says the following in this 80k episode:
Then as many of your listeners will know, in the space of AI research, people have been throwing around terms like ‘functional decision theory’ and ‘timeless decision theory’ and ‘updateless decision theory’. I think it’s a lot less clear exactly what these putative alternatives are supposed to be. The literature on those kinds of decision theories hasn’t been written up with the level of precision and rigor that characterizes the discussion of causal and evidential decision theory. So it’s a little bit unclear, at least to my likes, whether there’s genuinely a competitor to decision theory on the table there, or just some intriguing ideas that might one day in the future lead to a rigorous alternative.
I also think it is unclear to what extent UDT and updateless are different from existing ideas in academia that are prima facie similar, like McClennen's (1990) resolute choice and Meacham's (2010, §4.2) cohesive decision theory.[1] Resolute choice in particular has been discussed in a lot of detail, and for a long time (see the citations of McClennen's book). (And, FWIW, my sense is that most philosophers think that resolute choice is irrational and/or doesn't make sense, at least if it is cashed out as a choice rule.)
It also doesn't help that it is unclear what the difference between FDT and UDT is supposed to be.
(If UDT is supposed to be an LDT of some sort, then you might want to check out Spohn's (2012)[2] version of CDT, Fisher's (n.d) disposition-based decision theory, and Poellinger's (2013) discussion of Spohn's theory, for ideas in academia that are similar to the LDT-part of the theory. And then there is also Schwarz' critique of FDT, which would then also apply to UDT, at least partially.)
- ^
My own take, using the terminology listed here, is that the causalist version of Meacham's cohesive decision theory is basically "updateless CDT", that the evidentialist version is basically "updateless EDT", and that a Spohn-CDT version of cohesive decision theory is basically "U(C)DT/F(C)DT". I also think that resolute choice is much more permissive than e.g. cohesive decision theory and updatelessness. As a choice rule, it doesn't recommend anything close to "maximizing EU relative to your prior". Instead, it just states that (i) how you act ex ante in a dynamic choice problem should be the same as you would act in the normalised version of the problem, and (ii) you should be dynamically consistent (i.e., the most preferred plan should not change throughout the decision problem).
- ^
Note that in the published article, it says that the article was received in 2008.
Thanks.
I am pretty sure they're interchangeable however.
Do you have a reference for this? Or perhaps there is a quick proof that could convince me?
You might also find the following cases interesting (with self-locating uncertainty as an additional dimension), from this post.
Sleeping Newcomb-1. Some researchers, led by the infamous superintelligence Omega, are going to put you to sleep. During the two days that your sleep will last, they will briefly wake you up either once or twice, depending on the toss of a biased coin (Heads: once; Tails: twice). After each waking, they will put you back to sleep with a drug that makes you forget that waking. The weight of the coin is determined by what the superintelligence predicts that you would say when you are awakened and asked to what degree ought you believe that the outcome of the coin toss is Heads. Specifically, if the superintelligence predicted that you would have a degree of belief in Heads, then they will have weighted the coin such that the 'objective chance' of Heads is . So, when you are awakened, to what degree ought you believe that the outcome of the coin toss is Heads?
Sleeping Newcomb-2. Some researchers, led by the superintelligence Omega, are going to put you to sleep. During the two days that your sleep will last, they will briefly wake you up either once or twice, depending on the toss of a biased coin (Heads: once; Tails: twice). After each waking, they will put you back to sleep with a drug that makes you forget that waking. The weight of the coin is determined by what the superintelligence predicts your response would be when you are awakened and asked to what degree you ought to believe that the outcome of the coin toss is Heads. Specifically, if Omega predicted that you would have a degree of belief in Heads, then they will have weighted the coin such that the 'objective chance' of Heads is . Then: when you are in fact awakened, to what degree ought you believe that the outcome of the coin toss is Heads?
Epistemic Constraint: The probability distribution which the agent settles on cannot be self-refuting according to the beliefs. It must be a fixed point of : a such that .
Minor: there might be cases in which there is a fixed point , but where the agent doesn't literally converge or deliberate their way to it, right? (Because you are only looking for to satisfy the conditions of Brouwer/Kakutani, and not, say, Banach, right?) In other words, it might not always be accurate to say that the agent "settles on ". EDIT: oh, maybe you are just using "settles on" in the colloquial way.
A common trope is for magic to work only when you believe in it. For example, in Harry Potter, you can only get to the magical train platform 9 3/4 if you believe that you can pass through the wall to get there.
Are you familiar with Greaves' (2013) epistemic decision theory? These types of cases are precisely the ones she considers, although she is entirely focused on the epistemic side of things. For example (p. 916):
Leap. Bob stands on the brink of a chasm, summoning up the courage to try and leap across it. Confidence helps him in such situations: specifically, for any value of between and , if Bob attempted to leap across the chasm while having degree of belief that he would succeed, his chance of success would then be . What credence in success is it epistemically rational for Bob to have?
And even more interesting cases (p. 917):
Embezzlement. One of Charlie’s colleagues is accused of embezzling funds. Charlie happens to have conclusive evidence that her colleague is guilty. She is to be interviewed by the disciplinary tribunal. But Charlie’s colleague has had an opportunity to randomize the content of several otherwise informative files (files, let us say, that the tribunal will want to examine if Charlie gives a damning testimony). Further, in so far as the colleague thinks that Charlie believes him guilty, he will have done so. Specifically, if is the colleague’s prediction for Charlie’s degree of belief that he’s guilty, then there is a chance that he has set in motion a process by which each proposition originally in the files is replaced by its own negation if a fair coin lands Heads, and is left unaltered if the coin lands Tails. The colleague is a very reliable predictor of Charlie’s doxastic states. After such randomization (if any occurred), Charlie has now read the files; they (now) purport to testify to the truth of propositions . Charlie’s credence in each of the propositions conditional on the proposition that the files have been randomized, is ; her credence in each conditional on the proposition that the files have not been randomized is . What credence is it epistemically rational for Charlie to have in the proposition that her colleague is guilty and in the propositions that the files purport to testify to the truth of?
In particular, Greaves' (2013, §8, pp. 43-49) epistemic version of Arntzenius' (2008) deliberational (causal) decision theory might be seen as a way of making sense of the first part of your theory. The idea, inspired by Skyrms (1990), is that deciding on a credence involves a cycle of calculating epistemic expected utility (measured by a proper scoring rule), adjusting credences, and recalculating utilities until an equilibrium is
obtained. For example, in Leap above, epistemic D(C)DT would find any credence permissible. And I guess that the second part of your theory serves as a way of breaking ties.
We assume two rules of inference:
Necessitation:
Distributivity:
Is there a reason why this differs from the standard presentation of K? Normally, I think you would say that K is generated by the following (coupled with substitution):
Axioms:
- All tautologies of propositional logic.
- Distribution: .
Rules of inference:
- Necessitation: .
- Modus ponens: .
And, second, the agent will continually implement that plan, even if this makes it locally choose counter-preferentially at some future node.
Nitpick: IIRC, McClennen never talks about counter-preferential choice. Rather, that's Gauthier's (1997) approach to resoluteness.
as devised by Bryan Skyrms and Gerald Rothfus (cf Rothfus 2020b).
Found a typo: it is supposed to be Gerard. (It is also misspelt in the reference list.)
Some people know that they do not have a solution. Andy Egan, in "Some Counterexamples to Causal Decision Theory" (1999, Philosophical Review)
This should say 2007.
These people all defect in PD and two-box in Newcomb.
Spohn argues for one-boxing in Reversing 30 years of discussion: why causal decision theorists should one-box.
Thanks.
Roughly, you don't actually get to commit your future-self to things. Instead, you just do what you (in expectation) would have committed yourself to given some reconstructed prior.
Agreed.
Just as a literature pointer: If I recall correctly, Chris Meacham's approach in "Binding and Its Consequences" is ultimately to estimate your initial credence function and perform the action from the plan with the highest EU according to that function.
Yes, that's a great paper! (I think we might have had a footnote on cohesive decision theory in a draft of this post.) Specifically, I think the third version of cohesive decision theory which Meacham formulates (in footnote 34), and variants thereof, are especially relevant to dynamic choice with changing awareness. The general idea (as I see it) would be that you optimize relative to your ur-priors, and we may understand the ur-prior function as the prior you would or should have had if you had been more aware. So when you experience awareness growth, the ur-priors change (and thus the evaluation of a given plan will often change as well).
He doesn't talk about awareness growth, but open-mindedness seems to fit in nicely within his framework (or at least the framework I recall him having).
(Meacham actually applies the ur-prior concept and ur-prior conditionalization to awareness growth in this paper.)
What do you mean by "the Bayesian Conditionalization thing" in this context? (Just epistemically speaking, standard Conditionalization is inadequate for dealing with cases of awareness growth. Suppose, for example, that one was aware of propositions {X, Y}, and that this set later expands to {X, Y, Z}. Before this expansion, one had a credence P(X ∨ Y) = 1, meaning Conditionalization recommends remaining certain in X ∨ Y; i.e., one is only permitted to place a credence P(Z) = 0. Maybe you are referring to something like Reverse Bayesianism?)
It's fairly clear to me that the authors do not have any specific and precise method in mind, Bjerring or no Bjerring.
Of course they don't have a specific proposal in the paper. I'm just saying that it seems like they would want to be more precise, or that a full specification requires more work on counterpossibles (which you seem to be arguing against). From the abstract:
While not necessary for considering classic decision theory problems, we note that a full specification of FDT will require a non-trivial theory of logical counterfactuals and algorithmic similarity.
...
What mathematical propositions get chosen to be "upstream" or "downstream" has to depend on what you're thinking of as "doing the changing" or "doing the reacting" for the question at hand.
If this is in fact how we should think about FDT, the theory becomes very uninteresting since it seems like you can then just get whatever recommendations you want from it.
Right, but it's fairly clear to me that this is not what the authors have in mind. For example, they cite Bjerring (2014), who proposes very specific and precise extensions of the Lewis-Stalnaker semantics.
(See my response to gjm's comment.)
To be sure, switching to Bet 1 is great evidence that is true (that's the whole point), but that's not the sort of reasoning FDT recommends. Rather, the question is if we take the Peano axioms to be downstream of the output of the algorithm in the relevant sense.
As the authors make clear, FDT is supposed to be "structurally similar" to CDT [1], and in the same way CDT regards the history and the laws to be out of their control in Ahmed's problems, FDT should arguably regard the Peano axioms to be out of their control (i.e., "upstream" of the algorithm). What could be more upstream?
Regarding Sleeping Counterfact: there seems to be two updates you could make, and thus there should be conceptual space for two interesting ways of being updatelessness in this problem; you could be 'anthropically updateless', i.e., not update on your existence in the standard Thirder way, and you could also be updateless with respect to the researchers asking for money (just as in counterfactual mugging). And it seems like these two variants will make different recommendations.
Suppose you make the first update, but not the second. Then the evidentialist value of paying up would plausibly be .
Suppose, on the other hand, that you are updateless with respect to both variables. Then the evidentialist value of paying up would be .
Interesting! Did thinking about those variants make you update your credences in SIA/SSA (or else)?
No, not really! This was mostly just for fun.
My follow-up question for almost all of them though, is based on use of the word "should" in the question. Since it presumably is not any moral version of "should", it's presumably a meaning in the direction of "best achieves a desired outcome".
The 'should' only designates what you think epistemic rationality requires of you in the situation. That might be something consequentialist (which is what I think you mean by "best achieves a desired outcome"), like maximizing accuracy[1], but it need not be; you could think there are other norms[2].
To see why epistemic consequentialism might not be the whole story, consider the following case from Greaves (2013) where the agent seemingly maximises accuracy by ignoring evidence and believing an obviously false thing.
Imps. Emily is taking a walk through the Garden of Epistemic Imps. A child plays on the grass in front of her. In a nearby summerhouse are n further children, each of whom may or may not come out to play in a minute. They are able to read Emily ’s mind, and their algorithm for deciding whether to play outdoors is as follows. If she forms degree of belief 0 that there is now a child before her, they will come out to play. If she forms degree of belief 1 that there is a child before her, they will roll a fair die, and come out to play iff the outcome is an even number. More generally, the summerhouse children will play with chance , where is the degree of belief Emily adopts in the proposition that there is now a child before her. Emily ’s epistemic decision is the choice of credences in the proposition that there is now a child before her, and, for each , the proposition that the th summerhouse child will be outdoors in a few minutes’ time.
See Konek and Levinstein (2019) for a good discussion, though.
If I give the same answer twice based on the same information, is that scored differently from giving that answer once?
Once again, this depends on your preferred view of epistemic rationality, and specifically how you want to formulate the accuracy-first perspective. Whether you want to maximize individual, average or total accuracy is up to you! The problems formulated here are supposed to be agnostic with regard to such things; indeed, these are the types of discussions one wants to motivate by formulating philosophical dilemmas.
- ^
This is plausibly cashed out by tying your epistemic utility function to a proper scoring rule, e.g. the Brier score.
- ^
See e.g. Sylvan (2020) for a discussion of what non-consequentialism might look like in the general, non-anthropic, case.
Ah, okay, got it. Sorry about the confusion. That description seems right to me, fwiw.
Thanks for clarifying. I still don't think this is exactly what people usually mean by ECL, but perhaps it's not super important what words we use. (I think the issue is that your model of the acausal interaction—i.e. a PD with survival on the line—is different to the toy model of ECL I have in my head where cooperation consists in benefitting the values of the other player [without regard for their life per se]. As I understand it, this is essentially the principal model used in the original ECL paper as well.)
The effective correlation is likely to be (much) larger for someone using UDT.
Could you say more about why you think this? (Or, have you written about this somewhere else?) I think I agree if by "UDT" you mean something like "EDT + updatelessness"[1]; but if you are essentially equating UDT with FDT, I would expect the "correlation"/"logi-causal effect" to be pretty minor in practice due to the apparent brittleness of "logical causation".
Correlation and kindness also have an important nonlinear interaction, which is often discussed under the heading of “evidential cooperation in large worlds” or ECL.
This is not how I would characterize ECL. Rather, ECL is about correlation + caring about what happens in your opponent's universe, i.e. not specifically about the welfare/life of your opponent.
- ^
Because updatelessness can arguably increase the game-theoretic symmetry of many kinds of interactions, which is exactly what is needed to get EDT to cooperate.
Related: A bargaining-theoretic approach to moral uncertainty by Greaves and Cotton-Barratt. Section 6 is especially interesting where they highlight a problem with the Nash approach; namely that the NBS is variant to whether (sub-)agents are bargaining over all decision problems (which they are currently facing and think they will face with nonzero probability) simultaneously, or whether all bargaining problems are treated separately and you find the solution for each individual problem—one at a time.
In the 'grand-world' model, (sub-)agents can bargain across situations with differing stakes and prima facie reach mutually beneficial compromises, but it's not very practical (as the authors note) and would perhaps depend too much on the priors in question (just as with updatelessness). In the 'small-world' model, on the other hand, you don't have problems of impracticality and so on, but you will miss out on a lot of compromises.
Now, let's pretend you are an egalitarian. You still want to satisfy everyone's goals, and so you go behind the veil of ignorance, and forget who you are. The difference is that now you are not trying to maximize expected expected utility, and instead are trying to maximize worst-case expected utility.
Nitpick: I think this is a somewhat controversial and nonstandard definition of egalitarianism. Rather, this is the decision theory underlying Rawls' 'justice as fairness'; and, yes, Rawls claimed that his theory was egalitarian (if I remember correctly), but this has come under much scrutiny. See Egalitarianism against the Veil of Ignorance by Roemer, for example.
I agree that the latter two examples have Moorean vibes, but I don't think they strictly speaking can be classified as such (especially the last one). (Perhaps you are not saying this?) They could just be understood as instances of modus tollens, where the irrationality is not that they recognize that their belief has a non-epistemic generator, but rather that they have an absurdly high credence in , i.e. "my parents wouldn't be wrong" and "philosophers could/should not be out of jobs".
The same holds if Alice is confident in Bob's relevant conditional behavior for some other reason, but can't literally view Bob's source code. Alice evaluates counterfactuals based on "how would Bob behave if I do X? what about if I do Y?", since those are the differences that can affect utility; knowing the details of Bob's algorithm doesn't matter if those details are screened off by Bob's functional behavior.
Hm. What kind of dependence is involved here? Doesn't seem like a case of subjunctive dependence as defined in the FDT papers; the two algorithms are not related in any way beyond that they happen to be correlated.
Alice evaluates counterfactuals based on "how would Bob behave if I do X? what about if I do Y?", since those are the differences that can affect utility...
Sure, but so do all agents that subscribe to standard decision theories. The whole DT debate is about what that means.
I'm not claiming this (again, it's about relative not absolute likelihood).
I'm confused. I was comparing the likelihood of (3) to the likelihood of (1) and (2); i.e. saying something about relative likelihood, no?
I'm not saying this is likely, just that this is the most plausible path I see by which UDT leads to nice things for us.
I meant for my main argument to be directed at the claim of relative likelihood; sorry if that was not clear. So I guess my question is: do you think the updatelessness-based trade you described is the most plausible type of acausal trade out of the three that I listed? As said, ECL and simulation-based trade arguably require much fewer assumptions about decision theory. To get ECL off the ground, for example, you arguably just need your decision theory to cooperate in the Twin PD, and many theories satisfy this criterion.
(And the topic of this post is how decision theory leads us to have nice things, not UDT specifically. Or at least I think it should be; I don't think one ought to be so confident that UDT/FDT is clearly the "correct" theory [not saying this is what you believe], especially given how underdeveloped it is compared to the alternatives.)
I had something like the following in mind: you are playing the PD against someone implementing "AlienDT" which you know nothing about except that (i) it's a completely different algorithm to the one you are implementing, and (ii) that it nonetheless outputs the same action/policy as the algorithm you are implementing with some high probability (say 0.9), in a given decision problem.
It seems to me that you should definitely cooperate in this case, but I have no idea how logi-causalist decision theories are supposed to arrive at that conclusion (if at all).
What's your take on playing a PD against someone who is implementing a different decision algorithm to the one you are implementing, albeit strongly (logically) correlated in terms of outputs?
Insofar as I have hope in decision theory leading us to have nice things, it mostly comes via the possibility that a fully-fleshed-out version of UDT would recommend updating "all the way back" to a point where there's uncertainty about which agent you are. (I haven't thought about this much and this could be crazy.)
This was surprising to me. For one, that seems like way too much updatelessness. Do you have in mind an agent self-modifying into something like that? If so, when and why? Plausibly this would be after the point of the agent knowing whether it is aligned or not; and I guess I don't see why there is an incentive for the agent to go updateless with respect to its values at that point in time.[1] At the very least, I think this would not be an instance of all-upside updatelessness.
Secondly, what you are pointing towards seems to be a very specific type of acausal trade. As far as I know, there are three types of acausal trade that are not based on inducing self-locating uncertainty (definitely might be neglecting something here): (1) mutual simulation stuff; (2) ECL; and (3) the thing you describe where agents are being so updateless that they don't know what they value. (You can of course imagine combinations of these.) So it seems like you are claiming that (3) is more likely and more important than (1) and (2). (Is that right?)
I think I disagree:
- The third type seems to only be a possibility under an extreme form of UDT, whilst the other two are possible, albeit in varying degrees, under (updateless or updateful) EDT, TDT, some variants of (updateful or updateless) CDT[2], and a variety of UDTs (including much less extreme ones than the aforementioned variant) and probably a bunch of other theories that I don't know about or am forgetting.
- And I think that the type of updateless you describe seems particularly unlikely, per my first paragraph.
- It is not necessarily the case that your expectation of what you will value in the future, when standing behind the veil of ignorance, nicely maps onto the actual distribution of values in the multiverse (meaning the trade you will be doing is going to be limited).
(I guess you can have the third type of trade, and not the first and second one, under standard CDT coupled with the strong updatelessness you describe; which is a point in favour of the claim I think you are making—although this seems fairly weak.)
- ^
One argument might be that your decision to go updateless in that way is correlated with the choice of the other agents to go updateless in the same way, and then you get the gains from trade by both being uncertain about what you value. But if you are already sufficiently correlated, and taking this into account for the purposes of decision-making, it is not clear to me why you are not just doing ECL directly.
- ^
For example, Spohn's variant of CDT.
From Arif Ahmed's Evidence, Decision and Causality (ch. 5.4, p. 142-143; links mine):
Deliberating agents should take their choice to be between worlds that differ over the past as well as over the future. In particular, they differ over the effects of the present choice but also over its unknown causes. Typically these past differences will be microphysical differences that don’t matter to anyone. But in Betting on the Past they matter to Alice.
. . .
On this new picture, which arises naturally from [evidential decision theory]. . ., it is misleading to think of decisions as forks in a road. Rather, we should think of them as choices between roads that were separate all along. For instance, in Betting on the Past, Alice should take herself to be choosing between actualizing a world at which P was all along true and one at which it was all along false.
The difficulty is in how to weight the frequency/importance of the situations they face.
I agree with this. On the one hand, you could just have a bunch of Procreation problems which would lead to the FDTer ending up with a smaller pot of money; or you could of course have a lot of Counterfactual Muggings in which case the FDTer would come out on top—at least in the limit.
Ah, nice. I was just about to recommend sections 2.6.2 and 3 of Multiverse-wide Cooperation via Correlated Decision Making by Caspar.
From this, they trivially conclude that EDT will have higher stakes than CDT: if there are more Good Twins (Evil Twins), EDT will recommend one-boxing (two-boxing) very strongly, since this will provide evidence to you about many agents doing the same. But I'm not satisfied with this answer, because if you don't know whether more Good Twins or Evil Twins exist, you won't be obtaining that evidence (upon taking the decision)!
I don't think this is a situation of evidential symmetry which would warrant a uniform distribution (i.e. you can't just say that "you don't know"). (Moreover, there does not seem to be an overwhelmingly natural partition of the state space in this particular case, which arguably makes the Principle of Indifference inapplicable regardless—see section 3 of Greaves [2016].)
One weak piece of evidence is e.g. provided by the mediocrity principle: since I know for sure that there exists at least one agent who has my preferences and makes decisions in the way I do (me!)—and I don't know the opposite for sure—I should expect there to be more Good Twins than Evil Twins.
Moreover, I should probably expect there in general to be some correlation between decision theory and values, meaning that my (decision-theoretic) twins are by my lights more likely to be Good than Evil.
Procreation* gives both FDT and CDT agents (and indeed, all agents) the same dilemma. FDT agents procreate and live miserably; CDT agents don't procreate and almost certainly don't exist. FDT beats CDT in this dilemma.
This doesn't seem right: you already exist! In order to say that "FDT beats CDT" I think you have to argue that one should care about the number of branches you exist in—which is what you plausibly have uncertainty about, not about whether this very instance of you exists. (And this is arguably just about preferences, as Christiano writes about here. So it is unclear what it would even mean to say that "FDT beats CDT".) That is, this is about implementing a specific version of mixed-upside updatelessness or not—specifically, the multiverse version of MUU I describe here.
The authors consider the infinite case in section 5 of the paper. They conclude:
...if one wishes to avoid widespread incomparability between infinite worlds, then one will probably endorse a view that supports the Wager.
Surprisingly, Schwarz doesn't analyze CDT's and FDT's answer to Prisoner's Dilemma with a Twin (besides just giving the answers). It's worth noting FDT clearly does better than CDT here, because the FDT agent (and its twin) both get away with 1 year in prison while the CDT agent and its twin both get 5. This is because the agents and their twins are clones - and therefore have the same decision theory and thus reach the same conclusion to this problem. FDT recognizes this, but CDT doesn't. I am baffled Schwarz calls FDT's recommendation on this problem "insane", as it's easily the right answer.
I personally agree that cooperating in the Twin PD is the correct choice, but I don't think it is meaningful to argue for this on the grounds of decision-theoretic performance (as you seem to do). From The lack of performance metrics for CDT versus EDT, etc. by Caspar Oesterheld:
[T]here is no agreed-upon metric to compare decision theories, no way to asses even for a particular problem whether one decision theory (or its recommendation) does better than another. (This is why the CDT-versus-EDT-versus-other debate is at least partly a philosophical one.) In fact, it seems plausible that finding such a metric is “decision theory-complete” (to butcher another term with a specific meaning in computer science). By that I mean that settling on a metric is probably just as hard as settling on a decision theory and that mapping between plausible metrics and plausible decision theories is fairly easy.
Indeed, Schwarz makes a similar point in the post you are responding to:
Yudkowsky and Soares constantly talk about how FDT "outperforms" CDT, how FDT agents "achieve more utility", how they "win", etc. As we saw above, it is not at all obvious that this is true. It depends, in part, on how performance is measured. At one place, Yudkowsky and Soares are more specific. Here they say that "in all dilemmas where the agent's beliefs are accurate [??] and the outcome depends only on the agent's actual and counterfactual behavior in the dilemma at hand – reasonable constraints on what we should consider "fair" dilemmas – FDT performs at least as well as CDT and EDT (and often better)". OK. But how we should we understand "depends on … the dilemma at hand"? First, are we talking about subjunctive or evidential dependence? If we're talking about evidential dependence, EDT will often outperform FDT. And EDTers will say that's the right standard. CDTers will agree with FDTers that subjunctive dependence is relevant, but they'll insist that the standard Newcomb Problem isn't "fair" because here the outcome (of both one-boxing and two-boxing) depends not only on the agent's behavior in the present dilemma, but also on what's in the opaque box, which is entirely outside her control. Similarly for all the other cases where FDT supposedly outperforms CDT. Now, I can vaguely see a reading of "depends on … the dilemma at hand" on which FDT agents really do achieve higher long-run utility than CDT/EDT agents in many "fair" problems (although not in all). But this is a very special and peculiar reading, tailored to FDT. We don't have any independent, non-question-begging criterion by which FDT always "outperforms" EDT and CDT across "fair" decision problems.
From Cheating Death in Damascus (bold emphasis mine):
It’s less clear how we should model this case from the point of view of FDT, and there are a variety of options. The most natural and illustrative, we think, is to assume that what actions you would or would not perform in various (hypothetical or real) circumstances determines whether you’re a psychopath. What actions you would perform in which circumstances is in turn determined by your decision algorithm. On this reading of this case, the potential outputs of your decision algorithm affect both whether you’d press the button and whether you’re a psychopath.
In practice, this leads the FDT agent always to refrain from pressing the button, but for very different reasons from the CDT agent. [The FDT agent] reasons that if she were to press the button that kills so many people, then she would be a psychopath. She does not regard her psychopathic tendencies—or lack thereof—as a fixed state of the world isolated from what decision she actually makes here.
This seems like the right reasoning, at least on this understanding of what psychopathy is. Psychopaths just are people who tend to act (or would act) in certain kinds of ways in certain kinds of circumstances. We can take for granted that everyone is either born a psychopath or born a non-psychopath, and that [the FDT agent's] action cannot causally change this condition she was born with. Yet if this condition consists in dispositions to behave in certain ways, then whether [the FDT agent] is a psychopath is subjunctively tied to the decisions she actually makes. If you would not perform in circumstances , then you also would not be the kind of person who performs actions like in circumstances like . FDT vindicates just this sort of reasoning, and refrains from pressing the button for the intuitively simple reason of “if I pressed it, I’d be a psychopath” (without any need for complex and laborious ratification procedures). When we intervene on the value of the variable, we change not just what you actually do, but also what kind of person you are.