indexical uncertainty and the Axiom of Independence

Are you familiar with Cooperative Game Theory? I'm just learning it now, but it sounds very similar to what you're talking about, and maybe you can reused some of its theory and math. (For some reason I've only paid attention to non-cooperative game theory until recently.) Here's a quote from page 356 of "Handbook of Game Theory with Economic Applications, Vol 1":

Of all solution concepts of cooperative games, the core is probably the easiest to understand. It is the set of all feasible outcomes (payoffs) that no player (participant) or group of participants (coalition) can improve upon by acting for themselves.

The following is an honest non-rhetorical question: Is it not misleading to use the word 'cooperation' as you seem to be using it here? Don't you still get 'cooperation' in this sense if the subsets of agents are not causally interacting with each other (say) but have still semi-Platonicly 'merged' via implicit logical interaction as compared to some wider context of decision algorithms that by logical necessity exhibit comparatively less merging? This sets up a situation where an agent can (even accidentally) engineer 'Pareto improvements' just by improving its decision algorithm (or more precisely replacing 'its' decision algorithm (everywhere 'it' is instantiated, of course...) with a new one that has the relevant properties of a new, possibly very different logical reference class). It's a total bastardization of the concept of trade but it seems to be enough to result in some acausal economy (er, that is, some positive-affect-laden mysterious timeless attractor simultaneously constructed and instantiated by timeful interaction) or 'global cooperation' as you put it, and yet despite all that timeless interaction there are many ways it could turn out that would not look to our flawed timeful minds like cooperation. I don't trust my intuitions about what 'cooperation' would look like at levels of organization or intelligence much different from my own, so I'm hesitant to use the word.

(I realize this is 'debating definitions' but connotations matter a lot when everything is so fuzzily abstract and yet somewhat affect-laden, I think. And anyway I'm not sure I'm actually debating definitions because I might be missing an important property of Pareto improvements that makes their application to agents that are logical-property-shifting-over-time not only a useless analogy but a confused one.)

This question is partially prompted by your post about the use of the word 'blackmail' as if it was technically clear and not just intuitively clear which interactions are blackmail, trade, cooperation, et cetera, outside of human social perception (which is of course probably correlated with more-objectively-correct-than-modern-human meta-ethical truths but definitely not precisely so).

If the above still looks like word salad to you... sigh please let me know so I can avoid pestering you 'til I've worked more on making my concepts and sentences clearer. (If it still looks way too much like word salad but you at least get the gist, that'd be good to know too.)

A Paperclip maximizer might wipe out humanity, then catch up on its reflective consistency, look back, notice that there was a counterfactual future where a FAI is built, allot some of the collective preference to humanity, and restore it from the info remaining after the initial destruction (effectively constructing a FAI in the process).

This seems fishy to me, given the vast space of possible preferences and the narrowness of the target. Assuming your idea of preference compromise as the convergent solution, what weighting might a reflective AI give to all of the other possible preference states, especially given the mutually exclusive nature of some preferences? If there's any Occam prior involved at all, something horrifically complicated like human moral value just isn't worth considering for Clippy.

I'd like to note a connection between Vladimir's idea, and Robin Hanson's moral philosophy, which also involves taking into account the wants of counterfactual agents.

I'm also reminded of Eliezer's Three Worlds Collide story. If Vladimir's right, many more worlds (in the sense of possible worlds) will be colliding (i.e., compromising/cooperating).

I look forward to seeing the technical details when they've been worked out.

Ok, so I see that probability plays a role in determining one's "bargaining power", which makes sense. We still need a rule that outputs a compromise set of preferences when given a set of agents, their probabilities, individual preferences, and resources as input, right? Does the rule need to be uniquely fair or obvious, so that everyone can agree to it without discussion? Do you have a suggestion for what this rule should be?

Edit: I see you've answered some of my questions already in the other reply. This is really interesting stuff!

Here's a precise definition of "observer-moment", "trustworthiness" and everything else you might care to want defined. But I will ask you for a favor in return...

Mathematical formulation 1: Please enter a program that prints "0" or "1". If it prints "1" you lose $100, otherwise nothing happens.

Mathematical formulation 2: Please enter a program that prints "0" or "1". If it prints "1" you gain $10000 or lose $100 with equal probability, otherwise nothing happens.

Philosophical formulation by Vladimir Nesov, Eliezer Yudkowsky et al: we ought to find some program that optimizes the variables in case 1 and case 2 simultaneously. It must, must, must exist! For grand reasons related to philosophy and AI!

Now the favor request: Vladimir, could you please go out of character just this once? Give me a mathematical formulation in the spirit of 1 and 2 that would show me that your and Eliezer's theories have any nontrivial application whatsoever.

Strategy one has U1/2 in both A-utility and B-utility with the additional property that the utility is in the correct fork where it can be used (i.e. it truly exists).

Strategy two has U2/2 in both A-utilty and B-utility but the additional property that the utility produced is not going to be usable in the fork where it is produced (i.e. the actual utility is really U0/2 unless the utility can be traded for the opposite utility which is actually usable in the same fork).

Assuming that there is no possibility of trade (since you describe no method by which it is possible):

I don't see a requirement for trade existing in the counterfactual mugging problem so I accept it.

Since the above deal requires the possibility of trade to actually gain USABLE utility (arguably the only nonzero kind assuming that [PersonalUse OR Trade = Usability]) and I don't see the possibility for trade, I am justified in rejecting the above deal despite accepting the counterfactual deal.

You're presenting a standard PD, only distributed across possible worlds. Doesn't seem to be any difference between splitting into 2 possible worlds, and taking 2 prisoners into 2 different cells. So you would need to provide a solution, a mechanism for cooperation, that would also work for the PD. And you haven't.

Don't know what you mean by "accept counterfactual mugging". Especially since I just said I don't agree with your interpretation of it. I believe the counterfactual mugging is also just a rephrasing of the PD. You should keep the $100 unless you would cooperate in a one-shot PD. We all know that rational agents would do better by cooperating, but that doesn't make it happen.

If you are at a different location, that's a different world state.

If the same world contains two copies of you, you can be either copy within the same world.

I need to brush up on evolutionary game theory, but I don't see the correspondence between these two subjects yet. Can you take a standard puzzle involving indexical uncertainty, for example the Sleeping Beauty Problem, and show how to solve it using evolutionary game theory?

In the case of probability of ordinary observations, I think you can assign probabilities if your preferences over possible strategies satisfy some conditions, the major one being what you prefer to happen in one branch has to be independent of what you prefer to happen in another branch, i.e., the Axiom of Independence. If we ignore counterfactual-mugging type considerations, do you see any problems with this? If so can you give an example?

Let A = "I'm immunized with vaccine A", B = "I'm immunized with vaccine B", p = probability of being the original. The Axiom of Independence implies

p A + (1-p) A > p B + (1-p) A iff p A + (1-p) B > p B + (1-p) B

To see this, substitute A for C in the axiom, and then substitute B for C. This statement says that what I prefer to happen at one location doesn't depend on what happens at another location, which is false in the example. In fact, the right side of the iff statement is true while the left side is false.

Does this explanation help?

Yes, I meant to ask exactly that.

If the worlds are 4D crystals of timelines, two agents placed at different times are not much different from two agents placed in the same time-slice.

If you are choosing between A and B, the C option doesn't get implemented (equivalently: you prior states that's almost impossible, if you don't do that yourself), so the money is certainly there, $1000>$100.

“I receive either vaccine A or B, but the opposite of what the other guy receives” isn’t equivalent to “I receive either vaccine A or B”.

Yeah, I think this makes sense, at least if we assume a decision theory like EDT, where pressing C gives me a lot of evidence that the other guy also presses C, so I can think of the consequences of pressing C as "50% chance I receive A and the other guy receives B, 50% chance I receive B and the other guy receives A" which is not a gamble between the consequences of pressing A (we both receive A) and the consequences of pressing B (we both receive B) so Independence isn't violated.

I think at the time I wrote the post, I was uncritically assuming CDT (under the impression that it was the mainstream academic decision theory), and under CDT you're supposed to reason only about the causal consequences of your own decision, not how it correlates with the other guy's decision. In that case the consequences of pressing C would be "50% chance I receive A, 50% chance I receive B" and then strictly preferring it would violate Independence. (Unless I say that pressing C also has the consequence of C having been pressed, and I prefer that over A or B having been pressed, but that seems like too much of a hack to me, or violates the spirit/purpose of decision theory.)

I hope this is correct and makes sense. (It's been so long since I wrote this post that I had to try to guess/infer why I wrote what I wrote.) If so, I think that part of the post (about "preferential interaction") is more of an argument against CDT than against Independence, but the other part (about physical interactions) still works?