Babyeater's dilemma

The altruistic assumption given here seems implausible for a utility function ultimately derived from evolution, so while it's an interesting exercise I'm not sure there's anything to be worried about in practice.

I'm not worried by the result because there are two very implausible constraints: the number of possible utility functions and the utility of the compromise strategy. Given that there are, in fact, many possible utility functions, it seems really really unlikely that there is a strategy that has 3/10 the utility of the optimal strategy for every possible utility function. Additionally, some pairs of utility functions won't be conducive to high-utility compromise strategies. For example: what if one civilization has paperclip maximization as a value, and another has paperclip minimization as a value?

ETA: Actually, this reasoning is somewhat wrong. The compromise strategy just has to have average utility > x/n, where n is the number of utility functions, and x is the average utility of the optimal strategies for each utility function. (In the context of the original example, n = 10 and x = 10. So the compromise strategy just has to be, on average, > 1 for all utility functions, which makes sense.) I still submit that a compromise strategy having this utility is unlikely, but not as unlikely as I previously argued.

Edit: This comment is retracted. My comment is wrong, primarily because it misses the point of the post, which simply presents a usual game theory-style payoff matrix problem statement. Thanks to Tyrrell McAllister for pointing out the error, apologies to the readers. See this comment for details. (One more data point against going on a perceptual judgement at 4AM, and not double-checking own understanding before commenting on a perceived flaw in an argument. A bit of motivated procrastination also delayed reviewing Tyrrell's response.)

Humanity has also worked out the optimal strategy S0...S9 for each utility function. But they just happen to score poorly on all of the others:

f_i(S_i) = 10
f_i(S_j) = 1 for i != j

Who is following these strategies? The only interpretation that seems to make sense is that it's humanity in each case (is this correct?), that is S2 is the strategy that, if followed by humanity, would optimize aliens #2's utility.

In this case, the question is what do the f_i(S_j) mean. These are expected utilities of a possible strategy, but how do you compute them? CDT, TDT and UDT would have it differently.

In any case, it's conventional to mean by "expected utility of a possible decision" the value that you'll be actually optimizing. With CDT, it's computed in such a way that you two-box on Newcomb as a result, in TDT and UDT the bug is fixed and you one-box, but still by optimizing expected utility (computed differently) of the decision that you'd make as a result. Similarly for PD, where you one-box in UDT/ADT not because you take into account utilities of different agents, but because you take into account the effect on your own utility mediated by other agent's hypothetical response to your hypothetical decision, that is you just compute your own expected utility more accurately, and still just maximize only your own utility.

Cooperation in PD of the kind TDT and UDT enable is not about helping the other, it's about being able to take into account other's hypothetical cooperation arising in response to your hypothetical cooperation. Altruistic agents already have their altruism as part of their own utility function, it's a property of their values that's abstracted away at the level where you talk about utilities and should no longer be considered at that level.

So the answer to "What should you maximize?" is, by convention, "Your own expected utility, period." This is just what "expected utility" means (that is, where you can factor utility as expected value of a utility function over some probability distribution; otherwise you use "utility" in this role). The right question should be, "How should you compute your expected utility?", and it can't be answered given the setup in this post, since f_i are given as black boxes. (Alternatively, you could give a way of estimating utility as a black box, and then consider various ways of constructing an estimate of expected utility out of it.)

An interesting idea, but I'm afraid the idea is little more than interesting. Given all your premises, it does follow that compromise would be the optimal strategy, but I find some of them unlikely:

• That there is a small, easily computable number of potential utility functions, like 10 as opposed to 10^(2^100)
• I have qualms with the assumption that these computed utility functions are added. I would more readily accept them being mutually exclusive (e.g. one potential utility function is "absorb all other worlds" or "defect in all inter-species prisoner's games for deontological reasons")
• Though I won't give it a probability estimate, I consider "humanity has worked out that it's very likely that a lot of alien worlds exist" to be a potential defeater.

If none of those complaints holds up, I still see no reason to be worried about the result. Why worry about getting a higher score?

I think this result means that you understand the true prisoner's dilemma and acausal trade.

I'm not sure if this qualifies as a mistake per se, but it seems very implausible to me that the only advanced civilization-enabling utility functions are altruistic towards aliens. Is there evidence in favor of that hypothesis?

To make this more interesting, interpret it as a true prisoner's dilemma. I.E. the aliens care about something stupid like maximizing paperclips.

Anyone else worried by this result, or have I made a mistake?

To update my reply to Silas Barta, after a little reflection I would say this:

The various species are supposed to possess common knowledge of each other's utility functions, and of each other's epistemic beliefs about how these utility functions can be satsified.

Since the various species' preferences are described by utility functions, we must assume that each species has self-modified collectively (or so the humans believe) such that they collectively obey the von Neumann-Morgenstern axioms - this eliminates much of the complexity that I had in mind when I wrote my reply to Barta.

However, one further item of common knowledge would be helpful: common knowledge of whether the various species are likely to be timeless decision theorists. If they possess this common knowledge, then the dilemma is just a disguised version of a standard Newcomblike problem: the agents possess common knowledge of all the relevant factors that might influence the abstract computation that they implement in determining which strategy to employ. This is no different to the scenario in which two AIs can read one another's source code - except in this case they do it by magic (the scenario is entirely ridiculous, I'm afraid). And in that case co-operation in the prisoner's dilemma is the optimal choice.

On the other hand if they don't possess common knowledge of whether they are TDT-agents (rather than CDT-agents for example) then whether it is wise for humans to defect or co-operate depends on their probability estimate regarding whether the aliens are mostly TDT-agents, and their estimates of the aliens' own estimates whether the other species are TDT-agents, et cetera. I don't really know how that infinite regress would be resolved by the humans, and your premises give us little way of knowing what these probability estimates might be.

Anyone else worried by this result, or have I made a mistake?

This seems correct.

If some far away world implements utility function f2 and strategy S2, I intuitively feel like I ought to care more about f2(S2) than about f0(S2), even if my own utility function is f0. Provided, of course, that S2 doesn't involve destroying my part of the world.

I don't expect that humans, on meeting aliens, would try to impose our ethical standards on them. We generally wouldn't see their minds as enough like ours to see their pain as real pain. The reason I think this is that very few people think we should protect all antelopes from lions, or all dolphins from sharks. So the babyeater dillemma seems unrealistic to me.

You are worried that, given your assumptions, civilizations might not be willing to pay an extremely high price to do things that aliens would like if they knew about them, which they don't.

But one of your assumptions is that every civilization has a moral system that advocates attacking and enslaving everyone they meet who thinks differently from them.

It would be worrying if a slightly bad assumption led to a very bad conclusion, but a very bad assumption leading to a slightly bad conclusion doesn't strike me as particularly problematic.

An interesting question. Some thoughts here:

1. Does this type of reasoning mean it is a good idea to simulate lots of alien civilizations (across lots of different worlds), to see what utility functions emerge, and how frequently each type emerges?

2. It seems like detailed simulation is quite a sensible strategy anyway, if we're utility trading (detailed enough to create conscious beings). We could plausibly assume that each utility function f(i) assigns positive utility to the aliens of type (i) existing in a world, as long as their welfare in that world exceeds an acceptable threshold. (For instance, if we imagine worlds with or without humans, then we tend to prefer the ones with, unless they are being horribly tortured etc..) So by simulating alien species (i), and checking that they generally prefer to exist (rather than trying to commit suicide) we are likely doing them a favour according to f(i), and we can assume that since our TDT decision is linked to theirs, we are increasing the number of worlds humans exist in too.

I'm intrigued by the idea that TDT leads to a converged "average utilitity" function, across all possible worlds with TDT civilizations...

If you knew something about the expected development process of potential alien civilizations and could use that information to estimate a probability of them defecting in case like this, then which utility functions would you include in your set of aliens to cooperate with? Roughly, should you cooperate with each civilization proportional to your expectation of each civilization cooperating? Also, should you cooperate proportional to the number of expected civilizations implementing each utility function?

This seems unavoidable to me, as long as you first except "acausal trade", which it seems you must accept if you accept Pearl-style causality and additive consequentialism.

Humanity has also run a large number of simulations of how alien worlds evolve. It has determined that of those civilizations which reach the same level of advancement - that know their own utility function and have a strategy for optimizing it - there is an equal probability that they will end up with each of 10 possible utility functions.

No, humanity isn't going to do that. We'd be exposing ourselves to blackmail from any simulated world whose utility function had certain properties -- it'd be summoning a basilisk. For humanity's utility function in particular, there is an asymmetry such that the potential losses from acausal trade dramatically outweigh the potential gains.

Assuming that TDT doesn't apply, the fact that we would be in a prisoner's dilemma is irrelevant. The only rational option for humanity would be to defect by maximising its local utility - whether humans defect or cooperate in the dilemma has no effect on what the aliens choose to do.

So really the problem is only interesting from a timeless decision theory perspective (I feel that you might have made this more explicit in your post).

According to my sketchy understanding of TDT, if in a prisoner's dilemma both parties can see the other's source code, or otherwise predict the other's behaviours with extreme accuracy then it is rational for them to co-operate because it is physically possible for each of them to be almost certain that the other will co-operate, and they would be able to tell if the other were to suddenly change his mind.

I don't see how this generalises to the situation that you have described. If anyone thinks that TDT is in fact relevant to this problem, I would be interested in hearing their (lucid) explanation why this is so.

They are altruistic, in the sense that they care just as much about far-away aliens that they can't even see as they do about members of their own species.

none of the players has any causal influence on any other.

An aggregate score without any causal influence is meaningless. Without influence on each other, each should pursue its own best interest, not some meaningless compromise solution.

Anyone else worried by this result, or have I made a mistake?

Surely only utilitarians would be concerned by it. Others will just reject the "altruistic" assumption as being terribly unrealistic.

My thought is that there's no reason to believe the humans are right over any of the other groups. One person was born with one mind, another with another. There's no reason to pick one mind. As such, I'd pick the compromise, even if we additionally worked out that the other aliens wouldn't try the compromise.

I hope we don't need to worry about that. It's odd because you and the aliens can never know whether the others are defecting. It reminds me of Simulation Warfare (pdf) where you can change the odds that you are in a simulation just by choosing to simulate universes like yours and also change the nature of your universe by simulating types of universes you want to be in that are compatible with your universe so far. If your argument works, we would be retroactively putting ourselves into a cooperating universe... basically "causing" things to happen outside of our observable universe... (EDIT: removed expressions of future shock)