Beyond Astronomical Waste

Being run as a simulation in another universe isn't necessarily the only way to control what happens in that universe.

Had you seen Multiverse-wide cooperation via correlated decision-making, btw? (somewhat similar to acausal trade but differs in that it's based on the agents being similar to each other rather than modeling each other)

If there's some kind of measure of "observer weight" over the whole mathematical universe, we might be already much larger than 1/3^^^3 of it, so the total utilitarian can only gain so much. And even if there's no measure, I'm not sure my total utilitarianism would scale linearly to such numbers. But I'm very confused about all this.

I curated this post because it crystallised an important point regarding optimising the long-term future, that I've not seen anyone write down succinctly before (with reference to the relevant technical concepts, while still being short and readable).

Thanks. I agree with your overall conclusions.

On the specifics, Bostrom's simulation argument is more than just a parallel here: it has an impact on how rich we might expect our direct parent simulator to be.

The simulation argument applies similarly to one base world like ours, or to an uncountable number of parallel worlds embedded in Tegmark IV structures. Either way, if you buy case 3, the proportion of simulated-by-a-world-like-ours worlds rises close to 1 (I'm counting worlds "depth-first", since it seems most intuitive, and infinite simulation depth from worlds like ours seems impossible).

If Tegmark's picture is accurate, we'd expect to be embedded in some hugely richer base structure - but in Bostrom's case 3 we'd likely have to get through N levels of worlds-like-ours first. While that wouldn't significantly change the amount of value on the table, it might make it a lot harder for us to exert influence on the most valuable structures.

This probably argues for your overall point: we're not the best minds to be making such calculations (either on the answers, or on the expected utility of finding good answers).

What of exponential total utilitarianism? That's a total utilitarianism that multiplies the total utility by the exponential of the population. It may be very unlikely, but as population grows, it will eventually come to dominate.

That's why I think moral theories should be normalised independently, to prevent the super-population ones from winning just by default.

I think that at the time this post came out, I didn't have the mental scaffolding necessary to really engage with it – I thought of this question as maybe important, but sort of "above my paygrade", something better left to other people who would have the resources to engage more seriously with it.

But, over the past couple years, the concepts here have formed an important component of my understanding of robust agency [LW · GW]. Much of this came from private in-person conversations, but this post is the best writeup of the concept I'm currently aware of.

One thing I like about this post is the focus on philosophical competence. Previously, I'd thought of this question as dangerous to think about, because you might make philosophical mistakes that doomed you or your universe for (in retrospect) silly reasons.

My current model is more like "no, you-with-your-21st-century-human-brain shouldn't actually attempt to take actions aiming primarily to affect other universes on the macro scale. Negotiating with other universes is something you do when you're a literal galaxy brain that is quite confident in it's philosophy."

But, meanwhile, it seems that:

(note: low-to-mid confidence, still working through these problems myself, and I very much still philosophically confused about at least some of this)

– becoming philosophically competent, as a species, may be one of the most important goals facing humanity, and how this project interfaces with AI development may be crucially important. This may be relevant to people (like me) who aren't directly working on alignment but trying to have a good model of the strategic landscape.

– a concept not from this particular post, but relevant, is the notion that the question is not "are you in a simulation or not?", it's more like "to what degree are you in simulations? Which distribution of agents are you making choices on behalf of?". And this has some implications about how you should make choices that you should worry about now, before you are a literal galaxy brain. (many of which are more mundane)

– I think there may be a connection between "Beyond Astronominal Waste" and "Robustness to scale [LW · GW]." You can't be a galaxy brain now, but you can be the sort of person who would be demonstrably safe to scale up, in a way that simulations can detect, which might let you punch above your current weight, in terms of philosophical competence.

This is a post that's stayed with me since it was published. The title is especially helpful as a handle. It is a simple reference for this idea, that there are deeply confusing philosophical problems that are central to our ability to attain most of the value we care about (and that this might be a central concern when thinking about AI).

It's not been very close to areas I think about a lot, so I've not tried to build on it much, and would be interested in a review from someone who thinks in more detail about these matters more, but I expect they'll agree it's a very helpful post to exist.

I am not sure how one can talk about the observed universe and the number 3^^^3 in the same sentence, given that the maximum informational content is roughly 10^120 qubits, the rest is outside the cosmological horizon. Alternatively, if we talk about the simulation argument, then the expression "practical implications" seems out of place.

I just realised that the problem of the limited size of the Universe is isomorphic to the problem of how to survive the end of the universe, which I analysed here [LW · GW], but the escape routs described by OP are different, and more rely on acausal trade and simulation hacking than on physic manipulation.

MUH doesn't imply the existence of halting oracles. Indeed, the Computable Universe Hypothesis is supposed to be an extension of the Mathematical Universe Hypothesis, but CUH says that halting oracles do not exist.

Isn't all this massively dependent on how your utility $U$ scales with the total number $N$ of well-spent computations (e.g. one-bit computes)?

That is, I'm asking for a gut feeling here: What are your relative utilities for $10^{100}$, $10^{110}$, $10^{120}$, $10^{130}$ universes?

Say, $U(0)=0$, $U(10^100)=1$ (gauge fixing); instant pain-free end-of-universe is zero utility, and a successful colonization of the entire universe with a suboptimal black hole-farming near heat-death is unit utility.

Now, per definitionem, the utility $U(N)$ of a $N$-computation outcome is the inverse of the probability $p$ at which you become indifferent to the following gamble: Immediate end-of-the-world at probability $(1-p)$ vs an upgrade of computational world-size to $N$ at propability $p$.

I would personally guess that $U(10^{130})< 2 $; i.e. this upgrade would probably not be worth a 50% risk of extinction. This is massively sublinear scaling.

With quantum branching, our universe could have some number like a googolplex of stuff, maybe more. And philosophically, you're worried about the difference between that and 3^^^3? I get that there's a big gap there but I'd guess it's one that we're definitionally unable to do useful moral reasoning about.

I doubt that there's any moral difference between running a person and asking a magical halting oracle what they would have said.