how do we reconcile the ideas that 1) most imaginable expected utility maximizers would drive humanity to extinction, and 2) we humans are still alive, even though every single physical system in existence is mathematically equivalent to some class of flawless expected utility maximizers?

First of all, if everything is mathematically equivalent to an EU maximizer, then saying that something is an EU maximizer no longer represents meaningful knowledge, since it no longer distinguishes between fiction and reality [LW · GW]. As Eliezer once [LW · GW] beautifully put it:

Your strength as a rationalist is your ability to be more confused by fiction than by reality; if you are equally good at explaining any outcome you have zero knowledge. The strength of a model is not what it can explain, but what it can’t, for only prohibitions constrain anticipation. If you don’t notice when your model makes the evidence unlikely, you might as well have no model, and also you might as well have no evidence; no brain and no eyes.

So if you are capable of taking any self-consistent physical description of a system, whether it actually conforms to reality or not, and saying that this can be modeled as an EU maximizer set-up, then the fact that something is an EU maximizer no longer constrains expectations [LW · GW] by prohibiting certain world-states. Since we do [LW(p) · GW(p)] actually have certain expectations related to EU maximizers, this suggests that the discussion you are starting here is primarily semantic and not substantive [LW · GW] (as you are using the concept of EU maximizers in a manner that John Wentworth has described [LW(p) · GW(p)] as "confused").

With this in mind, the resolution to the apparent tension between the two statements you included in the section I quoted at the top of this comment is that it^[1] implicitly relies on the use of a counting argument [LW · GW] without properly considering what compact manifold [LW · GW] it arises out of. Put differently, we already know that humans are still alive, which screens off [LW · GW] other considerations that are not as narrowly [LW · GW] tailored to this fact. When you condition on any piece of knowledge, you explicitly change the probabilities of all other events in a manner that makes them compatible with what you are conditioning. It could not have been any other way [LW · GW]. So the space of "imaginable EU maximizers", after you condition, does not get a uniform subjective probability distribution [? · GW] or anything close to it, but rather one that puts mass only (or, at least, primarily) on those hypotheses that ensure humans would survive until now (since we know that we, in fact, have).

1. ^{^}

   By which I am referring to "the feeling that the statements are contradictory," not "your post."

(I endorse sunwillrise's comment [LW(p) · GW(p)] as a general response to this post; it's an unusually excellent comment. This comment is just me harping on a pet peeve of mine.)

So, within the ratosphere, it's well-known that every physical object or set of objects is mathematically equivalent to some expected utility maximizer

This is a wildly misleading idea which refuses to die.

As a meme within the ratosphere, the usual source cited is this old post by Rohin [LW · GW], which has a section titled "All behavior can be rationalized as EU maximization". When I complained to Rohin that "All behavior can be rationalized as EU maximization" was wildly misleading, he replied:

I tried to be clear that my argument was "you need more assumptions beyond just coherence arguments on universe-histories; if you have literally no other assumptions then all behavior can be rationalized as EU maximization". I think the phrase "all behavior can be rationalized as EU maximization" or something like it was basically necessary to get across the argument that I was making. I agree that taken in isolation it is misleading; I don't really see what I could have done differently to prevent there from being something that in isolation was misleading, while still being able to point out the-thing-that-I-believe-is-fallacious. Nuance is hard.

Point is: even the guy who's usually cited on this (at least on LW) agrees it's misleading.

Why is it misleading? Because coherence arguments do, in fact, involve a notion of "utility maximization" narrower than just a system's behavior maximizing some function of universe-trajectory. There are substantive notions of "utility maximizer", those notions are a decent match to our intuitions in many ways, and they involve more than just behavior maximizing some function of universe-trajectory. When we talk about "utility maximizers" in a substantive sense, we're talking about a phenomenon which is narrower than behavior maximizing some function of universe-trajectory.

If you want to see a notion of "utility maximizer" which is nontrivial, Coherence of Caches and Agents [LW · GW] gives IMO a pretty illustrative and simple example.

I agree with sunwillrise's comment.

This idea, the Texas Sharpshooter Utility Function, which looks at what happens and then paints the value 1 on it, is a surprisingly recurrent one on LW. But it does not work. It does not allow of any predictions. First you must see what happens; only then can you paint the target. Its present is uncomputable from its past.

The vast majority of planet-sized configurations of atoms are inimical to life, but here we are. Threats such as climate change, colliding asteroids, supervolcanoes, and AGI are not to be assessed by speculating on the generality of planet-sized configurations, but by considering the actually possible futures of the configuration we find ourselves in.