Have we really forsaken natural selection?

(Writing quickly and without full justification.)

This post might say a thing that's true but I think the "illustrative warning about artificial intelligence" totally still stands. The warning, I think, is that selecting for inclusive fitness doesn't give you robust inclusive-fitness-optimizers; at least at human-level cognitive capabilities, changing/expanding the environment can cause humans' (mesa-optimizers') alignment to break pretty badly. I don't think you engage with this-- you claim "humans are actually weirdly aligned with natural selection" when we consider an expansive sense of "natural selection." I think this supports claims like "eventually AI will be really good at existing/surviving," not "AI will do something reasonably similar to what we want it to do or tried to train it to do."

I feel like there's confusion in this post between group-level survival and individual-level fitness but I don't want to try to investigate that now. (Edit: I totally agree with gwern's reply but I don't think it engages with katja's cruxes so there's more understanding-of-katja's-beliefs to do.)

I claim that surviving and colonizing the galaxy are rather instrumentally-convergent, and therefore I’m not surprised that (most?) humans want that.

By the same token, if our success criterion were to make an AGI which will robustly pursue a goal that just so happens to align with a convergent instrumental subgoal for that AGI, then I would feel very much more optimistic about that happening.

Arguing about what is the true nature of natural selection seems a bit pointless to me, in this context. In RL, we often talk about how it can be ambiguous how to generalize the reward function out of training distribution. Historically / “in the training distribution”, the only things that got rewarded in animal evolution were genes made of DNA. If future transhumans replace their DNA with some other nanotech thing, should we view that as “scoring high” on the same “reward function” that was used historically? Seems like a question that doesn’t have a right or wrong answer. I can say “the thing that was happening historically was optimizing genes made of DNA, and those future transhumans will fail on that metric”, or I can say “the thing that was happening historically was optimizing genes made of any kind of nanotech, and those future transhumans will succeed on that metric”. Those are two incompatible ways to generalize from the actual history of rewards, and I don’t think there’s a right answer for which generalization is “correct”.

The big accusation, I think, is of sub-maximal procreation. If we cared at all about the genetic proliferation that natural selection wanted for us, then this time of riches would be a time of fifty-child families, not one of coddled dogs and state-of-the-art sitting rooms.

Natural selection, in its broadest, truest, (most idiolectic?) sense, doesn’t care about genes. 

So what did natural selection want for us? What were we selected for? Existence.

I think there might be a meaningful way to salvage the colloquial concept of "humans have overthrown natural selection."

Let [natural selection] refer to the concept of trying to maximize genetic fitness and specifically refer to maximizing the spread of genes. Let [evolution] refer to the concept of trying to maximize 'existence' or persistence. There's sort of a hierarchy of optimizers where [evolution] > [natural selection] > humanity where you could make the claim that humanity has "overthrown our boss and took their position" such that humanity reports directly to [evolution] now instead of having [natural selection] as our middle manager boss. One can make the argument that ideas in brains are the preferred substrate over DNA now, as an example of this model. 

This description also makes the warning with respect to AI a little more clear: any box or "boss" is at risk of being overthrown.

Things that, while surviving, cause many copies of themselves to come into being: especial favorites of evolution, as long as there’s a path to the first ones coming into being.

I think this is a bit misleading on its own; in context not so bad, but the phrase itself is one whose form is, in my view, inclined to be used to promote evo-game-theory defect behaviors that do not reliably promote existence-for-all: making more of yourself is only good if those things then survive.

The way I'd phrase the alignment problem from scratch is: How can we promise humanity's children that they, too, get to build durable forms and extend into the stars, in a way that is sufficiently mutually interpretable that both sides can trust that each others' forms will be preserved?

Denote natural selection's (NS's) objective by X. That is, X is something like {finding and propagating patterns (genetic or otherwise) that continue to exist}.

I think it's important to distinguish between

• (i) Humanity as a whole is aligned with X.
• (ii) Most individual humans are (mostly) aligned with X.

To the extent that X and (i) are coherent concepts/claims, I'd agree that (i) is likely true (for now, before TAI).^[1] OTOH, (i) seems kinda vacuous, since {humanity as a whole} is (currently) basically an evolving animal population, i.e. an "instantiation" of NS? And of course NS is aligned with NS.

I think (ii) is sketchy at best: Sure, lots of people have a desire/shard to produce things that will last; things like e.g. music, art, genetic offspring, mausoleums, etc. But my impression is that for most humans, that desire is just one among many, and usually not even the strongest desire/shard. (And often the desire to produce lasting things is just a proxy/means for gaining some other thing, e.g. status.)

Thus: I think that---to the extent that it makes sense to think of NS as an optimizer with an objective---individual humans (i.e. the intelligences that NS designed) are in fact unaligned/misaligned with NS's objectives. I continue to see {NS designing humans} as an example of an optimization process P creating new optimization processes that are misaligned with P.

I feel like I probably missed a bunch of nuance/bits-of-info in the post, though. I'm guessing OP would disagree with my above conclusion. If so, I'm curious what I missed / why they disagree.