An overall schema for the friendly AI problems: self-referential convergence criteria

"Evolve" has (at least) two meanings. One is the Darwinian one where heritable variation and selection lead to (typically) ever-better-adapted entities. But "evolve" can also just mean "vary gradually". It could be that values aren't (or wouldn't be, in a posthuman era) subject to anything much like biological evolution; but they still might vary. (In biological terms, I suppose that would be neutral drift.)

The failures have to die out.

I'm not sure that's true. Imagine some glorious postbiological future in which people (or animals or ideas or whatever) can reproduce without limit. There are two competing replicators A and B, and the only difference is that A replicates slightly faster than B. After a while there will be vastly more of A around than of B, even if nothing dies. For many purposes, that might be enough.

Economics. Posthumans still require mass/energy to store/compute their thoughts.

If there are mistakes made or the environment requires adaptation, a sufficiently flexible intelligence can mediate the selection pressure.

How can a numerical measure of a subjective state of consciousness be valuable if nobody experiences the total numerical measure?

The mere addition paradox also manifests for a single person. Imagine the state you are in. Now imagine if it can be (subjectively) improved by some means (e.g. fame, company, drugs, ...). Keep going. Odds are, you would not find a maximum, not even a local one. After a while, you might notice that, despite incremental improvements, the state you are in is actually inferior to the original, if you compare them directly. Mathematically, one might model this as the improvement drive being non-conservative and so no scalar map from states to scalar utility exists. Whether it is worth pushing this analogy any further, I am not sure.

It involves reasoning about facts rather than metaphysics.

Well, attempting to account for your grammar and figure out what you meant...

A correct model has to get a bunch of counterfactuals correct, and not just match an empirical dataset.

Yes, and? Causal modelling techniques get counterfactuals right-by-design, in the sense that a correct causal model by definition captures counterfactual behavior, as studied across controlled or intervened experiments.

I mean, I agree that most currently-in-use machine learning techniques don't bother to capture causal structure, but on the upside, that precise failure to capture and compress causal structure is why those techniques can't lead to AGI.

what is the difference between modelling it currently, and solving moral philosophy?

I think it's more accurate to say that we're trying to dissolve moral philosophy in favor of a scientific model of human evaluative cognition. Surely to a moral philosopher this will sound like a moot distinction, but the precise difference is that the latter thing creates and updates predictive models which capture counterfactual, causal knowledge, and which thus can be elaborated into an explicit theory of morality that doesn't rely on intuition or situational framing to work.

The reason we don't value honoring our ancestors is that we don't believe they continue to exist after death

This could be a case of typical mind fallacy. I can point to a number of statistical studies that show that a large number of Westerners claim that their ancestors do continue to exist after death.

Anyone who believes that some sort of heaven or hell exists.

And a lot of these people nonetheless don't accord their ancestors all that much in the way of honour...

That's shifting the definition of "siren world" from "something which looks very nice when simulated in high-resolution but has things horrendously wrong on the inside" to a very standard "Human beings imagine things in low-resolution and don't always think them out clearly."

You don't need to pour extra Lovecraft Sauce on your existing irrationalities just for your enjoyment of Lovecraft Sauce.

You're missing my point, which is that proposing you've got "an AI" (with no dissolved understanding of how the thing actually works underneath what you'd get from a Greg Egan novel) which "simulates" possible "worlds" is already engaging in several layers of magical thinking, and you shouldn't be surprised to draw silly conclusions from magical thinking.

Wireheading isn't a siren world, though. The point of the concept is that it looks like what we want, when we look at it from the outside, but actually, on the inside, something is very wrong. Example: a world full of people who are always smiling and singing about happiness because they will be taken out and shot if they don't (Lilly Weatherwax's Genua comes to mind). If the "siren world" fails to look appealing to (most) human sensibilities in the first place, as with wireheading, then it's simply failing at siren.

The point is that we're supposed to worry about what happens when we can let computers do our fantasizing for us in high resolution and real time, and then put those fantasies into action, as if we could ever actually do this, because there's a danger in letting ourselves get caught up in a badly un-thought-through fantasy's nice aspects without thinking about what it would really be like.

The problem being, no, we can't actually do that kind of "automated fantasizing" in any real sense, for the same reason that fantasies don't resemble reality: to fully simulate some fantasy in high resolution (ie: such that choosing to put it into action would involve any substantial causal entanglement between the fantasy and the subsequent realized "utopia") involves degrees of computing power we just won't have and which it just wouldn't even be efficient to use that way.

Backwards chaining from "What if I had a Palantir?" does lead to thinking, "What if Sauron used it to overwhelm my will and enthrall me?", which sounds wise except that, "What if I had a Palantir?" really ought to lead to, "That's neither possible nor an efficient way to get what I want."

In absolute terms, maybe, but that doesn't stop it being relatively better.

All of those could prevent reflective equilibria, while still being compatible with the ability for extensive self-modification.

It's possible. They feel very unstable, though.