Six Plausible Meta-Ethical Alternatives

Given your terminology without dispute, and then ignoring all debates about what ordinary human language refers to, yes 3-4. I think we have enough knowledge at this point to reject internalism out of hand, and if I were going to dispute your terminology then I would say that 2 is also internalism, just weaker internalism, and that the internalism/externalism debate shouldn't ought to be said to have things to do with realism, see e.g. "An Introduction to Contemporary Metaethics" in which externalist theories are still classified as realistic; I think a lot of what feels like a naively necessary quality of cognitivism/realism is actually particular kinds of non-naturalism in the standard schema. E.g. I would consider "a fact such that knowledge of it is inherently motivating to every possible mind" to be non-reductionist because it's a kind of Mind Projection Fallacy of the quality of motivating-ness that facts have to us, but that has nothing to do with whether our own morals have the property of cognitivism/realism. If I were further going to dispute terminology, I would replace a lot of what you would call "facts" with what I would call "validities" and try to ground them in values every time they involved any kind of preference or betterness or choice, since the laws of physics contain no little < or > signs. But on your scheme, yes 3-4.

I found this a useful summary of (at least some of) the possibilities.

A small issue is that I find that it's clearer to avoid using the word 'should' as much as possible when discussing meta-ethics, since leaving the goal implied ('should' do X for what end?) can be ambiguous. I think it's clearer to talk about what will happen if a being uses such-and-such a decision theory/ethics/etc, and when translating that to 'should' to be very clear what goal is being targeted.

edit: To give a specific example, using an unqualified 'should' can lead to (intentionally or unintentionally) equivocating between the values of the being under discussion and the values of its species/humanity/the author/some other ideal.

7 We don't have nearly enough knowledge to reason about multiverses and intergalactic civilizations and picking 6 possibilities which you happened to think of is privileging 6 hypotheses (described in poorly defined terms) out of a countless number. Maybe we ought to concentrate on models which can be tested/simulated/falsified instead.

“Most intelligent beings in the multiverse share similar preferences. This came about because there are facts about what preferences one should have”

Notably, this could come about for other reasons, such as evolutionary.

I would characterize Eliezer's metaethics slightly differently; I'd say he believes that 'moral facts' as conceived of by humans are a human-specific notion with no relevance to any other type of mind, but that they exist, and that he would place it between 2 and 3. Or more specifically, he'd endorse 3 with the caveat that if you restrict the domain of 'everyone' to humans, 2 would also be true.

I'd tentatively agree but don't feel informed enough to have a strong opinion or motivated to form one.

facts about what preferences one should have

The "should" here is not defined clearly enough (or at all!), even though this seems to be the central point in the debate. We have the intuition that the question is meaningful, but I suspect that it really isn't. I don't understand what this could possibly mean -- expect for trivial cases where you already specify a goal. I would leave it at "Most intelligent beings in the multiverse share similar preferences", with perhaps adding a qualifier like "evolved/intelligently designed". Note that this would then be answering a slightly different question than 3., 4. and 5.

My own view is roughly a 4.3 on the spectrum from 4. to 5.

The way "complexity of value" is used by Eliezer seems to suggest that he adheres to view 3, although I could well imagine him also going for 4 or 5.

I'm unsure about 6; I suspect/hope that you can just define "winning" clearly enough in whatever utility function you're interested in and decision theory will sort itself out. But maybe it's more complicated.

I'm not getting the essential difference between (3) and (4). It seems like (4) is just a special case of (3), in that (4)'s "process of reflection [that] is convergent at the individual level" could just be (3)'s "how to translate non-preferences ... into preferences".

Is the difference that, if (3) held, then there could be only one correct "process of reflection" of the kind described in (4), so that this one process would be the only correct path for all pre-rational intelligent agents to take to become rational?

Thanks for posting this! Is this list drawn up hodge-podge, or is there some underlying process that generated it? How likely do you think it is to be exhaustive?

It looks like your list is somewhat methodical, arising from combinations of metaethical desiderata/varyingly optimistic projections of the project of value loading?

Are you able to put probabilities to the possibilities?

For example, it turns out there is no one decision theory that does better than every other decision theory in every situation, and there is no obvious or widely-agreed-upon way to determine which one "wins" overall.

I'm very confident that no decision theory does better than every other in every situation, insomuch as the decision theories are actually implemented. For any implementing agent, I can put that agent in the adversarial world where Omega instantly destroys any agent implementing that decision theory, assuming it has an instantiation in some world where that makes sense (e.g. where 'instantly' and 'destroy' make sense). This is what we would expect on grounds of Created Already In Motion and general No Free Lunch principles.

The only way I currently see to resolve this is something along the lines of having a measure of performance over instantations of the decision theory, and some scoring rule over that measure over instantiations. Might be other ways, though.

Eliezer says that there are moral facts about what values every intelligence in the multiverse should have, but only humans are likely to discover these facts and be motivated by them.

Just to check, you mean

(A) For all I, there exists some M such that I should observe M

and not

(B) There exists some set of moral facts M, such that for each intelligence I, I should observe M

right?

So is this is roughly one aspect of why MIRI's position on AI safety concerns are different to similar parties? - that they're generally more sympathetic to possibilities futher away from 1 than their peers? I don't really know, but that's what the pebblesorters/value-is-fragile strain of thinking seems to suggest for me.

Well, then I'll attempt to mindread, and guess Eliezer's position is more like 5.

I'm not sure how much that's just an overly complicated way of saying "my own position is ~5, and projection is a thing," though. Does this mean I can reverse-mindread you and guess that, if you absolutely had to pick one of the six, you would pick 3 or maybe 4?

Since Eliezer somewhere said he wants FAI to extrapolate multiple past versions of you - I think he said, as many as possible - he seems to allow for 5.