Contra Nora Belrose on Orthogonality Thesis Being Trivial

The real issue with the orthogonality thesis, at least for the defensible version, is that it makes a claim that I think is true if we quantify over all possible intelligences, but the problem is that this has basically ~no implications for AI safety, even if rationalists are right, because it is far too weak to support AI risk claims.

It only says that an AI can have arbitrary goals, not that it is likely or will have arbitrary goals. You can justify any probability of AI having bad goals from probability 0 to probability 1, and everything in between.

Outside of history, it is completely useless, since it doesn't constrain your anticipation very much.

It feels way to easy to flip the sign:

« I think the orthogonality thesis is wrong. For instance, without rejecting the orthogonality thesis, one might think we should stop constructing AGI!

You might think this is stupid, but some significant people believe it. Clearly the orthogonality thesis is nontrivially confusion for cases like this. »

I think there are two parts that together make one important point.

The first part is that an intelligence can have any goal (a.k.a. the orthogonality thesis).

Second part is that most arbitrarily selected goals are bad. (Not "bad from the selfish short-sighted perspective of the puny human, but good from the viewpoint of a superior intellect", but bad in the similar sense how rearranging the atoms in your body randomly could hypothetically cure your cancer, but almost certainly will kill you instead.)

The point is, as a first approximation, "don't build a super-intelligence with random goals, expecting that as it gets smart enough it will spontaneously converge towards good".

If you have a way to reliably specify good goals, then you obviously don't have to worry about orthogonality thesis. As far as know, this is not the case we have now.

*

I wrote this before seeing the actual tweets. Now that I saw them, it seems to me that Yudkowsky basically agrees with my interpretation of him, which is based on things he wrote a decade ago, so at least the accusation about "rewriting history" is false.

(The part about "trivial, false, or unintelligible", uhm, seems like it could just as well be made about e.g. natural selection. "What survives, survives" is trivial, if you don't also consider mutations. So do we have a motte and bailey of natural selection, where the motte just assumes selection without discussing the mechanism of mutations, and the bailey that includes the mutations? Does this constitute a valid argument against evolution?)

In fact it seems that the linked argument relies on a version of the orthogonality thesis instead of being refuted by it:

For almost any ultimate goal - joy, truth, God, intelligence, freedom, law - it would be possible to do it better (or faster or more thoroughly or to a larger population) given superintelligence (or nanotechnology or galactic colonization or Apotheosis or surviving the next twenty years).

Nothing about the argument contradicts "the true meaning of life" -- which seems in that argument to be effectively defined as "whatever the AI ends up with as a goal if it starts out without a goal" -- being e.g. paperclips.

In the sense that the Orthogonality Thesis considers goals to be static or immutable, I think it is trivial.

I've advocated a lot for trying to consider goals to be mutable, as well as value functions being definable on other value functions. And not just that it will be possible or a good idea to instantiate value functions this way, but also that they will probably become mutable over time anyway.

All of that makes the Orthogonality Thesis - not false, but a lot easier to grapple with, I'd say.

FWIW I strongly agree with what I thought was the original form of the orthogonality thesis - i.e. that a highly intelligent agent could have arbitrary goals - but am was confused by Eliezer's tweet linked by Nora Belrose's tweet post.

Eliezer:

if a superintelligence can calculate which actions would produce a galaxy full of paperclips, you can just as easily have a superintelligence which does that thing and makes a galaxy full of paperclips

This seems to me to be trivially correct if what is meant by "calculate which actions" means calculating down to every last detail. 

On the other hand, if the claim were something like that we can't know anything what an AI would do without actually running it (which I do not think is the claim, just stating an extreme interpretation), then it seems obviously false.

And if it's something in between, or something else entirely, then what specifically is the claim?

It's also not clear to me why that trivial-seeming claim is supposed to be related to (or even a stronger form than) other varieties of the "weak" form of the orthogonality thesis.  Actually that's obvious, I should have bothered thinking about it, like, at all. So, yeah, I can see why this formulation is useful - it's both obviously correct and does imply that you can have something make a galaxy full of paperclips.