Non-orthogonality implies uncontrollable superintelligence

It sounds implausible when you put it like that, but suppose the only practical way to build a superintelligence is through some method that severely constrains the possible goals it might have (e.g., evolutionary methods, or uploading the smartest humans around and letting them self-modify), and attempts to build general purpose AIs/oracles/planning tools get nowhere (i.e., fail to be competitive against humans) until one is already a superintelligence.

Maybe when Bostrom/Armstrong/Yudkowsky talk about "possibility" in connection with the orthogonality thesis, they're talking purely about theoretical possibility as opposed to practical feasibility. In fact Bostrom made this disclaimer in a footnote:

The orthogonality thesis implies that most any combination of final goal and intelligence level is logically possible; it does not imply that it would be practically easy to endow a superintelligent agent with some arbitrary or human-respecting final goal—even if we knew how to construct the intelligence part.

But then who are they arguing against? Are there any AI researchers who think that even given unlimited computing power and intelligence on the part of the AI builder, it's still impossible to create AIs with arbitrary (or diverse) goals? This isn't Pei Wang's position, for example.

I don't think I've seen that particular reversal of the position before. Neat.

Yep. I'm calling that the "no Oracle, no general planning" position in my paper.

build any computational system which generates a range of actions, predicts the consequences of those actions relative to some ontology and world-model, and then selects among probable consequences using criterion X.

Nothing mysterious here: this naive approach has incredibly low payoff per computation, and even if you start with such system, and get it to be smart enough to make improvements, the first thing it'll be improving is changing it's architecture.

If I gave you 10^40 flops, which probably can support 'super intelligent' mind, your naive approach would still be dumber than a housecat on many tasks. For some world evolution & utility, you can do inverse of the 'simulate and choose' much better (think towering exponents times better) than brute-force 'try different actions'. In general you can't. Some functions are easier to find inverse of, than others. A lot easier.

Sounds uncontroversial, to me. I wouldn't expect to be able to create a non-broken AI, even a comparitively trivial one, that thinks 1+1=3. On the other hand, I do think I could create comparitively trivial AIs that leverage their knowledge of arithmetic to accomplish widely varying ends. Simultaneous Location and Mapping, for example, works for a search and rescue bot or a hunt/kill bot.

Not exactly true... You need to conclude "can be taken by its creators to make it return a different answer while it remains an Oracle". With that caveat inserted, I'm not sure what your point is... Depending on how you define the terms, either your implication is true by definition, or the premise is agreed to be false by pretty much everyone.

It's useful for hitting certain philosophers with. Canonical examples: moral realists sceptical of the potential power of AI.

Assuming it knew where that threshold was ahead of time.

So you think that true moral behavior excludes choice? (More generally, once someone chooses their morality, no more choices remain to be made?)

I think so. What choice is there in the field mathematics? I don't see that mathematicians ever had any choice but to eventually converge at the same answer given the same conjecture. Why would that be different given an objective morality?

I thought that is what the Aumann's agreement theorem states and the core insight of TDT, that rational agents will eventually arrive at the same conclusions and act accordingly.

The question would be what happens if a superintelligence was equipped with goals that contradict reality. If there exists an objective morality and the goal of a certain AI was to maximize paperclips while maximizing paperclips was morally wrong then that would be similar to the goal of proving that 1+1=3 or to attain faster-than-light propagation.

ETA I suppose that if there does exist a sort of objective morality but it is inconsistent with its goals, then you would end up with an unfriendly AI anyway. Since such an AI would attempt to pursue its goals given the small probability that there is no objective morality for one reason or the other.

Let's give it a try... In the space of computable functions, there is a class X that we would recognize as "having goal G". There is a process SI we would identify as self-improvement. Then converge implies that for nearly any initial function f, the process SI will result in f being in X.

If you want to phrase this in an updateless way, say that "any function with property SI is in X", defining X as "ultimately having goal G".

If you want a complete, coherent account of what non-orthogonality would be, you'll have to ask one of its proponents.

What type of reasoning would you prefer to be used when talking about superintelligences?