Comment by bender on We're all in this together · 2023-12-05T22:03:07.211Z · LW · GW

There is no such thing as an "aligned superintelligence". For a very simple reason: there is no way to make sure that a superior intelligence aligns to an inferior intelligence.

A superintelligence will see us humans the way us humans currently see other animals. Animals don't control humans, it's the other way around. This doesn't mean that a superintelligence will kill us though. We actually cannot know what a superintelligence will think and what it will decide, it's impossible, because we are inferior by definition, like a cat or a dog trying to understand what we are discussing here.

In other words, either you have superintelligence or you have alignment, you cannot have both.