Oversight Misses 100% of Thoughts The AI Does Not Think

[writing quickly, sorry for probably being unclear]

If the AI isn't thinking about how to deceive the humans who are operating it, it seems to me much less likely that it takes actions that cause it to grab a huge amount of power.

The humans don't want to have the AI grab power, and so they'll try in various ways to make it so that they'll notice if the AI is trying to grab power; the most obvious explanation for why the humans would fail at this is that the AI is trying to prevent them from noticing, which requires the AI to think about what the humans will notice.

At a high enough power level, the AI can probably take over the world without ever explicitly thinking about the fact that humans are resisting it. (For example, if humans build a house in a place where a colony of ants lives, the human might be able to succeed at living there, even if the ants are coordinatedly trying to resist them and the humans never proactively try to prevent the ants from resisting them by eg proactively killing them all.) But I think that doom from this kind of scenario is substantially less likely than doom from scenarios where the AI is explicitly thinking about how to deceive.

You probably don't actually think this, but the OP sort of feels like it's mixing up the claim "the AI won't kill us out of malice, it will kill us because it wants something that we're standing in the way of" (which I mostly agree with) and the claim "the AI won't grab power by doing something specifically optimized for its instrumental goal of grabbing power, it will grab power by doing something else that grabs power as a side effect" (which seems probably false to me).

Forget about "sharp left turn", you must be imagining strapping on a rocket and shooting into space.

(I broadly agree with Buck's take but mostly I'm like "jeez how did this AGI strap on a rocket and shoot into space")

I would expect a superhuman AI to be really good at tracking the consequences of its actions. The AI isn't setting out to wipe out humanity. But in the list of side effects of removing all oxygen, along with many things no human would ever consider, is wiping out humanity. 

AIXI tracks every consequence of its actions, at the quantum level. A physical AI must approximate, tracking only the most important consequences. So in its decision process, I would expect a smart AI to extensively track all consequences that might be important.

I might disagree with this. It seems like to achieve anything, the AI's planning process will have to be robust to the perturbations of humans strategically disrupting its plans, because otherwise it just gets turned off. This seems very close to explicitly thinking about how to counter human plans.

My rephrasing of the question: can a fight between real-life optimizers be well-modeled by which one has "more optimization power" along a single dimension, or does the winning agent have to model and counter the losing agents's strategies?

• arguments for:
  • you can win aerial dogfights by disrupting the other craft's OODA loop rather than a specific strategy
  • Skill in many adversarial games seems to be well-modeled by a single ELO score rather than multiple dimensions
• argument against:
  • The good regulator theorem [LW · GW] says there's some correspondence between the actions of an optimizer and the structure of its environment, which seem likely to take the form of explicit planning
  • Humans can defeat non-robust optimization processes that have huge amounts of optimization power in one distribution just by putting them in a different distribution. Pests multiply until humans deploy a targeted pesticide; temperature equilibrates between indoors and outdoors until we install an air conditioner.

As some other commenters have said, the analogy with other species (flowers, ants, beavers, bears) seems flawed. Human beings are already (limited) generally intelligent agents. Part of what that means is that we have the ability to direct our cognitive powers to arbitrary problems in a way that other species do not (as far as we know!). To my mind, the way we carelessly destroy other species' environments and doom them to extinction is a function of both the disparity in both power and the disparity in generality, not just the former. That is not to say that a power disparity alone does not constitute an existential threat, but I don't see the analogy being of much use in reasoning about the nature of that threat.

If the above is correct, perhaps you are tempted to respond that a sufficiently advanced AI would replicate the generality gap as well as the power gap. However, I think the notion of generality that is relevant here (which, to be sure, is not the only meaningful notion) is a 0 to 1 phase transition. Our generality allows us to think about, predict, and notice things that could thwart our long term collective goals. Once we start noticing such things, there is no level of intelligence an unaligned third-party intelligence can reach which somehow puts us back in the position of not noticing, relative to that third-party intelligence.

Just commented on IRC the other day that 

The mode of human extinction will not be "I must tile the universe with paperclips and humans are in the way", but more like "Oops, I stepped on that bug"

Right now, you're writing a "why not just" series, and under that headline, it makes sense to treat these proposals individually.

I'd also appreciate it if you spent some time addressing the idea that successful AI safety will be a synthesis of these strategies, and perhaps others yet to be found. Right now, I can't update my perceptions of the value of these individual proposals very much, because my baseline expectation is that AI safety will rely on a combination of them.

I also expect that figuring out what's achievable in combination will require significant technical refinements of each individual proposal. For that reason, it doesn't surprise me that the manner of synthesis is unclear. Generally, we let a thousand flowers bloom in the world of science. While I don't think this is appropriate for very hazardous technologies, it does seem appropriate for AI safety. Of course, I am only an interested observer. Just chiming in.

I would presume that the AI would know that humans are likely to try to resist a takeover attempt, and to have various safeguards against it. It might be smart enough to be able to overcome any human response, but that seems to only work if it actually puts that intelligence to work by thinking about what (if anything) it needs to do to counteract the human response. 

More generally, humans are such a major influence on the world as well as a source of potential resources, that it would seem really odd for any superintelligence to naturally arrive on a world-takeover plan without at any point happening to consider how this will affect humanity and whether that suggests any changes to the plan. 

If you assume that the overseer could robustly detect that the AI wants to kill humans, they could probably just as robustly detect that it is not aiming to operate under the constraint of keeping humans alive, happy etc while optimizing for whatever it is trying to optimize.

This has far too many assumptions, and the final claims are too strongly stated. One of the important assumptions being that it can trivially destroy humanity without trying. Comparing humanity to a specific kind of flower is ridiculous. Even a strongly superhuman AI would be more like humans vs rats than humans vs. a flower.  Could humanity eliminate all rats in the world if we wanted to? Maybe, but we wouldn't accomplish much of anything else.

Another assumption is that whatever is overseeing the AI is vastly stupider than the AI, even with all the tools made for the purpose. If you can make a superhuman AI, you can make a superhuman AI and Human system too (which is an easier task).

A lot of the reason humans managed to conquer the natural world is because we are essentially super-cooperative, in the sense that humans can form groups of 10-1000 or more and not have the social system totally break down. We aren't much more intelligent than other animals, we are way more cooperative in groups than other animals.