AI-kills-everyone scenarios require robotic infrastructure, but not necessarily nanotech

Just a comment, not meant to contradict in the post:

Indeed, to kill literally everyone, you don't need to kill everyone at the same time. You only need to weaken/control civilization enough such that it can't threaten you. And then mop up the remainders when it becomes relevant. (But yes, you do need some way of operating in the real world at some point, either through robots or through getting some humans to do the work for you. But since enslavement worked in the past, the latter shouldn't be hard.)

I think there's no need for secrecy. If AI can develop a datacenter maintained by robots or other similar tech, human companies will be happy to buy and sell it, and help with the parts the AI can't yet do. Think of it as a "rising tide" scenario, where the robot sector of the economy outgrows the human sector. Money translates to power, as the robot sector becomes the highest bidder for security services, media influence, lobbying etc. When there comes a need to displace humans from some land and resources, it might look to humans less like a war and more like a powerful landlord pushing them out, with few ways to organize and push back. Similar to enclosures in early-modern England.

If humans are mostly dead (say a supervirus gets us), you can probably scavenge parts from our civilization for long enough to bootstrap to building better infrastructure. I think you'd just need a single robot and a roomful of equipment hooked up to a rooftop of solar panels, if you were clever. A single human has enough resources to supply that.

Strongly upvoted for thinking at a gears level about what happens once the AI has killed us all, rather than just saying that it would release a supervirus even though that sounds like it would be suicidal to the AI.

Obviously, nobody will be happy with an AI takeover anywhere.

I don't think this is obvious. When NetDragon appointed an AI CEO it didn't generate widespread negative reactions. https://www.analyticsinsight.net/chinese-game-company-appointed-an-ai-to-be-the-ceo/

If that trend continues I could imagine a future where a country has an AI president without it needing to be shrouded in secrecy. See also: Law of Undignified Failure

some counterpoints, in no particular order of strength:

• biological agents that follow any kind of actual science are a rather slow vector of extinction. You can either make a biological weapon extremely viral or extremely deadly, but not both, as the hosts need to live long enough to spread it.  Simple quarantine measures would slow that process to a crawl, especially if the plagues are obviously deadly and artificial, which would motivate more robust reaction than COVID did.
• Nukes are a poor choice for the AI to use to wipe out humans, since its the one weapon that AI and robots are just as (if not more) vulnerable to as humans are,due to EMP. If the nukes start flying back and forth, its not immediately obvious that AI nodes and robotic hubs would not be the worse off.
• Humans are already self-replicating general intelligence robots with a lot of built-in robustness and grid independence. Its not obvious to me that AI using mundane robotics/drones could easily outfight or even outbreed humans. 
• The total cost of outfitting a relatively fit adult human with a rifle, a full body Hazmat suit, and cable cutters is lower than the cost of producing a combat drone of similar capability. Such a human would be extremely dangerous to the robotic hub and supply chains, and the supercomputers the AI resides on. Without nanotech, AI is very susceptible to asymmetrical warfare, sabotage, arson, and plain old analog assault (dynamite, molotovs, cutting power cables, sledgehammer to the server etc.)
• there is no reason to believe the AI in this scenario would be invulnerable to hacking, or viruses. E-viruses are much easier and faster to produce, mutate, randomize, and spread than biological agents. Even if the AI itself would be too rugged to malware to be killed that way, the grid is not. If humans were facing extinction It is likely we would just destroy the global network with malicious software and strategic use of sledgehammers and bolt-cutters. After that, the war is reduced to 1970s style analog level, at which humans have advantage.

Mostly I agree with the logic of this post though it often sounds like a lot of human things that a human would do with the strength of an AI. It's hard for me to imagine a superintelligent immortal agent with no finite self caring about hoarding resources or exerting influence on animals way down the evolutionary ladder. To a superintelligence wouldn't actively wiping out humanity in the blink of an eye be no different than simply waiting for us to die out naturally in an arbitrary number of years? Is the assumption that the superintelligence would see humanity as a future threat?

Even so, I think this post is useful in exploring the types of AIs we'll see in the future and ways they could be misused. For supernerds like me it's so refreshing to see common scifi staples like misuse of genetic engineering and AI discussed in a serious and realistic way.

Who knows how many ways there are to remove Humanity as a threat? Neutrino bombs, waking up Yellowstone, pumping some chemical in the air that finishes global warming, maybe a particle accelerator can create more of some particle that's supposed to be uniformly distributed throughout the cosmos, locally changing some physical "constant" just enough that anything larger than a pug dies of liver failure.

And then it steers a nuclear submarine to train an octopus, or uses smartphones to groom some survivors, or it waits for a more useful species to evolve, or it reconfigured a smart-circuit into an ansible and is getting paid by the closest aliens for saving them a war 50 million years hence, or it turns a quantum computer into an outcome pump and directly probablity-warps for its utility function.

I think the core point to consider in terms of this are:

1. is the AI simply indifferent due to misalignment, or actually malicious? The latter seems more improbable (it's a form of alignment after all!) but if the AI does not need nor plan to survive its extermination of us then we've got much less chances, and it's got a lot more paths open to it;

2. does the AI even need to kill everyone? Collapsing civilisation to the point where mounting a coordinated, effective response is impossible might be enough. A few stone age primitives won't pose a threat to an army of superintelligent robots, they can be left around until such time comes that environmental degradation kills them anyway (I can't imagine AI caring much to keep the Earth unpolluted or its air breathable);

3. would the AI start a fight if it thinks it has a chance not to win it? Depends on how its reward estimation works, is it one for risky bets that win big or a careful planner? I suppose the best (albeit somewhat funny and scary to think about) outcome would be an extremely careful AI that might consider killing us, but it's just gotta be sure, man, and it never feels like it has enough of a safety margin to take the plunge. Jokes aside, for any scenario other than "superintelligent AI with powerful nanotech at its fingertips" that's gotta count. Going to war might have benefits but will also come with risks. And at the very least the AI needs to be able to set up infrastructure to keep maintenance, replacements, energy supply etc. ongoing to its own hardware.

4. that said, if AI started developing efficient robotics that revolutionize productivity, would we say no? And if those robots populated every factory and key infrastructure node, would we stop there? And if we tried to design a kill switch, just to be safe, even though at that point we'd be so dependent on the robots that activating it would just be almost as harmful to us as to the AI, would a smart enough AI not be able to get around it? So honestly a scenario in which AI controls just enough to support itself and then decides it can simply collapse our civilisation with a moderately virulent super-plague would eventually be reached no matter what.

AI could manage the human race and keep everybody so happy that we don't care that our reproduction levels are beneath replacement levels and then they can just play the long game. Happily sunset the human race.

AI can also become Singleton without killing humans and without robots, just by enslaving them.

Well if this is the case then the AI can get all the robots it wants afterwards.

”Whom the gods would destroy, they first make mad.”

One way to kill everyone would be to turn the planet into a Peoples Temple and have everyone drink the kool-aid.

I think even those of us who think of ouselves as psychologically robust will be defenseless against the manipulations of a future GPT-9.

You're an AI. You've got an internet connection. You can make copies of yourself. You're capable of reading instructions for how to take over insecure computers. 

Within about a minute you're on almost every computer in the world. 

Fight that.