The way AGI wins could look very stupid

I think this extrapolates far from one example and I'm not sure the example applies all that well. 

Old engines played ugly moves because of their limitations, not because playing ugly moves is a super power. They won anyway because humans cannot out calculate engines. 

AlphaZero plays beautiful games and even todays standard engines don't play ugly or dumb looking moves anymore. I think in the limit superior play will tend to be beautiful and elegant. 

If there is a parallel between early super human chess and AGI takeover it will be that AGI uses less than brillant strategies that still work because of flawless or at least vastly superhuman execution. But these strategies will not look dump or incomprehensible. 

This seems like a generalization of something that humans are also guilty of. The way we win against other animals also can look kind of dumb from the perspective of those animals.

Suppose you're a cheetah. The elegant, smart way to take down pray is chase them down in a rapid sprint. The best takedowns are ones where you artfully outmaneuver your prey and catch them right at a moment when they think they are successfully evading you.

Meanwhile you look on humans with disdain. They can take down the same prey as you, but they do it in dumb ways. Sometimes they throw things at the prey to kill it. Other times they run it down until it collapses from heat exhaustion. That works, you have to admit, but it seems ugly and inelegant.

Because we have greater general intelligence we can try strategies that other animals can't. These strategies look really weird to them on their own values. I read your point as saying we're in a similar relationship to AGI: it will be able to do things we can't and so will sometimes solve problems in weird, inelegant ways that nonetheless work.

When humans choose their actions, they often think about the impact those actions will have on their status. They often don't play to win; they play to impress the observers. (Yes, winning is impressive, but if winning using certain moves is more impressive than winning using some other moves, even if the probability of the latter strategy is greater, many will choose the former.) AI would not care about status, if it expects that humans will soon be dead. AI would not over-complicate things when not necessary, because it is not trying to signal its superhuman intelligence to a hypothetical observer.

For example, people keep making fun of the "Nigerian prince" scams, but they continue to exist, because apparently they work. Who knows, maybe the same technology can be used to destroy humanity. Like, send everyone an SMS at the same time, asking them to follow your commands, and promising them millions of dollars if they obey. Ask for something simple and harmless first to train compliance, then ask them to do something such that if 1 person in 1000 does it, the civilization will collapse. Maybe 1 in 1000 will actually do it.

(Among other reasons, this plan sounds stupid, because the phone operators could trivially stop it by blocking the SMS functionality for a while. Yeah, but maybe if you approach everyone at the same time and the whole action takes less than an hour, they won't react quickly enough. In hindsight, it will be obvious that disabling SMS quickly would have been the right move, but at the moment... it will seem just like a weird prank, and disabling SMS will seem like a very serious move with possible impact on profits that requires approval of the important people; and if that happens on a weekend, people will hesitate to bother the important ones.)

Also, you only need to destroy humanity once, so if you try dozen stupid plans in parallel, even if each of them is more likely to fail than to succeed...

I think when we say that an adversarial attack is "dumb" or "stupid" what we are really implying is that the hack itself is really clever but it is exploiting a feature that is dumb or stupid.  There are probably a lot of unknown-to-us features of the human brain that have been hacked together by evolution in some dumb, kludgy way that AI will be able to take advantage of, so your example above is actually an example of the AI being brilliant but us humans being dumb.  But I get what you are saying that that whole situation would indeed seem "dumb" if AI was able to hack us like that.  

This reminds me of a lecture 8-Bit Guy did on phone phreaking in the 1980s. "How Telephone Phreaking Worked."  Some of those tricks do indeed seem "dumb," but it's dumb more in the sense that the telephone network was designed without sufficient forethought to be susceptible to someone playing a blue whistle that you could get from a Captain Crunch cereal box that just happened to play the correct 2600 hz frequency to trick phones into registering a call as a toll-free 1-800 call.  The hack itself was clever, but the design it was preying upon and the overall situation was kinda dumb. 

Solving for "A viable attack, maximum impact" given an exhaustive list of resources and constraints seems like precisely the sort of thing GPT-4-level AI can solve with aplomb when working hand in hand with a human operator. As the example of shooting a substation, humans could probably solve this in a workshop-style discussion with some Operations Research principles applied, but I assume the type of people wanting to do those things probably don't operate in such functional and organized ways. When they do, it seems to get very bad.

The LLM can easily supply cross-domain knowledge and think within constraints. With a bit of prompting and brainstorming, one could probably come up with a dozen viable attacks in a few hours. So the lone bad actor doesn't have to assemble a group of five or six people who are intelligent, perhaps educated, and also want to do an attack. I suspect the only reason people already aren't prompting for such methods and then setting up automation of them is the existence of guardrails. When truly opensource LLMs get to GPT-4.5 capability and good interfaces to the internet and other software tools (such as phones), we may see a lot of trouble.  Fewer people would have the drive and intellect needed (at least early on) to carry out such an attack, but those few could cause very outsized trouble.

TL;DR:  The "Fun" starts waaaaaaay before we get to AGI.

Thomas Griffiths' paper Understanding Human Intelligence through Human Limitations argues that the aspects we associate with human intelligence – rapid learning from small data, the ability to break down problems into parts, and the capacity for cumulative cultural evolution – arose from the 3 fundamental limitations all humans share: limited time, limited computation, and limited communication. (The constraints imposed by these characteristics cascade: limited time magnifies the effect of limited computation, and limited communication makes it harder to draw upon more computation.) In particular, limited computation leads to problem decomposition, hence modular solutions; relieving the computation constraint enables solutions that can be objectively better along some axis while also being incomprehensible to humans: 

A key attribute of human intelligence is being able to break problems into parts that can individually be solved more easily, or that make it possible to reuse partial solutions discovered through previous experience. These methods for making computational problems more tractable such ubiquitous part of human intelligence that they seem to be an obligatory component of intelligence more generally. One example of this is forming subgoals. The early artificial intelligence literature, inspired by human problem-solving, put a significant emphasis on reducing tasks to a series of subgoals. 

However, forming subgoals is not a necessary part of intelligence, it’s a consequence of having limited computation. With a sufficiently large amount of computation, there is no need to have subgoals: the problem can be solved by simply planning all the way to the final goal. 

Go experts have commented that new AI systems sometimes produce play that seems alien, precisely because it was hard to identify goals that motivated particular actions [13]. This makes perfect sense, since the actions that taken by these systems are justified by the fact that they are most likely to yield a small expected advantage many steps in the future rather than because they satisfy some specific subgoal.

Another example where human intelligence looks very different from machine intelligence is in solving the Rubik’s cube. Thanks to some careful analysis and a significant amount of computation, the Rubik’s cube is a solved problem: the shortest path from any configuration to an unscrambled cube has been identified, taking no more than 20moves [45]. However, the solution doesn’t have a huge amount of underlying structure – those shortest paths are stored in a gigantic lookup table. Contrast this with the solutions used by human solvers. A variety of methods for solving the cube exist, but those used by the fastest human solvers require around 50 moves. These solutions require memorizing a few dozen to a few hundred “algorithms” that specify transformations to be used at particular points in the process. Methods also have intermediate subgoals, such as first solving an entire side. 

(Speedruns [LW(p) · GW(p)] are another relevant intuition pump.) 

This is why I don't buy the argument that [LW(p) · GW(p)] "in the limit, superior strategies will tend to be beautiful and elegant", at least for strategies generated by AIs far less limited than humans are w.r.t. time, compute and communication. I don't think they'll necessarily look "dumb", just not decomposable into human working memory-sized parts, hence weird and incomprehensible (and informationally overwhelming) from our perspective. 

An AI, even if it's capable of manipulating the world beyond our understanding, will likely opt for non-chaotic plans over chaotic plans. A plan built on control of a web of people via emotional manipulation is still subject to the chaos of random things happening to those people, differences in personality, the full entropy of the human experience. What is predictable is an X-maximizer will seek plans that minimize chaos, thus it's preference for those plans would not seem as weird to us.

For a really good example of what I would consider a 'dumb' way for AGI misalignment to be problematic, I recommend "accelerando" by charles stross. It's available in text/html form for free from his web site. Even now, after 20 years, it's still very full of ideas.

(FYI, sections are about ten years apart in the book, but last I read it it seemed like the dates are off by a factor of two or so. Eg. 2010 in the book corresponds loosely to 2020 in real life, 2020 in the book corresponds loosely to 2040, etc.)

In that book, the badness largely comes from increasingly competent / sentient corporate management and legal software.

A useful model is Elon Musk. He has an unusual talent of making massively successful decisions that were seen as dumb by most experts at the time. 

The list includes:

• Launching a rocketry startup
• Pursuing reusable rockets
• Investing into a car startup
• Pursuing electric sport cars
• Launching a satellite internet service (too early to judge, but is starting to look like a genius move too)
• Buying Twitter (same)

I don't feel very confident guessing his IQ, but I think it's safe to assume he's somewhere in the top 0.1% of all humans. 

Funnily enough, many people still call him dumb, even after SpaceX and Tesla (e.g. read any thread about him on Reddit). 

As Henry David Thoreau said,

We are apt to class those who are once-and-a-half-witted with the half-witted, because we appreciate only a third part of their wit.

Money is condensed optimisation. AI which can generates money may be stupid, but it can buy clever things. In other words, we may not need something significantly clever than bitcoin to take over the world. 

Bitcoin failed to destroy Earth for now, but some combination of crypto money and LLM could do it.

Another dumb but plausible way that AGI gets access to advanced chemicals, biotech, and machinery; someone asks "how do I make a lot of street drug X" and it snowballs from there.