Well, kind of. The Spaniards succeeded by killing off/holding hostage their opposition's leadership cadre, and through the accidental deployment of biological weapons-they were immune the illnesses that spread from them to the Aztecs, Incas and Mayans.

They won the battle, and they might have won later if they engaged in a full-scale invasion, but it is not completely clear form historical outcomes that they actually had a decisive strategic advantage.

One answer is that using your intelligence to to improve your own cognitive architecture is an entirely new field of investment. The economic growth that accrues from modern investing looks steady from inside the modern economy, but it's explosive from the perspective of a premodern society.

One distinction that may matter here is that something is only counted for as money if it is actually traded between distinct agents. If I got 10 000 times better at making houses from scratch and made a house for me but no one else, there would be no increase in the local economy. If I started selling houses, this would be different.

Superintelligence can be thought of as a large island, where there is a lot of internal trade going on that is invisible to the outside economy. If that was counted as value, then the economy would also be exploding. This internal trade opacity that is proper of intelligence and other intra-agent changes seems to be responsible for the distinctive growth extrapolations.

While not exactly investment, consider the case of an AI competing with a human to devise a progressively better high-frequency trading strategy. An AI would probably:

• be able to bear more things in mind at one time than the human
• evaluate outcomes faster than the human
• be able to iterate on its strategies faster than the human

I expect the AI's superior capacity to "drink from the fire hose" together with its faster response time to yield a higher exponent for the growth function than that resulting from the human's iterative improvement.

One difference is hardware overhang. When new AI technologies are created, many times the amount of hardware they need is available to run them.

However, if an international agreement was reached in advance, and some of the AGI sandbox problems were solved, we might be able to restrict AGI technology to a bounded amount of hardware for a time-if theoretical results which we do not have yet showed that was the appropriate course of action.

We have all kinds of work to do.

There are some qualitative shifts in growth rate with investment. For example, very small amounts of money basically cannot be invested in the financial due to the high fixed costs associated with money laundering regulation, etc.

On the other hand, the more income people have, the more they consume, which reduces growth. Levels of consumption deemed adequate by our ancestors, and indeed consistent with high savings rates, are not viewed as acceptable any more.

Market is more or less stabilized. There are powers and superpowers in some balance. (gain money sometimes could be illusion like bet (and get) more and more in casino).

If you are thinking about money making - you have to count sum of all money in society. If investments means bigger sum of values or just exchange in economic wars or just inflation. (if foxes invest more to hunting and eat more rabbits, there could be more foxes right? :)

In AI sector there is much higher probability of phase-transition (=explosion). I think that's the diference.

How?

1. Possibility: There could be probably enough HW and we just wait for spark of new algorithm.

2. Possibility: If we count agriculture revolution as explosion - we could also count massive change in productivity from AI (which is probably obvious).

Note that a 10,000-fold increase would leave you with a modest fraction of total output. A 1,000-fold increase would still leave you smaller than world's militaries.

The theft scenario relies on you being better at theft than the rest of the world. From the outside AI looks more suited to productive activity than war (which also requires big capital investments in personnel and equipment), so I normally think of this as being a counterbalancing factor (that is, it seems significantly more likely that an economically dominant firm would find many of its productive assets confiscated by force, than that a productive firm would begin stealing the world’s resources without serious repercussions). Of course a primary consideration in this discussion is the nature of conflict; in the current world a sufficiently sophisticated AI might fare reasonably well in all-out conflict due to the world being completely unprepared, but I would be quite surprised if that were still the case when the development of human-level AI actually looked plausible. It seems to again come down to the possibility of a rapid and unexpected jump in capabilities.

The most realistic paths along these lines seem to depend on abrupt changes in the importance of different resources. For example, you might think that AI research capacity would change from a small part of the world to the lion's share (or some even more specific kind of research). If such a change were slow it wouldn't much matter, but if it were fast it could also lead to a decisive strategic advantage, as a company with 0.01% of the world’s resources could find itself with 90% after a drastic reevaluation.

Of course, whether such an abrupt change should happen is again closely related to whether an abrupt change in capacities will happen (though it relies on the further assumption that such a change is unanticipated). Overall I think that it is not too likely, and that the probability can be driven down substantially by clearer communication and decision-making. It still seems like a scenario worth thinking about.

All told, agree with Bostrom that there are a number of reasons to think that very fast changes in capability would lead to a decisive strategic advantage, especially if it came as a surprise to most of the world. Short of rapid changes, I don’t see much reason to think of AI as exceptional.

Regarding evolution I disagree completely with the implicit model of how the future will go or why it might contain morally valuable things. See here for a discussion of my view. On my view, a "singleton" with this more limited capacity would not be very important. Indeed, I'm not even sure what it would mean.

Alternate history is not falsifiable, of course, but that scenario doesn't look all that likely to me. Russia successfully recovered from losing a very large chunk of its territory, a great deal of its army, and most of its manufacturing capacity to Germans in 1941-1942. Losing a few cities (even assuming the bombers could get through -- there were no ICBMs and Russia in 1945 had a pretty good air force and AA capabilities) would not cripple Russia. I would guess that it would just incentivize it to roll over the remainder of Europe. It's not like Stalin ever cared about casualties.

Plausible, though note that while the US (a large fraction of the world) may have dominated the world by that route, it would have been hard for any small group involved to do so.

Probably not: Some nerdy superintelligent AI systems will emerge but humans will try their utmost to shut off early enough. Humankind will become very creative to socialize AGI. The highest risk is that a well funded intelligence agency (e.g. NSA) will be first. Their AI system could make use of TAO knowledge to kill all competing projects. Being nerdy intelligent it could even manipulate competing AI projects that their AIs get "mental" illnesses. This AI will need quite a long time of learning and trust-building until it could take over world dominion.

Bostrums book is a wake-up call. In his book presentation Authors@Google (final six minutes Q&A) he claimed that only half a dozen scientists are working full time on the control problem worldwide. This is by far not enough to cope with future risks. More and effective funding is needed.

Bostrum does not want to destroy his straightforward FOOM-DOOM scenario. He does not discuss technical means of monitoring but only organizational ones.

I fully agree with Bostrum that too few people are working on the control problem. In 2007 Stephen Omohundro asked for synergistic research between psychologists, sociologists and computer system engineers on the control problem. Today we have to conclude that progress is limited.

We have many technical options at hand to prevent that an AI project can obtain decisive strategic advantage:

• Prevent content overhang of powerful inside knowledge (taboos, secrecy, fighting organized internet crime).
• Prevent control access by keeping life supporting infrastructures independent from internet.
• Prevent hardware overhang by improving immunity to cyberattacks (hard crypto).
• Law against backdoors in any system.
• Transparency (AI development, cyberattack monitoring).
• Develop weak-AI with superintelligent capability of monitoring AIs (thought police)
• Develop fixed social conscience utility function.

"What is the best way to push it [risk of doom] down." was Bostrums last sentence at his book presentation. This should be a further point of our discussion.

Well, no life form has achieved what Bostrom calls a decisive strategic advantage. Instead, they live their separate lives in various environmental niches.

One way to look at it is this: Suppose a dominant AGI emerged which was largely running the planet and expanding into the galaxy.

Would it then be impossible to engineer another AGI which survived modestly in some niche or flew off at near the speed of light in a new direction? No.

For the first AGI to be the only AGI, all other AGI development would have to cease without such "niche AGIs" ever being created.

An AGI could be extremely unobtrusive for tens of thousands of years at a time, and even be engaging in some form of self-improvement or replication.

"Sterilizing" matter of all of the "niche AGI" it contains could be quite an involved process.

One possibility to prevent smaller group gain strategic advantage is something like operation Opera.

And it was only about nukes (see Elon Musk statement)...

While space projects are more like the Peacock's tail (flamboyant displays, a theme we will return to later in the book), bombs are more like poison indicators (such as colors black red and yellow in snakes).

AI is actually powerful. As Elon puts it, with AI, you are summoning a demon.

Comparing AI with space projects could be beneficial on some dimensions, but it seems to me that comparison with bombs and other military technology, that are more like the ability to poison or summon demons, should take evidential precedence.

On p82 Bostrom argues that even if one research group has access to another's code, it would still take them months to change approach, which could lead to the leaders getting a full FOOM's worth of headstart. But I don't really understand why he thinks this. If I have access to your code, can't I just run it as soon as I notice you're running it? True, you could obfuscate it, or protect your production-box more carefully. But if I have that good monitoring systems, I'm probably a government, in which case I can afford to have a team dedicated to reverse-engineering your approach.

That seems to depend on the size of the world economy. There may be an absolute minimum of organizational ability for some complex entities to exist - Toby Ord points out that if the human population was much smaller, we wouldn't be able to create the infrastructure that allows for computers.

If the absolute structure of an easy to create Superintelligence requires hundreds of millions of agents in concert, it would appear that a large fraction of the economy is necessary for an observer of a world with a few billion people.

A more obvious note is that the closer you are to representing half the economy, the easier it is to reach the point in which you control 51% of it (which may coincide with strategic advantage in some cases).

Bostrom makes an interesting point that open literature tends to benefit small teams relative to large ones.

Also, he points out that it is easier for a government to nationalise the most successful projects than to ban all projects. The most successful projects are likely to have significant funding, hire well-known experts, go to conferences, etc. Small basement projects, on the other hand, could much more easily escape detection. Unfortunately, this is a pro-arms-race dynamic: it is easy for governments to try to outrace small teams with their own teams than to ban development.

The point about countries defecting from global collaborations is also interesting. It's relatively hard to defect on a nuclear-sharing project: you actually need to build your own reactors, acquire fissile material, etc. But running your own AI project might only requires you to steal some thumb drives.

The "Monitoring" subsection. Bostrom makes good points about why state actors would be likely to seek to monopolize AGI development, if they took it to be a threat to their power monopolies. I hadn't given the possibility sufficient consideration until this point. These actors do seem disinterested so far, but (as Bostrom points out) that might well be a pretense, and even if not it doesn't mean they'll remain so for too long to matter.

A human is capable to run a large company or a government earliest at the age of 30.

LOL. Do you really think so?