If you have a human-level intelligence which can read super-fast, and you set it free on the internet, it will learn a lot very quickly. (p71)

But why would you have a human-level intelligence which could read super-fast, which hadn't already read most of the internet in the process of becoming an incrementally better stupid intelligence, learning how to read?

Similarly, if your new human-level AI project used very little hardware, then you could buy heaps more cheaply. But it seems somewhat surprising if you weren't already using a lot of hardware, if it is cheap and helpful, and can replace good software to some extent.

I think there was a third example along similar lines, but I forget it.

In general, these sources of low recalcitrance would be huge if you imagine AI appearing fully formed at human-level without having exploited any of them already. But it seems to me that probably getting to human-level intelligence will involve exploiting any source of improvement we get our hands on. I'd be surprised if these ones, which don't seem to require human-level intelligence to exploit, are still sitting untouched.

Overall, I thought the case in this chapter for faster over slower takeoffs was weak.

The main considerations pointing to low recalcitrance for AI seem to be the possibility of hardware and content overhangs - though these both seem likely to have been used up already, and are anyway one-off events - and the prospect of hardware growth in general continuing to be fast, which seems reasonable but doesn't distinguish AI from any other software project.

So the argument for fast growth has to be based on optimization power increasing a huge amount, as I think Bostrom intends. The argument for that is that first people will become interested in the project, causing it to grow large enough to drive most of its own improvement, and then it will recursively self-improve to superintelligence.

I agree that optimization power applied to the problem will increase if people become very interested in it. However saying optimization power will increase seems very far from saying the project will grow large enough to rival the world in producing useful inputs for it, which seems an ambitious goal. Optimization power applied to different projects increases all the time and this doesn't happen.

It would have been good to have a quantitative sense of the scale on which the optimization power is anticipated to grow, and of where the crossover is thought to be. One of these two has to be fairly unusual for a project to take over the world it seems, yet the arguments here are too qualitative to infer that anything particularly unusual would happen, or to distinguish AI from other technologies.

I don't mean to claim that the claim is false - that would be a longer discussion - just that the case here seems insufficient.

This chapter seems to present some examples of how algorithmic recalcitrance could be very low, but I think it doesn't, in the relevant sense. Two of the three arguments in that part of the chapter (p69-70) are about how low recalcitrance might be mistaken for high recalcitrance, rather than about how low recalcitrance would occur. (One says that a system whose performance is the maximum of two parts might shift its growth from that of one part to that of the other; the other says we might be biased to not notice growth in dumb-seeming entities). The third argument (or first, chronologically) is that a key insight might be discovered after many other things are in place. This is conceivable, but seems to rarely happen at a large scale, and comes with no particular connection with intelligence - you could make exactly the same argument about any project (e.g. the earlier intelligence augmentation projects).

In order to model intelligence explosion, we need to be able to measure intelligence.

Describe a computer's power as . What is the relative intelligence of these 3 computers?

Is 2 twice as smart as 1 because it can compute twice as many square roots in the same time? Is it smarter by a constant C, because it can predict the folding of a protein with C more residues, or can predict weather C days farther ahead?

If we want to ask where superintelligence of some predicted computational power will lie along the scale of intelligence we know from biology, we could look at evolution over the past 2 billion years, construct a table estimating how much computation evolution performed in each million years, and see how the capabilities of the organisms constructed scaled with computational power.

This would probably conclude that superintelligence will explode, because, looking only at more and more complex organisms, the computational power of evolution has decreased dramatically owing to larger generation times and smaller population sizes, yet the rate of intelligence increase has probably been increasing. And evolution is fairly brute-force as search algorithms go; smarter algorithms should have lower computational complexity, and should scale better as genome sizes increase.

In figure 7 (p63) the human and civilizational baselines are defined relative to 2014. Note that if human civilization were improving in its capabilities too - perhaps because of individual cognitive enhancements, technological progress, or institutional improvements - the crossover point would move upwards, while the 'strong superintelligence point' would not. Thus AI could in principle reach the 'strong superintelligence' point before the 'crossover'.

Crossover involves the project contributing more to itself than the outside world does. Note that this is not implied by even the quickest growth. A massive project might still mostly engage in trade with the rest of society, benefiting largely from their innovations, and contributing its own to wider progress. The picture above shows an AI project growing to rival the whole world in capability, at which point it might naturally contribute a lot to its own growth, relative to what the world contributes. However the world can also grow. If we are talking about a particular AI project, then other AI projects are a natural part of the rest of the world to be experiencing fast growth at the same time. And if fast-growing AI projects were improving every labor-based part of the economy, then outside growth should also increase. We will consider this sort of thing more in a future chapter about whether single projects take power, but I think it is worth asking whether reaching 'crossover' should be the default expectation.

A step change in the rate of gain in average organizational efficiency is perhaps conceivable, but it is hard to see how even the most radical scenario of this kind could produce anything faster than a slow takeoff, since organizations operated by humans are confined to work on human timescales. (p67)

It seems a bit unclear how fast 'human timescales' are, especially since organizations usually consist of humans and other equipment. Are securities traded at human timescales? What are the speed limits for organizations of humans?

There's an argument against fast takeoff based on computational complexity theory.

Fast takeoff seems to imply that there is a general purpose algorithm that, given large-but-practically-possible amount computational resources, could solve problem instances with real-life relevance in many different domains. (If there is just a bunch of domain-specific algorithms, takeoff cannot be as fast.)

Complexity theory tells that it might not be the case. Many relevant problem classes are believed to be computationally hard.

For example, if the AGI wants to "solve" (i.e. significantly optimize) economics, it might have to deal with large-size instances of task scheduling problems. Since we know that the best possible general purpose algorithm for task scheduling is unlikely to run faster than exponential time, even exponential hardware and software speedup won't make optimal task scheduling tractable! Therefore, the exponential "jump" in AGI's algorithmic capabilities during the initial self-optimization period would not lead to corresponding exponential "jump" in its problem-solving capabilities.

It's still possible (although unlikely) that this general purpose algorithm could "beat" any existing domain-specific algorithm. Even more, the argument still stands even if we assume that the AGI is a better problem solver in any strategically relevant field than the combined forces of human experts and narrow AI. The point is that this "better" is unlikely to give the AGI strategical dominance. I think that the capability to solve problems humanity can not solve on its own is to be required for strategical dominance.

On the other hand, its worth noting that general-purpose solvers of computationally hard problems have seen large practical success in the last decade. (SAT and constraint programming). This seems to weaken the argument, but to what extent?

It's a pity that Bostrom never mentions complexity classes in his book.

Why does the insect 'milestone' help to see the future? p68

Bostrom says that a fast takeoff would leave scant time for human deliberation (p64). While it certainly leaves much less time to deliberate than the other scenarios, I can imagine two days allowing substantially more deliberation fifty years hence than it would now. Information aggregation and deliberation technologies could arguably be improved a lot. Automatic but non-agent systems might also be very good by the time human-level agents are feasible.

What projects can you think of that are closest to 'crossover'?

I found data on renewable funding and progress (see above), to test how extra interest and funding translates to progress, because this is a key part of Bostrom's argument for fast AI progress prior to the 'crossover'. Can you find similar data for any other areas?

So the base rate for the kind of transition entailed by a fast or medium takeoff scenario, in terms of the speed and magnitude of the postulated change, is zero: it lacks precedent outside myth and religion.

Bostrom says this, but seems to use it only as a nice way of introducing a position to argue against - that a medium or fast transition is unlikely - and not so much as evidence. Yet it seems to me that this is potentially substantial evidence, and deserves attention.

If we want to combine this outside view evidence with other object level arguments, one way to do it is to ask how often there are explosive transitions after there have seemed such good reasons for explosive transitions. That is, if this is the first time in history that optimization power has seemed set to substantially increase while recalcitrance has seemed likely to decrease so much, then the previous evidence that explosive transitions have not happened is perhaps irrelevant, and we might expect a large burst of progress. However if humans often look at the future and see declining recalcitrance and increasing optimization power (to the extent that Bostrom sees these things), and yet there haven't been such transitions, then we should hesitate in accepting that we will get one this time.

Have things looked this way before?

One might complain that the word 'takeoff' for the progression from human-level intelligence to superintelligence biases the discussion toward supposing that this progress will be fast, or in some other way resemble the launch of an airplane or rocketship.

Another way to put this, avoiding certain ambiguities, is 'a point at which the inputs to a project are mostly its own outputs', such that improvements to its outputs feed back into its inputs.

This doesn't seem to follow. Most inputs of the system, for a very long number of time-steps (certainly before the conception of deception) will come from the external world at is learns.

The relevant inputs here are only those that will cause self improvement of the relevant kind, which would be likely outnumbered by non self-improvement inputs, such as perceptions, query results etc...

It could be, that some bellow human level (general or not) intelligence explodes into (general) superintelligence.

It's worth to consider that case, too.

If you learned that there would definitely be a fast takeoff, what would you do? What about a moderate or slow one?

The recalcitrance for making networks and organizations in general more efficient is high. A vast amount of effort is going into overcoming this recalcitrance, and the result is an annual improvement of humanity's total capacity by perhaps no more than a couple of percent.

This is the first time we have close to a quantitative estimate for recalcitrance: a couple of percent a year for something as big as human society, and 'a vast amount' of effort. Do you think this recalcitrance is high relative to that of other systems under consideration? Also, what efforts do you think count as 'overcoming this recalcitrance'?

What empirical evidence could you look at to better predict the future winner of the Foom Debate? (for those who looked at it above)

Has anything else had ambiguous-maybe-low recalcitrance and high optimization power applied in the past? What happened?

What did you find most interesting in this week's reading?

Did you change your mind about anything as a result of this week's reading?

How plausible do you find the key arguments in this chapter? (see list above)

What is going on with those renewable funding and performance curves?

Given all that is in this chapter, how likely do you find the different speeds of takeoff? (slow, moderate, and fast) (p63-64)

What do you think of 'rate of change in intelligence = optimization power/recalcitrance' (p65) as a useful model of progress in intelligence for different systems?

I am slightly concerned by the equation

• Rate of change in intelligence = Optimization Power / Recalcitrance

When introduced, Bostrom merely describes it as a monotonic function, which seems reasonable. But why do we assume that a quotient is the natural functional form?

The problem is is that superintelligence has been re-defined. Under current models it aint gonna happen no matter how sexy the presentation and thoughts inputted. Eating potatoes or whatever doesn't make you smarter. Take all the German scientists back end of 19th-20th century. Stodgy European food. And it worked. They delivered. Now Asians have much better cuisine but they are now catching up and may yet end up as the best is yet to come. Drugs are good. Esp Opium. Cognitive enhancement big time. Yet again those cultures that enjoyed it staid within their cultural milieu for ages. Again things have changed but that option should be made available to all everywhere. The rest is gimmicky. Interesting in itself as itself. Take whole brain emulation. I mean that is what babies are. Re-creating brains and or designing wetware is silly. Plus the brain so emulated may have ideas of its own and turn DALEK on us. The essence of intelligence is resonant sentience. Meditation techniques esp Ch'an-Zen [and related] Buddhist practices will give results but this takes years and even then no guarantee. But this self enhancement seems more logical than a lot of machines thinking they think by merely puzzling out solutions that are in essence given anyway. The Mind. That is the centre of everything. Molecular enhancement will for now be the best bet.