Why an Intelligence Explosion might be a Low-Priority Global Risk

It is true that a Harpy Eagle can lift more than three-quarters of its body weight while the Boeing 747 Large Cargo Freighter has a maximum take-off weight of almost double its operating empty weight (I suspect that insects can do better).

Which doesn't (automatically) mean that the 747 has worse design than the eagle. Smaller things (constructions, machines, animals) are relatively stronger than bigger things not because of their superior design, but because physics of materials is not scale invariant (unless you somehow managed to scale the size of atoms too). If you made a 1000:1 scaled copy of an ant, it wouldn't be able to lift objects twenty times heavier.

Why haven’t we seen a learning algorithm teaching itself chess intelligence starting with nothing but the rules?

We have now, depending on how you interpret "teach itself". It wasn't given anything but the rules and how to play against itself.

One issue I've mentioned before and I think is worth addressing is how much of the ability to quickly self-improve might depend on strict computational limits from theoretical computer science (especially complexity theory). If P != NP in a strong sense then recursive self-improvement many be very difficult.

More explicitly, many problems that are relevant for recursive self-improvement (circuit design and memory management for example) explicitly involve graph coloring and traveling salesman variants which are NP-hard or NP-complete. In that context, it could well be that designing new hardware and software will quickly hit diminishing marginal returns. If P, NP. coNP, PSPACE, and EXP are all distinct in a strong sense, then this sort of result is plausible.

There are problems with this sort of issue. One major one is that the standard distinctions between complexity classes are all in terms of Big-Os. So it could well be that the various classes are distinct but that the constants are small enough such that for all practical purposes one can do whatever one wants. There are also possible loopholes. For example, Scott Aaronson has shown that if one has access to closed time-like curves then one can quickly solve any problem in PSPACE. This is one of the more exotic loopholes. Quantum computers also may have more efficient computation. At this point, while almost everyone believes that P != NP, the notion that BQP doesn't contain NP seems substantially more uncertain. Finally, and most mundanely, it may be that even as the general problems are tough, the average cases of NP problems may not be difficult, and the specific examples that actually come up may have enough regularity that they can be solved more efficiently. Indeed, in one sense, part of why Deolalikar's attempted proof that P !=NP failed is that statistically, NP looks easy on average (more particularly, k-SAT is NP-complete for k >=3, but k-SAT for general k looks statistically a lot like 2-SAT).

It would seem to me that at this point that a lot more attention should be paid to computational complexity and what it has to say about the plausibility of quick recursive self-improvement.

Why haven’t we seen a learning algorithm teaching itself chess intelligence starting with nothing but the rules?

AIXI-MC can teach itself Pacman; Pacman by default is single-player so the game implementation has to be already done for AIXI-MC. I suppose you could set up a pair of AIXI-MCs like is sometimes done for training chess programs, and the two will gradually teach each other chess.

Evolution versus Intelligence

I didn't read all of this post, unfortunately, I didn't have time before class. But I wanted to mention my thoughts on this section. This seemed like a very unfortunate analogy. For one, the specific example of flying is wildly biased against humans, who are orders of magnitude handicapped by the square cube law. Secondly, you can think of any number of arbitrary counterexamples where the advantage is on the side of intelligence. For a similar counterexample, intelligence has invented cars, trains, and boats which allow humans to travel at velocities far superior to evolution's animals and for longer periods of time

Even if the AGI is not told to hold, e.g. compute as many digits of Pi as possible, I consider it an far-fetched assumption that any AGI intrinsically cares to take over the universe as fast as possible to compute as many digits of Pi as possible. Sure, if all of that are presuppositions then it will happen, but I don’t see that most of all AGI designs are like that. Most that have the potential for superhuman intelligence, but who are given simple goals, will in my opinion just bob up and down as slowly as possible.

It seems to be a kind-of irrelevant argument, since the stock market machines, query answering machines, etc. that humans actually build mostly try and perform their tasks as quickly as they can. There is not much idle thumb-twiddling in the real world of intellligent machines.

It doesn't much matter what machines who are not told to ack quickly will do - we want machines to do things fast, and will build them that way.

Before I dive into this material in depth, a few thoughts:

First, I want to sincerely congratulate you on being (it seems to me) the first in our tribe to dissent.

Second, it seems your problem isn't with an intelligence explosion as a risk all on its own, but rather as a risk among other risks, one that is farther from being solved (both in terms of work done and in resolvability), and so this post could use a better title, i.e., "Why an Intelligence Explosion is a Low-Priority Global Risk", which does not a priori exclude SIAI from potential donation targets. If I'm wrong about this, and you would consider it a low-priority thing to get rid of the global risk from an intelligence explosion even aside from other global risks, I'll have to ask for an explanation.

Edit: It seems my comment has been noted and the title of the post changed.

I can't really accept innovation as random noise. That doesn't seem to account for the incredible growth in the rate of new technology development. I think a lot of developments are in fact based on sophisticated analysis of known physical laws - e.g. a lot of innovation is engineering versus discovery. Many foundational steps do seem to be products of luck; such as the acceptance of the scientific method.

given all of the apparent disadvantages of the blind idiot God, evolution was able to come up with altruism, something that works two levels above the individual and one level above society.

Much human altruism is fictional - humans are nice to other humans mostly because being nice pays.

There are low-level selection explanations for most of the genuine forms of human altruism. IMHO, the most promising explanations are:

• Altruism is the result of manipulation by other humans;
• Altruism is the result of manipulation by memes.
• Altruism is the result of overgeneralisation due to cognitive limitations;
• Altruism towards kin (and fictive kin) is explained by kin selection.

The critical similarity is that both rely on dumb luck when it comes to genuine novelty.

Someone pointed out that a sufficiently powerful intelligence could search all of design space rather than relying on "luck".

I read it on the Web, but can't find it - search really sucks when you don't have a specific keyword or exact phrasing to match.

This should be in Main. I'd much rather have this than "how I broke up with my girlfriend" there.

(Otherwise, I don't have much to say because I basically agree with you. I find your arguments kinda weak and speculative, but much less so than arguments for the other side. So your skepticism is justified.)

Many humans are not even capable of handling the complexity of the brain of a worm.

I don't think that's the right reference class. We're not asking is something is sufficient, but if something is likely.

Our “irrationality” and the patchwork-architecture of the human brain might constitute an actual feature. The noisiness and patchwork architecture of the human brain might play a significant role in the discovery of unknown unknowns because it allows us to become distracted, to leave the path of evidence based exploration...The noisiness of the human brain might be one of the important features that allows it to exhibit general intelligence.

If you can figure this out, and a superintelligent AI couldn't assign it the probability it deserves and investigate and experiment with it, does that make you supersuperintelligent?

Also, isn't the random noise hypothesis being privileged here? Likewise for "our tendency to be biased and act irrationally might partly be a trade off between plasticity, efficiency and the necessity of goal-stability."

But that expert systems are better at certain tasks does not imply that you can effectively and efficiently combine them into a coherent agency.

Why do these properties of expert systems matter, as no one is discussing combining them?

Why haven’t we seen a learning algorithm teaching itself chess intelligence starting with nothing but the rules?

There's progress along these lines.

It is claimed that an artificial general intelligence might wipe us out inadvertently

"Inadvertently" gives the wrong connotations.

For an AGI, that was designed to design paperclips, to pose an existential risk, its creators would have to be capable enough to enable it to take over the universe on its own, yet forget, or fail to, define time, space and energy bounds as part of its optimization parameters

What if the AI changed some of its parameters?

It would appear that we have reached the limits of what it is possible to achieve with computer technology, although one should be careful with such statements, as they tend to sound pretty silly in 5 years

--John von Neumann

A lot of this post sounds like fake ignorance. If you just read over it, you might think the questions asked are genuinely unknown, but if you think for a bit, you can see we have quite a lot of evidence and can give a rough answer.

We simply don’t know if intelligence is instrumental or quickly hits diminishing returns.

Well humans are doing ok for themselves, it seems to have accelerating returns up to the level of a smart human. Whats more, intelligence gets more valuable with increasing scale, and with cheaper compute. When controlling a roomba, you are controling a few watts. An algorithm that took a 1kw computer cluster to run, and improved efficiency by 5% wouldn't be worth it. But it would be worth it to control a power station. Whatsmore, the human brain seems a long way from the theoretical limits of compute. So as a lower bound, imagine what a team of smart humans could do running at 1000 times speed, and then imagine that cost you < 1 watt in energy. 

Can intelligence be effectively applied to itself at all?

Yes. Hence the fields of psycology and AI research.

It seems that you need to apply a lot more energy to get a bit more complexity.

Doesn't seem to match evolutionary record.

What evidence do we have that the payoff of intelligent, goal-oriented experimentation yields enormous advantages (enough to enable an intelligence explosion) over evolutionary discovery relative to its cost?

The way humans can easily do things that evolution never could. The fact that evolution is a really stupid algorithm, its generally much faster to make the problem space differentiable, then use gradient descent.

Just a reminder that risk from AI can occur without recursive self-improvement. Any AGI with a nice model of our world and some goals could potentially be extremely destructive. Even if intelligence has diminishing returns, there is a huge hardware base to be exploited and a huge number of processors working millions of times faster than brains to be harnessed. Maybe intelligence won't explode in terms of self-improvement, but it can nevertheless explode in terms of pervasiveness and power.

Due to the absence of any signs of intelligence out there, especially paper-clippers burning the cosmic commons, we might conclude that unfriendly AI could not be the most dangerous existential risk that we should worry about.

I view this as one of the single best arguments against risks from paperclippers. I'm a little concerned that it hasn't been dealt with properly by SIAI folks - aside from a few comments by Carl Shulman on Katja's blog.

I suspect the answer may be something to do with anthropics - but I'm not really certain of exactly what it is.

Intelligence is no solution in itself, it is merely an effective searchlight for unknown unknowns and who knows that the brightness of the light increases proportionally with the distance between unknown unknowns? To enable an intelligence explosion the light would have to reach out much farther with each increase in intelligence than the increase of the distance between unknown unknowns. I just don’t see that to be a reasonable assumption.

We do have some data on historical increases in intelligence due to organic and cultural evolution. There's the fossil record of brain sizes, plus data like the Flynn effect. The process has been super-exponential. The intelligence eplosion has been going on for about 500 million years so far. As Moravec puts it:

The largest nervous systems doubled in size about every fifteen million years since the Cambrian explosion 550 million years ago. Robot controllers double in complexity (processing power) every year or two.

Machine intelligence looks set to be a straightforwards continuation of this long and well-established trend towards bigger brains in the brainiest creatures.

I agree with the above, yet given all of the apparent disadvantages of the blind idiot God, evolution was able to come up with altruism, something that works two levels above the individual and one level above society. So far we haven’t been able to show such ingenuity by incorporating successes that are not evident from an individual or even societal position.

I don't understand this paragraph. What does "something that works two levels above the individual and one level above society" mean? Or the follow-up sentence?

It seems that if you increase intelligence you also increase the computational cost of its further improvement and the distance to the discovery of some unknown unknown that could enable another quantum leap. It seems that you need to apply a lot more energy to get a bit more complexity.

I don't necessarily think it's true that you need to know an unknown unknown to reach a "quantum leap". This is a very qualitative reasoning about intelligence. You could simply increase the speed. Also, evolution didn't make intelligence by knowing some unknown unknown, it was a result of trial and error. Further intelligence improvement could use the same method, just faster.

The AGI would have to acquire new resources slowly, as it couldn’t just self-improve to come up with faster and more efficient solutions. In other words, self-improvement would demand resources. The AGI could not profit from its ability to self-improve regarding the necessary acquisition of resources to be able to self-improve in the first place.

If the AGI creates a sufficiently convincing business plan / fake company front, it might well be able to command a significant share of the world's resources on credit and either repay after improving or grab power and leave it at that.

The first several points you make seem very weak to me, however starting with the section on embodied cognition the post gets better.

Embodied cognition seems to me like a problem for programmers to overcome, not an argument against FOOM. However it serves as a good basis for your point about constrained resources; I suspect that with sufficient time and leeway and access to AIM an AGI could become an extremely effective social manipulator. However this seems like the only avenue which it would obviously have the ability to get and process responses easily without needing hardware advances which would be difficult to acquire. Pure text messages don't require much memory or much bandwidth, and there are vast numbers of people accessible to interact with at a time, so it would be hard to restrict an AI's ability to learn to talk to people, but this is extremely limited as a way of causing existential risk.

Your paperclip maximizing argument is one that I have thought out before, but I would emphasize a point you seem to neglect. The set of mind designs which would take over the universe seems dense in the set of all generally intelligent mind designs to me, however not necessarily among the set of mind designs which humans would think to program. I don't think that humans would necessarily need to be able to take over the world to create a monster AI, but I do think that a monster AI implies something about the programmer which may or may not be compatible with human psychology.

Overall your post is decent and interesting subject matter. I'm not finding myself persuaded but my disagreements feel more empirical than before the reading which is good!

If you accept the premise that life is not unique and special then one other technological civilisation in the observable universe should be sufficient to leave potentially observable traces of technological tinkering.

Due to the absence of any signs of intelligence out there, especially paper-clippers burning the cosmic commons, we might conclude that unfriendly AI could not be the most dangerous existential risk that we should worry about.

It seems like an argument for DOOM - but what if getting this far is simply very difficult?

Then we could be locally first, without hypothesizing that we are surrounded by the ashes of many other failed technological civilisations.

In which case, machine intelligence might well represent humanity's biggest danger.

Lack of aliens just illustrates the great filter. It doesn't imply that it lies ahead of us. Indeed, we can see from the billions of years of progress and the recent invention of space travel for the first time that much of it lies behind us.

We simply don’t know if intelligence is instrumental or quickly hits diminishing returns.

General intelligence -- defined as the ability to acquire, organize, and apply information -- is definitionally instrumental. Greater magnitudes of intelligence yield greater ability to acquire, organize, and apply said information.

Even if we postulate an increasing difficulty or threshold of "contemplative-productivity" per new "layer" of intelligence, the following remains true: Any AGI which is designed as more "intelligent" than the (A)GI which designed it will be material evidence that GI can be incremented upwards through design: and furthermore that general intelligence can do this. This then implies that any general intelligence that can design an intelligence superior to itself will likely do so in a manner that creates a general intelligence which is superior at designing superior intelligences, as this has already been demonstrated to be a characteristic of general intelligences of the original intelligence's magnitude.

Furthermore; as to the statements about evolution -- evolutionary biology maximizes/optimizes for specific constraints that we humans in designing do not do. There is no evolutionary equivalent of the atomic bomb, nor of the Saturn-V rocket. Evolution furthermore typically will retain "just enough" of a given trait to "justify" the energy-cost of maintaining said trait.

Evolutionarily speaking, abstracting general intelligence has been around less than the blink of an eye.

I don't know that your position is coherent given these points. (Though I do want to point out for re-emphasis that I nowhere in this stated that seed-AGI is either likely or unlikely.)

There is a significant difference between intelligence and evolution if you apply intelligence to the improvement of evolutionary designs:

• Intelligence is goal-oriented.
• Intelligence can think ahead.
• Intelligence can jump fitness gaps.
• Intelligence can engage in direct experimentation.
• Intelligence can observe and incorporate solutions of other optimizing agents.

Much of this seems pretty inaccurate. The first three points are true, but not realy the issue - and explaning why would go uncomfortably close to the topic I am forbidden from talking about - but for the last two points, surely every organism is its own experiment, and mutualistic symbiosis allows creatures to use the results of experiments performed by members of other species.

Intelligence does make some differences, but not really these differences.