Criticisms of intelligence explosion

My problem with the focus on the idea of intelligence explosion is that it's too often presented as motivating the problem of FAI, but it's really not, it's a strategic consideration right there besides Hanson's malthusian ems, killer biotech and cognitive modification, one more thing to make the problem urgent, but still one among many.

What ultimately matters is implementing humane value (which involves figuring out what that is). The specific manner in which we lose ability to do so is immaterial. If intelligence explosion is close, humane value will lose control over the future quickly. If instead we change our nature through future cognitive modification tech, or by experimenting on uploads, then the grasp of humane value on the future will fail in orderly manner, slowly but just as irrevocably yielding control over to wherever the winds of value drift blow.

It's incorrect to predicate the importance, or urgency of gaining FAI-grade understanding of humane value on possibility of intelligence explosion. Other technologies that would allow value drift are for all purposes similarly close.

(That said, I do believe AGIs lead to intelligence explosions. This point is important to appreciate the impact and danger of AGI research, if complexity of humane value is understood, and to see one form that implementation of a hypothetical future theory of humane value could take.)

Too much theory, not enough empirical evidence. In theory, FAI is an urgent problem that demands most of our resources (Eliezer is on the record saying that the only two legitimate occupations are working on FAI, and earning lots of money so you can donate money to other people working on FAI).

In practice, FAI is just another Pascal's mugging/ lifetime dilemma/ St. Petersburg Paradox. From XiXIDu's blog:

To be clear, extrapolations work and often are the best we can do. But since there are problems such as the above, that we perceive to be undesirable and that lead to absurd consequences, I think it is reasonable to ask for some upper and lower bounds regarding the use and scope of certain heuristics.

[...]

Taking into account considerations of vast utility or low probability quickly leads to chaos theoretic considerations like the butterfly effect. As a computationally bounded and psychical unstable agent I am unable to cope with that. Consequently I see no other way than to neglect the moral impossibility of extreme uncertainty.

Until [various rationality puzzles] are resolved, or sufficiently established, I will continue to put vastly more weight on empirical evidence and my intuition than on logical implications, if only because I still lack the necessary educational background to trust my comprehension and judgement of the various underlying concepts and methods used to arrive at those implications.

You should have entitled this post "Criticisms of Criticisms of intelligence explosion" :-)

(2) the claim that intelligence explosion is likely to occur within the next 150 years

A "scientific" prediction with a time-frame of several decades and no clear milestones along the way is equivalent to a wild guess. From 20 Predictions of the Future (We’re Still Waiting For):

Weather Control: In 1966, a radio documentary, 2000 AD, was aired as a forum for various media and science personalities to discuss what life might be like in the year 2000. The primary theme running through the show concerned a prediction that no one in the year 2000 would have to work more than a day or two a week, and our leisure time would go through the roof. With so much free time, you can imagine that we would not want our vacations or day trips ruined by nasty weather, and therefore we should quickly develop a way to control the weather, shaping it to our needs. Taking the lighting from the clouds or the wind from the tornadoes were among the predictions, yet they were careful to note that we might not take weather control too far because of political reasons. Unfortunately, we here in the 2000’s still work full weeks, and we still get our picnics rained out from time to time.

One could argue that weather control is an easier problem than AGI (e.g. powerful enough equipment could "unwind" storms and/or redirect weather masses).

I boldly claim that my criticisms are better than those of the Tech Luminaries:

• Is an Intelligence Explosion a Disjunctive or Conjunctive Event?
• Why an Intelligence Explosion might be a Low-Priority Global Risk (as an addition: No Basic AI Drives and my recent comment here)

Also see:

• The Curve of Capability by rwallace
• The Betterness Explosion by Robin Hanson
• Is The City-ularity Near? by Robin Hanson
• A summary of Robin Hanson's positions by Robin Hanson
• How far can AI jump? by Katja Grace

Crocker’s rules declared, because I expect this may agitate some people:

(1) the claim that intelligence explosion is plausible, (2) the claim that intelligence explosion is likely to occur within the next 150 years, and (3) the claim that intelligence explosion would have a massive impact on civilization.

I accept (1) and (3). Where I depart somewhat from the LW consensus is in the belief that anyone is going to accept the idea that the singularity (in its intelligence explosion form) should go ahead, without some important intervening stages that are likely to last for longer than 150 years.

CEV is a bad idea. I am sympathetic towards the mindset of the people who advocate it, but even I would be in the pitchfork-wielding gang if it looked like someone was actually going to implement it. Try to imagine that this was actually going to happen next year, rather than being a fun thing discussed on an internet forum – beware far mode bias. To quote Robin Hanson in a recent OB post:

Immersing ourselves in images of things large in space, time, and social distance puts us into a “transcendant” far mode where positive feelings are strong, our basic ideals are more visible than practical constraints, and where analysis takes a back seat to metaphor.

I don’t trust fallible human programmers to implement soundly “knowing more”, “thinking faster” and “growing up together”, and deal with the problems of “muddle”, “spread” and “distance”. The idea of a “last judge” as a safety measure seems like a sticking plaster on a gaping wound. Neither do I accept that including all of humanity is anything other than misplaced idealism. Some people seem to think that even a faulty CEV initial dynamic magically corrects itself into a good one; that might happen, but not with nearly a high enough probability.

Another problem that has been scarcely discussed: what happens if, as Eliezer’s CEV document suggests might happen, the thing shuts itself down or the last judge decides it isn’t safe? And the second time we try it, too?

But the problem remains that a superintelligence needs a full set of human values in order for it to be safe. I don’t see any other tenable proposals for implementing this apart from CEV, therefore I conclude that building a recursively improving superintelligence is basically just unsafe, given present human competence levels. Given that fact, to conclude that because we are likely to obtain the means to bring about a (positive or negative) singularity at some point we cannot prevent it from happening indefinitely is like saying that because we possess nuclear technology we can’t prevent a nuclear extinction event from happening indefinitely. If FAI is an “impossible” challenge and NAI (No AI) is merely very difficult, there is something to recommend NAI.

That doesn’t mean to say that I disprove of what Eliezer et al are doing. The singularity is definitely an extremely important thing to be discussing. I just think that the end product is likely to be widespread recognition of the peril of playing around with AI, and this (along with appropriately severe action taken to reduce the peril) is just as much a solution to Yudkowsky’s fear that a bunch of above-average AI scientists can “learn from each other’s partial successes and accumulate hacks as a community” as is trying to beat them to the punch by rushing to create a positive singularity.

Although this is unfair there is probably some truth to the idea that people who devote their lives to studying AI and the intelligence explosion are likely to be biased towards solutions to the problem in which their work achieves something really positive, rather than merely acting as a warning. That is not to pre-judge the issue, but merely to recommend that a little more skepticism than normal is due.

On the other hand there is another tenable approach to the singularity that is less widely recognised here. Wei Dai’s posts here and here seem very sensible to me; he suggests that intelligence enhancement should have priority over FAI research:

Given that there are known ways to significantly increase the number of geniuses (i.e., von Neumann level, or IQ 180 and greater), by cloning or embryo selection, an obvious alternative Singularity strategy is to invest directly or indirectly in these technologies, and to try to mitigate existential risks (for example by attempting to delay all significant AI efforts) until they mature and bear fruit (in the form of adult genius-level FAI researchers). [...]

The chances of success seem higher, and if disaster does occur as a result of the intelligence amplification effort, we're more likely to be left with a future that is at least partly influenced by human values.

He quotes Eliezer as having said this (from pages 31-35 here):

If AI is naturally far more difficult than intelligence enhancement, no harm done; if building a 747 is naturally easier than inflating a bird, then the wait could be fatal. There is a relatively small region of possibility within which deliberately not working on Friendly AI could possibly help, and a large region within which it would be either irrelevant or harmful.

Wei Dai points out that it is worth distinguishing the ease of creating uFAI in comparison to FAI, rather than lumping these together as “AI”.

I also think that the difference in outcomes between “deliberately not working on Friendly AI” and “treating unsupervised AI work as a terrible crime” are worth distinguishing.

Even if human intelligence enhancement is possible, there are real, difficult safety considerations; I would have to seriously ask whether we wanted Friendly AI to precede intelligence enhancement, rather than vice versa.

This depends on the probability one assigns to CEV working. My probability that it would work given present human competence levels is low, and my probability that anyone would actually let it happen is very low.

The benefit of intelligence enhancement is that changes can be as unhurried and incremental as one likes (assuming that the risk of someone building uFAI is not considered to be imminent, due to stringent security measures); CEV is more a leap of faith.

This is like the kid version of this page. Where are the good opposing arguments? These are all terrible...

Distinguish positive and negative criticisms: Those aimed at demonstrating the unlikelihood of an intelligence explosion and those aimed at merely undermining the arguments/evidence for the likelihood of an intelligence explosion (thus moving the posterior probability of the explosion closer to its prior probability).

Here is the most important negative criticism of the intelligence explosion: Possible harsh diminishing returns of intelligence amplification. Let f(x, y) measure the difficulty (perhaps in expected amount of time to complete development) for an intelligence of IQ x to engineer an intelligence of IQ y. The claim that intelligence explodes is roughly equivalent to the thesis that f(x, x+1) decreases relatively quickly. What is the evidence for this claim? I haven't seen a huge amount. Chalmers briefly discusses the issue in his article on the singularity and points to how amplifying a human being's intelligence from average to Alan Turing's level has the effect of amplifying his intelligence-engineering ability from more or less nil to being able to design a basic computer. But "nil" and "basic computer" are strictly stupider than "average human" and "Alan Turing," respectively. It's evidence that a curve like f(x, x-1) - the difficulty of creating a being slightly stupider than yourself given your intelligence level - decreases relatively quickly. But the shapes of f(x, x+1) and f(x, x-1) are unrelated. The one can increase exponentially while the other decays exponentially. (Proof: set f(x, y) = e^(y^2 - x^2).)

See also JoshuaZ's insightful comment here on how some of the concrete problems involved in intelligence amplification are linked to to some (very likely) computationally intractable problems from CS.

the claim that intelligence explosion is likely to occur within the next 150 years

"We have made little progress toward cross-domain intelligence."

That is, while human-AI comparison is turning to the advantage of AIs, so-called, in an increasing number of narrow domains, the goal of cross-domain generalization of insight seems as elusive as ever, and there doesn't seem to be a hugely promising angle of attack (in the sense that you see AI researchers swarming to explore that angle).

Meat brains are badly constrained in ways that non-meat brains need not be.

Agreed; and there's an overbroad reading of this claim, which I'm kind of worried people encountering it (e.g. in the guise of Eliezer's argument on "the space of all possible minds") can inadvertently fall into: assuming that just because we can't imagine them, there are no constraints that apply to any class of non-meat brains.

The movie that runs through our minds when we imagine "AGI recursive self-improvement" goes something like a version of Hollywood hacker movies, except with the AI in the role of the hacker. It's sitting at a desk wearing mirrorshades, and looking for the line in its own code that has the parameter for "number of working memory items". When it finds it, it goes "aha!" and suddenly becomes twice as powerful as before.

That is vivid, but probably not how it works. For instance, "number of working memory items" can be a functional description of the system, without having an easily identifiable bit of the code where it's determined, just as well in an AI's substrate as as in a human mind.

"Response: There are physical limits to how intelligent something can get, but they easily allow the intelligence required to transform the solar system."

How do you know this? I would like some more argument behind this response. In particular, what if some things are impossible? For instance, it might be true that cold fusion is unacheivable, we will never travel faster than the speed of light (even by cheating) and nanotech suffers some hard limits.

Science fiction becomes science fact

I grate my teeth whenever someone intentionally writes this cliche; "science fact" isn't a noun phrase one would use in any other context.

"Intelligence requires experience and learning, so there is a limit to the speed at which even a machine can improve its own intelligence."

It's not like pure thought alone could have ruled out, say, Newtonian mechanics.

What phrase would you use to describe the failure to produce an AGI over the last 50 years? I suspect that 50 years from now we will might be saying "Wow that was hard, we've learnt a lot, specific kinds of problem solving work well, and computers are really fast now but we still don't really know how to go about creating an AGI". In other words the next 50 years might strongly resemble the last 50 from a very high level view.

In particular, we are amassing vast computational capacities without yet understanding the algorithmic keys to general intelligence.

We are amassing vast computational capacities without yet understanding much of how to use them effectively, period. The Raspberry Pi exceeds the specs of a Cray-1, and costs a few hundred thousand times less. What will it be used for? Playing games, mostly. And probably not games that make us smarter.

A common criticism is that intelligence isn't defined, is poorly defined, cannot be defined, can only be defined relative to human beings, etc.

I guess you could lump general criticism of the computationalist approach to cognition in here (dynamicism, embodiment, ecological psychology, etc). Perhaps intelligence explosion scenarios can be constructed for alternative approaches but they seem far from obvious.

Example: "...it's more likely than not in my view that the two species will comfortably and more or less peacefully coexist--unless human interests start to interfere with those of the machines." (Casti, Tech Luminaries Address Singularity)

Even for someone who hasn't read the sequences, that sounds like a pretty huge "unless." If the two don't have exactly the same interests, why wouldn't their interests interfere?

"Humans will be able to keep up with AGI by using AGI's advancements themselves."

Response: Human engineers cannot take a powerful algorithm from AI and implement it in their own neurobiology. Moreover, once an AGI is improving its own intelligence, it's not clear that it would share the 'secrets' of these improvements with humans.

Why not? I can think of a couple possible explanations, but none that hold up to scrutiny. I'm probably missing something.

1. Humans can't alter their neural structure. This strikes me as unlikely. It is possible to create a circuit diagram of a neural structure, and likewise possible to electrically stimulate a neuron. By extension, it should be possible to replace a neuron with a circuit that does something identical to that neuron. This circuit could most certainly be altered. This may not be the most straightforward way of doing things, but it does mean alteration of neural structure is possible.

2. People would be unable to understand the developments the AI makes, and therefore could not implement them. The first part of this is probably true. I can't understand how some programs I wrote a year ago work, but I can see what the program does. I can't see why useful developments couldn't be implemented, even if we didn't understand how they worked.

People are very good at using tools like dictionaries, computer programs, and even other people to produce a more useful output than they could produce on their own. You can see this is true by looking at any intelligence test. How much better could someone do on a vocabulary test with access to a dictionary. Similarly, calculators break arithmetic tests and a computer with MatLab would allow for far superior performance on current math/logic tests. This is because the human-tool system can outperform the human alone. Two people working together can likewise outperform one person. So tools of intelligence ≤ humans can be implemented as extensions of the human mind. I don't see any reason this rule would not hold for a tool of higher intelligence than the user.

"AGI won't be a big deal; we already have 6 billion general intelligences on Earth."

Problem of scale. AGI would let us have 6 trillion general intelligences. Having a thousand intelligences that like working without pay for every human that exists "won't be a big deal"?

I think otherwise. And that's even assuming these AGIs are 'merely' "roughly human-equivalent".

There are two kinds of people here. Those who thinks that an intelligence explosion is unlikely, and those who thinks it is uncontrollable.

I think it is likely AND controllable.

On this page I will collect criticisms of (1) the claim that intelligence explosion is plausible, (2) the claim that intelligence explosion is likely to occur within the next 150 years, and (3) the claim that intelligence explosion would have a massive impact on civilization. Please suggest your own, citing the original source when possible.

Wouldn't this make more sense on the wiki?

Another criticism that's worth mentioning is an observational rather than modeling issue: If AI were a major part of the Great Filter then we'd see it in the sky from when the AGIs started to control space around them at a substantial fraction of c. This should discount the probability of AGIs undergoing intelligence explosions. How much it should do so is a function of how much of the Great Filter one thinks is behind us and how much one thinks is in front of us.

You are missing a train of argument which trumps all of these lines of criticism: the intelligence explosion is already upon us. Creating a modern microprocessor chip is a stupendously complex computational task that is far far beyond the capabilities of any number of un-amplified humans, no matter how intelligent. There was a time when chips were simple enough that a single human could do all of this intellectual work, but we are already decades past that point.

Today new chips are created by modern day super-intelligences (corporations) which in turn are composed of a mix of a variety of specialized meat brains running all kinds of natural language software and all kinds of computers running a complex ecosystem of programs in machine languages. Over time more and more of the total computational work is shifting from the meat brains to the machines. For a more concrete example of just one type of specialized machine that is a node in this overall meta-system, look at the big expensive emulation engines that are used to speed up final testing.

There is a trend to over-emphasis human's computational/intellectual contributions. When machines get better at particular sub-tasks or domains such as chess, we simply exclude them from our broad notions of 'intelligence'. Thus we tend to define intelligence at whatever humans are currently best at.

From that perspective I can predict a future intelligence extinction event as eventually all of the tasks move over to the machine substrate and eventually there are no tasks left favoring humans, and we are forced to conclude that intelligence no longer exists.

More seriously, the intelligence explosion is already well under way, we just aren't seeing it, for the most part. It's too big and abstract, too far from the simpler more visible physical manifestations we would expect.

That being said, there is certainly a further potential phase transition ahead when machine understanding of natural language matches then exceeds human level, when all of the existing human knowledge moves onto machines and they can run existing natural language 'software'. I find it likely this will lead to a further stage in the explosion, perhaps a transition to another S-curve of growth and dislocation, but such a transition would not fundamentally change either the computational problems important for progress (such as creating new chips), or the nature of the solutions.

Computers already do a huge chunk of the important work, thus completely taking humans out of the loop can only lead to limited additional gains in these fields where computers are already important (research, engineering).

Computational complexity may place strong limits on how much recursive self-improvement can occur, especially in a software context. See e.g. this prior discussion and this ongoing one. In particular, if P is not equal to NP in a strong sense this may place serious limits on software improvement.

The main problems that I see are as Eliezer told in the Singularity Summit: there are problems regarding AGI that we don't know how to solve even in principle(I'm not sure if this applies to AGI ingeneral or only to FAGI). So it might well be that we won't solve these problems ever.

The most difficult part will be to ensure the friendliness of the AI. The biggest danger is someone else carelessly making an AGI that is not friendly.

I have a reason to believe it has less than a 50% chance of being possible. Does that count?

I figure after the low-hanging fruit is taken care of, it simply becomes a question of if a unit of additional intelligence is enough to add another additional unit of intelligence. If the feedback constant is less than one, intelligence growth stops. If it is greater, the intelligence grows until it falls below one. It will vary somewhat with intelligence, and it would have to fall below one eventually. We have no way of knowing what the feedback constant is, so we'd have to guess that there's a 50% chance of it being above 1, and a 50% chance of being below. Furthermore, it's unlikely to be right next to 1, so if it is possible, it will most likely get pretty darn intelligent.

Also, I figure that if the constant is below 1, or even close, humanity will be incapable of creating an AI from scratch, though if they get uploaded they could improve themselves.

This could be its own web site.

Actually, the intelligence explosion has been in progress for a long time now - and effective intelligence is already increasing with unprecidented rapidity.

It is a combination of bad terminology and misunderstandings that pictures an explosion in intelligence as being a future event.