A Premature Word on AI

Ok, sure. Maybe Bayesianism is much more broadly applicable than it seems. And maybe there are fewer fundamental breakthroughs still needed for a sufficient RSI-AI theory than it seems. And maybe the fundamentals could be elaborated into a full framework more easily than it seems. And maybe such a framework could be implemented into computer programming more easily than it seems. And maybe the computing power required to execute the computer program at non-glacial speeds is less than it seems. And maybe the efficiency of the program can be automatically increased more than seems reasonable. And maybe "self improvement" can progress further into the unknown than seems reasonably to guess. And maybe out there in the undiscovered territory there are ways of reasoning about and subsequently controlling matter that are more effective than seems likely, and maybe as these things are revealed we will be too stupid to recognize them.

Maybe.

To make most people not roll their eyes at the prospect, though, they'll have to be shown something more concrete than a "Maybe AI is like a nuclear reaction" metaphor.

You presume your know that a few big innovations are the key, rather than many small innovations. (See my added to today's post.) My sense is that the standard A.I. old-timer only puts moderate weight on computing power limitations - it is more about lacking enough small innovations, seeing how slow they come and how many seem needed. And I don't see who has made claims about the "production of intelligence in the vicinity of human."

Bambi, the same line of argument would cause any observers, standing off observing the course of Earthly evolution, to conclude that these so-called hypothesized "humans" were impossible.

Actually, the same @Hanson. If there weren't deep keys to optimization, humans would be unable to achieve a sudden huge advantage over other lifeforms, using brains optimized in the entire absence of direct selective pressures for inventing guns or space shuttles. It would just be a matter of slowly evolving swords, slowly evolving guns, slowly evolving nuclear weapons...

It seems to me that the evidence available to us, in which to fight out this disagreement, consists mainly of the course of biological evolutionary history. Where we see that modern AI researchers are stuck in very much the mode of evolution producing prehuman lifeforms: They have to do lots of work to produce one special-purpose cognitive system.

So they conclude that general-purpose systems are impossible, based on historical experience. QED. Humans are a counterexample, but the actual disappointing laboratory experience seems so much more salient than that.

You presume your know that a few big innovations are the key, rather than many small innovations.

That isn't bambi's presumption. Take a look at the things Eliezer has claimed about Bayesian methodology, especially in regards to the scientific world.

Robin, How do old-timers address reverse-engineering the human brain + computer power increases? Folks like Kurzweil predict that'll be done by the middle of this century. If the old-timers are skeptical about that, what the articulated criticisms?

@HA: AI is a huge field - maybe Robin and I should stick to discussing ideas that at least one of us thinks is plausible.

Human behavior may require many small innovations plus a few big ones, at least one of which our immediate primate ancestors lacked. Observed difficulties of modern A.I. researchers could come from them having all of the needed big ones, but only a few of the small innovations. In this case they would be correct to say that what holds them back is collecting lots of small innovations.

Eliezer, your observers would hopefully have noticed hundreds of millions of years of increasing world-modeling cognitive capability, eventually leading to a species with sufficient capacity to form a substrate for memetic progress, followed by a hundred thousand years and a hundred billion individual lives leading up to now.

Looking at a trilobyte, the conclusion would not be that such future development is "impossible", but perhaps "unlikely to occur while I'm eating lunch today".

Most progress is NOT an exponential increase. There have been past periods of remarkably rapid change, and they've always tended to peter out after a while - leading to lengthy periods of stagnation or regression at worst and slow increase at best.

As Egan has pointed out, exponential increase is a curse - it either hits up against unsurmountable obstacles and ceases or depletes its own resource base and destroys itself. We do not find bacteria suddenly mutating into grey goo and consuming the Earth, nor do we find cockroaches dominating all other forms of life. Even ecologically invasive organisms fall back into sustainable patterns sooner or later, or they go extinct.

Sudden, world-altering transitions are possible, but they're extremely rare. Given that so many artificial intelligence advocates have been extremely optimistic about future progress, and past predictions have been so consistently overstepping what actually happened, that we should be extremely skeptical about foreseeing a sudden jump.

@Bambi: Your observers are not anywhere remotely near imaginative enough to suppose something like a human, having never seen a human and having "no reason to believe anything like that is possible". They would simply expect evolution to continue the way it always has. In hindsight, whatever happens in AI, whatever it may be, will seem just as inevitable as the way you're making it sound; the trick is seeing it in advance, for which purpose the future is always absurd.

@Hanson: Do you seriously put forth such a view, or are you playing Devil's Advocate? It seems to me that in my chosen challenge of reflectivity there are basic things that I do not understand, and while it is always extremely dangerous to say this sort of thing, I don't know of anyone else who does understand it. The notion that we have collected all major insights needed for AI sounds very odd to me; people just don't seem that unconfused.

Eliezer, see here. Our data on bio, tech, and business innovation suggests most value comes from many small innovations. My experience in A.I. fits this applying to A.I. also.

Looking from mostly-outside the field (in which I've been interested in for decades but haven't had opportunity to be seriously involved), it seems to me now that part of the problem is that there are so many different possible goals and levels of achievement which might be called "AI" but which won't really satisfy most people once the chrome wears off. (Every new computing technology seemed to be calling itself AI for awhile after computing first became widely discussed -- just doing arithmetic was once "clearly" the domain of intelligence; later, it only applied to really clever things like expert systems [/sarcasm]. I haven't seen so much of that lately, so either the common understanding of computing has become more realistic or perhaps I've just trained myself to ignore it.)

For instance:

Let's say I managed to hook together an English parsing engine, a common-sense engine/database (such as Cyc), and an inference engine in such a way as to be able to take factual knowledge (such as that which might be expressed in English as "Woozle usually goes shopping in the mornings on Mondays and Fridays" plus sensed information such as the current day/time and my apparent or explicit absence from my desk) and respond to natural-language questions like "Where is Woozle?" with something like "Woozle is not here right now; Woozle has probably gone shopping and should be back by midday" or even "Woozle has never returned home from Friday shopping later than 11:23, And most likely* will be back by 10:37." (Where the software understands that "most likely" is a reasonable expression to use for, say, a 2-sigma variance or something like that.)

Would I have created AI? Or just a really useful piece of software? (Either way, why aren't there more programs which pull together these techniques? Or am I making too many assumptions about what an "inference engine" can do? According to Wikipedia, there are even "reasoning engines", which sounds extremely shiny...)

If that's not enough, then what's left? Creativity? Inductive reasoning?

In other words... where are we? Where's the roadmap of what's been done and what needs to be done?

Eliezer:

This is NOT premature! You just saved yourself at least one reader you were about to lose (me). I (and I suspect many others) have not been among the most regular readers of OB because it was frankly not clear to me whether you had anything new to say, or whether you were yet another clever, but ultimately insubstantial "lumper" polymath-wannabe (to use Howard Gardner's term) who ranged plausibly over too many subjects. Your 'physics' series especially, almost made me unsubscribe from this blog (I'll be honest: though you raised some interesting thoughts there, that series did NOT impress me).

But with this post, for what it is worth, you've FINALLY seriously engaged my attention. I'll be reading your follow-up to this thread of thinking very carefully. I too, started my research career fascinated by AI (wrote an ELIZA clone in high school). Unlike you, my more mature undergrad reaction to the field was not that AI was hard, but that it was simply an inappropriate computer science framing of an extremely hard problem in the philosophy of mind. I think you would agree with this statement. Since philosophy didn't seem like a money-making career, my own reaction was to steer towards a fascinating field that neighbors AI, control theory (which CS people familiar with history will probably think of as "stuff beyond subsymbolic AI, or modern analog computing"). Back when I started grad school in control theory (1997), I was of the opinion that it had more to say about the philosophy problem than AI. My opinion grew more nuanced through my PhD and postdoc, and today I am a somewhat omnivorous decision-science guy with a core philosophical world-view based on control theory, but stealing freely from AI, operations research and statistics to fuel my thinking both on the practical pay-the-bills problems I work on, as well as the philosophy problem underlying AI.

Oddly enough, I too have been drafting my first premature essay on AI, corresponding to yours, which I have tentatively titled, "Moving Goalposts are Good for AI." I should post the thing in the next week or two.

I suspect you won't agree with my conclusions though :)

Woozle, I agree with your sentiments, but I think there's an even more important question that needs to be asked. Perhaps it's another way of stating your 'roadmap' question.

What do we mean by 'intelligence'? Science emphasizes the importance of producing clear, functional definitions of terms before any work should be attempted - because that's what's needed to make progress. Where are our clear definitions of the focal concept? What precisely are we trying to accomplish?

The field has never managed to explicitly state precisely what it's trying to produce... and rather unsurprisingly it's never managed to accomplish a great deal, except for specific programs that set specific goals.

Artificial stupidity, in the sense of systems that specialize in simple, definable behaviors and goals, will almost certainly be much earlier than AI, but relatively few people are working on it.

Woozle, I agree with your sentiments, but I think there's an even more important question that needs to be asked. Perhaps it's another way of stating your 'roadmap' question.

What do we mean by 'intelligence'? Science emphasizes the importance of producing clear, functional definitions of terms before any work should be attempted - because that's what's needed to make progress. Where are our clear definitions of the focal concept? What precisely are we trying to accomplish?

The field has never managed to explicitly state precisely what it's trying to produce... and rather unsurprisingly it's never managed to accomplish a great deal, except for specific programs that set specific goals.

Artificial stupidity, in the sense of systems that specialize in simple, definable behaviors and goals, will almost certainly be much earlier than AI, but relatively few people are working on it.

I largely agree with Robin's point that smaller incremental steps are necessary.

But Eliezer's point about big jumps deserves a reply. The transitions to humans and to atomic bombs do indicate something to think about -- and for that matter, so does the emergence of computers.

These all seem to me to be cases where the gradually rising or shifting capacities encounter a new "sweet spot" in the fitness landscape. Other examples are the evolution of flight, or of eyes, both of which happened several times. Or trees, a morphological innovation that arises in multiple botanical lineages.

Note that even for innovations that fit this pattern, e.g. computers and atomic bombs, enormous amounts of incremental development are required before we can get to the sweet spot and start to expand there. (This is also true for biological evolution of course.)

I think most human innovations (tall building, rockets, etc.) are due to incremental accumulation of this sort, rather than finding any big sweet spots.

I should also note that decades before the atomic bomb, the actual production of energy from nuclear fission (geothermal) and fusion (the sun) was clear, if not understood in detail. Similarly the potential of general purpose computers was sensed (e.g. by Ada Lovelace) far before we could build them. This foreknowledge was quite concrete -- it involved detailed physical accounts of existing sources of energy, automation of existing computing techniques, etc. So this sort of sweet spot can be understood in quite detailed ways well before we have the technical skills to reach it.

Using this model, if AGI arrives rapidly, it will be because we found a sweet spot, over and above computing. If AGI is feasible in the near future, that implies that we are near such a sweet spot now. If we are near such a sweet spot, we should be able to understand some of its specific form (beyond "it uses Bayesian reasoning") and the limitations that keep us from getting to it immediately.

I agree with Eliezer that Bayesian methods are "forced", and I also feel the "Good Old Fashioned AI" folks (certainly including Shank and McCarthy) are not good forecasters, for many reasons.

However Bayesian approaches are at the root of existing impressive AI, such as Thrun's work on autonomous vehicles. I have been watching this work fairly closely, and it is making the normal sort of incremental progress. If there's a big sweet spot nearby in the fitness landscape, these practitioners should be able to sense it. They would be well qualified to comment on the prospects for AI, and AGI in particular. I would be very interested in what they have to say.

Eliezer: Good article. I always feel angry when I read some futurist's claim where there is no real scientific or bayesian basis for it, but because the futurist has some status this is put forth as the gospel. It's a social game, people posing as authorities when they didn't do their homework. Maybe we should apply the saying: "Shut up and calculate!"

PS: The same can be said for all kind of gurus in other fields as well.

To echo what Caledonian said my biggest point of skepticism is the notion of "general intelligence." I've seen people try to define general intelligence, say as the ability to adapt to a changing environment, but these definitions tend to be circular.

Eliezer appears to be of the opinion that human cultural evolution necessitates general intelligence. I don't buy it. The cultural evolution of swords and guns certainly didn't take the form of somebody sitting down thinking about the problem until blood beaded on their brow. Most periods of progress have been marked by ecological change; either environmental change, the interaction of many previously isolated cultures or the introduction of a new tool. We even acknowledge this when we talk about exponential progress; human capabilities haven't change, it's the positive feedback created by interactions between our tools that has created modern society.

So, even if we define general intelligence arbitrarily as what humans do that other animals don't, I don't see strong evidence for it. Obviously there are difference between humans and other animals but why can't specific difference like tool-use and language explain human cultural evolution? Why do we need to also hypothesize general intelligence (even as an outcome of the specific differences)?

To me you might as well be talking about "general splendidness." Some of the things I've seen written about super-intelligence even read like they are talking about general splendidness or general awesomeness or some other entirely arbitrary property of being-able-to-do-whatever-I-need-it-to.

Eliezer, a refuting link would be appreciated, so I can get a better sense of where you're coming from (and the strength/weakness of your position) when you dismiss functional models of human brains appearing before AGI.

@HA: The usual: http://singinst.org/AIRisk.pdf

Section 11, in your case.

The other objections along the lines of, "Oh my God, I've never seen anyone define "intelligence" or "general intelligence", it must have never ever been done by anyone on Earth, how stupid Eliezer must be for discussing this with such blind faith!" are about matters discussed either at the link above, or in a draft of mine up at http://www.sl4.org/wiki/KnowabilityOfFAI.

If you think there's something obvious that has never ever been discussed, chances are extremely high that you are ignorant of discussions that have already taken place. Among the people who, like, actually think about this stuff. This specifically excludes all the sloppy futurism you've read in popular sources like IEEE Spectrum. The real analysts are not stupid. I, personally, am not stupid. Thank you for appreciating this.

I mean, go ahead and ask your questions. Just don't imply that I've never deeply thought about Singularity issue XYZ, foolish mortal.

The bottom line here is that Schank (and Brooks and Norvig) has vastly more experience, and celebrated achievements, in trying to build a general A.I. Schank tried to summarize what he has learned about what sort of rates of progress we can reasonably expect, and of course he finds it hard to articulate precisely the reasons for those judgments. Because Schank admits to being too optimistic in his youth, Eliezer feels free to dismiss Schank's judgment as "sloppy futurism", and so worth far less weight than his own analysis. If we think we know why Schank would dismiss Eliezer's judgement here, we have a good disagreement case study here to consider.

Eliezer, I read the link to section 11. After having done so, I strongly encourage other commentors here to read Eliezer's piece also. In large part, because I think you, Eliezer, are overstating what you've accomplished with this piece. I won't go into detailed criticism here, but instead, I'll restrict it narrowly to my original question in this thread.

The piece you linked to doesn't in any authoritative way dismiss the possibility that functional human brain modeling will occur before AGI is developed. If anything, the piece seems to cede a reasonable chance that either could happen before the other, and then talks about why you think it's important for AGI (of the friendly variety) to be developed before function human brain modeling reaches a certain threshhold level of being likely to result in catastrophic risk.

If your piece represents the apex of critical thinking on this topic, we're in the very, very beginning stages, and we need, in my opinion, to attract a much higher caliber of thought and intellectual work to this topic. For example, I think it's a step down from the level of rigor Kurzweil has put into thinking about the topic of likelihood of AGI vs. brain modeling and timelines.

So, I'm still interested in Robin's take and the take of old-timers on this topic.

If you think there's something obvious that has never ever been discussed, chances are extremely high that you are ignorant of discussions that have already taken place.

The nature of general intelligence has been discussed ad nauseam, but we've been extraordinarily unproductive in defining that property in humans, much less working out a reliable way to measure it, and we certainly haven't made much progress in reproducing it.

Hardware power has increased steadily. Has development in software kept pace? Why do you think that is?

@HA: No, it's not a knockdown crush. It's one short section out of a long document. Maybe I'll do a knockdown crush on OB, or SIAI's blog, later after having had time to build more background. You might be able to find more detailed discussion in the SL4 / Extropians / wta-talk archive, or not - even I don't remember what's in there any more. Certainly, though, it's an idea that's been discussed.

@Robin:

Because Schank admits to being too optimistic in his youth, Eliezer feels free to dismiss Schank's judgment as "sloppy futurism", and so worth far less weight than his own analysis.

There is a sharp distinction between what Schank may have learned about the human mind in his career - admittedly, I wouldn't put too much weight even on that, because I think my era supports me; most of the Elders here are formidable old warriors with hopelessly obsolete arms and armor - but anyway, there's a sharp distinction between what Schank knows about cognitive science, and how he applies that to make predictions six decades out. Sort of akin to the distinction between what Einstein knew about physics, and how he applied it to talk about an impersonal deity who must have created the universe. There are people who aspire to be careful in all realms including futurism, but they are very rare, and I see no evidence so far that Schank is one of these people.

Eliezer: human development looks to me like a poor source for insights on general intelligence. Consider the many human traits - hunting, tools, clothing, fire - that pre-date any hint of generality and ought really to be considered instinct. Consider that behavioral modernity is something like 50 kiloyears old, anatomically modern humans existed without a hint of culture for about 150 kiloyears before that point, and some isolated groups of anatomically modern, cultured, talking, music-making humans remained stuck in the stone age until the 1960s. Also, consider how few car-driving humans can perform the math that defines their car. It's evident to me that we are so very marginally generally intelligent as a species, that you would be hard put to nail down the signal amongst the noise.

@Julian: Correct, but if you step outside the evolutionary biology of human intelligence, there's no way anyone can follow you except by being able to do high-falutin' theoretical thinking of their own. Meanwhile, the actual evolutionary history strongly contradicts assertions like "you need exponential computing power for linear performance increases" or "the big wins are taken at the start so progress is logarithmic with optimization power expended on the problem".

Also @HA: I once asked Kurzweil why he kept talking about human brain modeling. He said, in effect, "Because that way we don't have to understand intelligence." I said, in effect, "But doing it without understanding is always going to be the most difficult way, not the least difficult way." He said, "But people aren't willing to believe you can understand intelligence, so I talk about human brain modeling because it clearly shows that AI is possible." I said, "That's a conservative assumption in some ways, but you're using it to reassure people about the benevolence of AIs, which is not conservative; you can't get real futuristic predictions by presuming that the thought experiments easiest for the public to accept are the way things will actually happen in real life." If K has ever responded to that, I have not seen it.

The fundamental folly in AI is trying to bypass the hard problems instead of unraveling the mysteries. (And then you say you can build AI in 10 years, even though you're not sure exactly how it will work, because you think you can get by without understanding the mysteries...) Modeling the human brain fits exactly into this pattern. Now, just because something fits the pattern of a folly, does not make it technologically impossible; but if you really wanted the case for abstract understanding and against brain modeling, that would be a longer story. It involves observations like, "The more you model, the more you understand at the abstract level too, though not necessarily vice versa" and "Can you name a single case in the history of humanity where a system has been reverse-engineered by duplicating the elementary level without understanding how higher levels of organization worked? If so, how about two cases? Are either of them at all important?"

There's a reason why we don't build walking robots by exactly duplicating biological neurology and musculature and skeletons, and that's because, proof-of-concept or not, by the time you get anywhere at all on the problem, you are starting to understand it well enough to not do it exactly the human way.

Many early "flying machines" had feathers and a beak. They didn't fly. Same principle. The only reason it sounds plausible is because flying seems so mysterious that you can't imagine actually, like, solving it. So you imagine doing an exact anatomical imitation of a bird; that way you don't have to imagine having solved the mystery.

If you want more than that, it'll take a separate post at some point.

Poke's comment is interesting and I agree with his / her discussion of cultural evolution. But it also is possible to turn this point around to indicate a possible sweet spot in the fitness landscape that we are probably approaching. Conversely, however, I think the character of this sweet spot indicates scant likelihood of a very rapidly self-bootstrapping AGI.

Probably the most important and distinctive aspect of humans is our ability and desire to coordinate (express ourselves to others, imitate others, work with others, etc.). That ability and desire is required to engage in the sort of cultural evolution that Poke describes. It underlies the individual acquisition of language, cultural transmission, long term research programs, etc.

But as Eric Raymond points out, we are just good enough at this to make it work at all. A bunch of apes trying to coordinate world-wide culture, economy and research is a marginal proposition.

Furthermore we can observe that major creative works come from a very small number of people in "hot" communities -- e.g. Florence during the Renaissance. As Paul Graham points out, this can't be the result of a collection of uniquely talented individuals, it must be some function of the local cultural resources and incentives. Unfortunately I don't know of any fine grained research on what these situations have in common -- we probably don't even have the right concepts to express those characteristics.

A mundane version of this is the amazing productivity of a "gelled team", in software development and other areas. There is some interesting research on the fine grained correlates of team productivity but not much.

So I conjecture that there is a sweet spot for optimized "thinking systems" equivalent to highly productive human teams or larger groups.

Of course we already have such systems, combining humans and digital systems; the digital parts compensate for human limitations and decrease coordination costs in various ways, but they are still extremely weak -- basically networked bookkeeping mechanisms of various sorts.

The natural direction of evolution here is that we improve the fit between the digital parts and the humans, tweak the environment to increase human effectiveness, and gradually increase the capabilities of the digital environment, until the human are no longer needed.

As described this is just incremental development. However it is self-accelerating; these systems are good tools for improving themselves. I expect we'll see the usual sigmoid curve, where these "thinking systems" relatively quickly establish a new level, but then development slows down as they run into intrinsic limitations -- though it is hard to predict what these will be, just as Ada Lovelace couldn't predict the difficulties of massively parallel software design.

From here, we can see a sweet spot that is inhabited by systems with the abilities of "super teams", perhaps with humans as components. In this scenario any super team emerges incrementally in a landscape with many other similar teams in various stages of development. Quite likely different teams will have different strengths and weaknesses. However nothing in this scenario gives us any reason to believe in super teams that can bootstrap themselves to virtual omniscience or omnipotence.

This development will also give us deep insight into how humans coordinate and how to facilitate and guide that coordination. This knowledge is likely to have very large consequences outside the development of the super teams.

Unfortunately, none of this thinking gives us much of a grip on the larger implications of moving to this sweet spot, just as Ada Lovelace (or Thomas Watson) didn't anticipate the social implications of the computer, and Einstein and Leo Szilard didn't anticipate the social implications of control over nuclear energy.

Eliezer, in your AIRisk paper, at the end of section 11 you summarize your position:

I do not assign strong confidence to the assertion that Friendly AI is easier than human augmentation, or that it is safer. There are many conceivable pathways for augmenting a human. Perhaps there is a technique which is easier and safer than AI, which is also powerful enough to make a difference to existential risk. If so, I may switch jobs. But I did wish to point out some considerations which argue against the unquestioned assumption that human intelligence enhancement is easier, safer, and powerful enough to make a difference.

Eliezer, Schank and Brooks and Norvig and all the distinguished authors at IEEE Spectrum all live in your era! They are writing now, now fifty years ago. You call them "formidable old warriors with hopelessly obsolete arms and armor", and say they do not "aspire to be careful in all realms including futurism."

I'm sure they are keeping up with some developments, and I'm sure they think they are trying to be careful. So it seems to come down to your thinking you know better than them all what are the really important developments and the real ways to be careful.

I will further point out that the past of human cognitive development strongly indicates that our progress isn't a function of our brainpower alone, but our cultural inheritance - and that the increase in complexity and power of our societies has been in large part dependent upon population numbers.

It may well be the case that exponentially-increases are necessary for linear improvements. Maybe not, but human history and prehistory don't seem to constitute arguments against the possibility.

@Hal: "Human augmentation" means things like neuropharmeceuticals, gene therapy, or even brain-computer interfaces, not full-scale human-equiv uploading and simulation before the first AI.

Robin, I would consider Norvig a very separate issue from Schank or Brooks. Either that or Stuart Russell wrote all the good parts of "AI a modern approach" and I doubt he did.

Anyway, as I originally suggested here, being born more than fifty years before someone - in some cases, more than twenty years before someone - is good cause to be suspicious of trying to use Aumann on them. Of course I'm sure you can find many bright <50 researchers who would agree with Schank!

But in any case, maybe we should swap the whole scenario to consider my disagreement with Norvig, whom I confess to be a powerful and prestigious mind with modern arms and armor?

Eliezer: As I said, there are plenty of circular definitions of intelligence, such as defining it as an "powerful optimization process" that hones in on outcomes you've predefined as being the product of intelligence (which is what your KnowabilityOfAI appears to do). Perhaps for your needs such a (circular) operational definition would suffice: take the set of artifacts and work backwards. That hardly seems helpful in designing any sort of workable software system though.

Re: modeling the human brain. Modeling the human brain would involve higher levels of organization. The point is that those higher levels of organization would be actual higher levels of organization that exist in real life and not the biologically implausible fantasies "AI researchers" have plucked out of thin air based on a mixture of folk psychology, introspection and wishful thinking.

Even talking about intelligence as optimization seems questionable, as the majority of intelligent behaviors don't involve optimization at all. Quite the opposite, in fact - we're able to set our own goals, and experience a "reward sensation" when we accomplish them. If we optimized our ability to induce a reward in ourselves, we would very quickly decay into setting trivially easy goals and basking in the glow of our success.

As it is, we get bored and distracted. When humans tell stories, which seems to be a fundamental part of our psychology and is possibly one of the very few things that are uniquely human, we don't optimize them. We elaborate upon them, making them 'interesting'. Creativity isn't an optimization process, although its results can be evaluated by optimization-seeking processes with their own goals.

Are we sure that we're not treating intelligence as optimization simply because optimization is something that we already have a great deal of mathematical work on?

Caledonian, I think Eliezer's going off of his distinction (in Knowability of AI and elsewhere) between "optimal" and "optimized", which more colloquial senses of the words don't include. There may be more optimal ways of achieving our goals, but that doesn't take away from the fact that we regularly achieve results that

(1) we explicitly set out to do (2) we can distinguish clearly from other results (3) would be incredibly unlikely to achieve by random effort.

I.e. this comment isn't close to optimal, but it's optimized enough as a coherent reply in a conversation that you'd ascribe a decent level of intelligence to whatever optimization process produced it. You wouldn't, say, wonder if I were a spambot, let alone a random word generator.

"There's a reason why we don't build walking robots by exactly duplicating biological neurology and musculature and skeletons, and that's because, proof-of-concept or not, by the time you get anywhere at all on the problem, you are starting to understand it well enough to not do it exactly the human way."

That's trivially (usually) true, which is why I'm curious about the degree to which AI old-timers have considered and dismissed human brain modeling in conjunction with the types of exponential technological increases Kurzweil publicized when they claim that their experience leads them to believe AGI isn't likely to arise for several generations (on the order of 100 years), rather than in one generation. Particularly if they've been working mostly on abstract AGI rather than doing substantial work with human brain modeling, too.

Eliezer, it seems you are saying that all else equal a younger person is more likely to be right in a disagreement. This is why you feel comfortable disagreeing with "old-timers." But I'm guessing you don't apply that logic to people younger than yourself, that you have some other considerations that end up making your age the age near most likely to be right. I'll also guess that twenty years from now you will have found some other considerations that make that age near the best.

another clever, but ultimately insubstantial "lumper" polymath-wannabe who ranged plausibly over too many subjects.

I hope Venkat is not implying that it is better to specialize in one science or technology. The decision to specialize (and the related decision never to be seen making a mistake) is IMHO the how many scientists and technologists limit their own effectiveness and ability to be responsible.

I think that Eliezer and Robin are both right. General AI is going to take a few big insights AND a lot of little improvements.

I think that Eliezer and Robin are both right. General AI is going to take a few big insights AND a lot of small improvements.

Eliezer--

In principle, it seems like AI should be possible. Yet what reason do we have to think we will work it out on any time frame worth discussing? It's worth reflecting on what the holy grail of AI seems to be: creating something that combines the best of human and machine intelligence. This goal only makes sense unless you think they are two very different things. And the differences run deeper than that. We have:

Different driving forces in development: gene survival vs. specific goals of thinking humans Different design styles: a blind watchmaker notorious for jury-rigged products, working over a very long period of time vs. design by people with significant ability to reflect on what they're doing, working over a short period of time Different mechanisms for combining innovations: sex vs. sharing information in a way accessible to minds Different micro-circuitry: digital logic circuits vs. neurons--messy bags of cytosol relying on neurotransmitters, ions, ion channels, all devices with analog behavior *Different overall design vision: Von Neumann architecture vs. nothing so straightforward as von Neumann architecture.

Do you get what I'm saying? Why think that within a few lifetimes we can replicate the fruits of millions of years of evolution, using something entirely different than the thing we're trying to replicate?

Oh, and chew on this: we don't even understand how the nematode worm brain works.

It's better to spend your life trying and failing than doing nothing at all. And a lot of people, unless they're given a significant goal, won't even try. General AI? Let's get to it. Edge detection in a visual field? Yawn. If that's my life might as well sit around and play Xbox all day.

Regarding the definition of "intelligence": It's not hard to propose definitions, if you assume the framework of computer science. Consider the cognitive architecture known as an "expected-utility maximizer". It has, to a first approximation, two parts. One part is the utility function, which offers a way of ranking the desirability of the situation which the entity finds itself in. The other part is the problem-solving part: it suggests actions, selected so as to maximize expected utility.

The utility function itself offers a way to rate the intelligence of different designs for the problem-solving component. You might, for example, average the utility obtained after a certain period of time across a set of test environments, or even across all possible environments. The point is that the EUM is supposed to be maximizing utility, and if one EUM is doing better than another, it must be because its problem-solver is more successful.

The next step towards rating the intrinsic intelligence of the problem-solving component is to compare its performance, not just across different environments, but when presented with different utility functions. Ideally, you would take into account how well it does under all possible utility functions, in all possible environments. (Since this is computer science, a "possible environment" will have a rather abstract definition, such as "any set of causally coupled finite-state machines".)

There are issues with respect to how you average; there are issues with respect to whether the "intelligence" you get from a definition like this is actually calculable. Nonetheless, those really are just details. The point is that there are rigorous ways to rank programs with respect to their ability to solve problems, and whether or not such a ranking warrants the name "intelligence", it is clearly of pragmatic significance. An accurate metric for the problem-solving capability of a goal-directed entity tells you how effective it will be in realizing those goals, and hence how much of an influence it can be in the world.

And this allows me to leap ahead and present a similarly informal account of what a Singularity is, what the problem of Friendliness is, and what the proposed solution is. A Singularity happens when naturally evolved intelligences, using the theory and technology of computation, create artificial intelligences of significantly superior problem-solving capability ("higher intelligence"). This superiority implies that in any conflict of goals, a higher intelligence wins against a lower intelligence (I'm speaking in general), because intelligence by definition is effectiveness in bringing about goal states. Since the goals of an artificial intelligence are thoroughly contingent (definitely so in the case of the EUM cognitive architecture), there is a risk to the natural intelligences that their own goals will be overwhelmed by those of the AIs; this is the problem of Friendliness. And the solution - at least, what I take to be the best candidate solution - is to determine the utility function (or its analogue) of the natural intelligences, determine the utility function of an ideal moral agent, relative to the preferences of that evolved utility function, and use that idealized utility function as the goal system of the artificial intelligences.

That, in schema, is my version of the research program of Eliezer's Institute. I wanted to spell it out because I think it's pretty easy to understand, and who knows how long it will be before Eliezer gets around to expounding it here, at length. It may be questioned from various angles; it certainly needs much more detail; but you have, right there, a formulation of what our situation is, and how to deal with it, which strikes me as eminently defensible and doable.

Mitchell, I think the No Free Lunch theorems of Wolpert and Macready imply that if you take two EUMs and average their results over all possible problems, the two will be necessarily indistinguishable. So your measure of intelligence would imply that unequal general intelligence is impossible.

Of course, you could probably rescue your definition by not including all possible problems, but only realistic ones. In other words, the reason it is possible to derive the NFL theorems is because you can invent a theoretical world where every attempt to accomplish something in a reasonable manner fails, while every unreasonable attempt succeeds. With more realistic limitations on the set of possible problems, you should be able to define general intelligence in the way stated.

Re: "knockdown crush" - here's Peter Voss on what seems to me to be obvious:

"His [Kurzweil's] approach is fundamentally looking at the wrong solution. He is talking about reverse engineering the human brain. Now, to me that's really, really hard way to solve the problem, you know, it's using tools that are really not well suited for building a human brain or reverse engineering the human brain. So I favour the engineering approach, where you say what is the problem we are trying to solve, and how can we best solve it using the technology and the tools and the equipment we have available. So that's the approach we are using - reverse engineering the brain I think is really, really hard - and that could take many decades."

• http://www.singinst.org/media/interviews/petervoss

Tim, what seems obvious to me is that the uncertainty bars on this topic seem to be large, even with the best takes of experts on it. Incidentally, a question like this (and various subsidiary questions pegged to incremental progress in both AGI and brain modeling) seems to me to be well-suited to prediction markets. What'll come first: a complete, functional model of a mouse brain, or a chatbot that can consistently pass a version of the Turing Test administered by a median competent 2008 era 8 year old over the course of 1 hour. Will either arrive in the next 20 years? It seems to me that the expert consensus "we don't know". It's frustrating to me how reported knowledge is warped by expressed overconfidence in particular predictions. It seems to me folks doing so are engaging in status/heirarchy plays. I think they should be socially punished, rather than rewarded, by the rest of us when they misleadingly express confidence in a prediction rather than give us their best estimates and models of apparent reality.

@HA: Eliza has fooled grown-ups. It arrived 42 years ago.

@Eliezer: I oppose Venkat, please stick to the logically flowing, inferential distance strategy. Given the subject, the frustration's worth it to build a solid intuitive foundation.

@ Robin Hanson:

"Because Schank admits to being too optimistic in his youth, Eliezer feels free to dismiss Schank's judgment as "sloppy futurism", and so worth far less weight than his own analysis."

I don't think this comment is fair at all. You quoted Schank as saying, "It seemed a safe answer since no one could ever tell me I was wrong." At least to my mind, when I read this I think Schank is essentially saying, "Yeah, I'll admit it, I was doing sloppy futurism because I figured I could get away with it." In which case, accusing him of being a "sloppy futurist" doesn't seem unwarranted.

@ mitchell porter:

"...to compare its performance, not just across different environments, but when presented with different utility functions."

If you look at my formal definition of "environments" in my universal intelligence paper I actually take into account all utility functions, and also all temporal preference functions. Essentially these are built into the environments.

@ Unknown:

"I think the No Free Lunch theorems of Wolpert and Macready imply that if you take two EUMs and average their results over all possible problems, the two will be necessarily indistinguishable. So your measure of intelligence would imply that unequal general intelligence is impossible."

Yes, that is what would happen if you simply took the expectation with respect to a uniform distribution over the environments. However, if you take an Occam's Razor prior, such as the algorithmic probability prior used in Solomonoff induction extended to environments (as done by Hutter), then the NFL theorems no longer hold. This is the reason why my mathematical definition of universal intelligence doesn't have the problem you describe.

Moravec on a related issue:

"Expert systems and neural nets are nonspecific programming techniques for encoding decision criteria and learning input-output relationships, that are often inferior to more problem-specific encoding, inference and learning algorithms. It is highly improbable that future fully intelligent machines will be built with either technique alone, though both may be used in places. Blind biological evolution may be stuck with solutions once chosen, but intelligence-guided technology is not so limited."

http://www.frc.ri.cmu.edu/~hpm/project.archive/general.articles/1994/940430.Brainmakers.review.html

Even Kurzweil talks about the role of the conventional AI toolbox:

"There are many intricate ways to combine the varied methods in AI's toolbox. For example, one can use a genetic algorithm to evolve the optimal topology (organization of nodes and connections) for a neural net or a Markov model. The final output of the GA-evolved neural net can then be used to control the parameters of a recursive search algorithm. We can add in powerful signal- and image-processing techniques that have been developed for pattern-processing systems. Each specific application calls for a different architecture."

• Ray Kurzweil on http://www.kurzweilai.net/articles/art0683.html?printable=1

Patrick (orthonormal), if we're going to change the meaning of the term that dramatically, we might as well run about being terrified of ant colonies. They meet all of the criteria you mention, after all. Shouldn't we be afraid they'll turn the entire world into their hive?

Like 'grey goo' nanotech scenarios, runaway self-improving AI is a conceivable danger, but almost certainly not a likely one. The obstacles in the way of robust general intelligence, as those in the way of total or even partial mass conversion, are quite prohibitive.

On a more serious note, Eliza only fooled people 1) back before there were lots of chatbots, so the idea of a computer conversation wouldn't tend to enter into people's minds and 2) who thought they were talking to a psychotherapist, which establishes such low expectations for intelligence and coherence of conversation that it's not really a fair test of the ability to masquerade as human.

Even the successful chatbots only operate in domains where conversations are inanely stupid.

Recoving Irrationalist. I just tried out Eliza (I tried it like most of us, years ago pre-internet). It impressed me a lot compared to modern competition-winners like jabberwocky. If entered into the same competition I think it should rightfully beat them.

However, I don't know if Eliza would fool a median 8 year old over the course of 1 hour (it's different from fooling some adults over the course of a couple minutes). My convo with Eliza (unfortunately the first 10 lines or so got erased, but it's similar to what remained):

"* What's your name?

Names don't interest me.

• Names interest most humans. That is quite interesting.
• Names interest most humans. Do you expect a different answer by repeating yourself?
• I'm testing the limits of your intelligence, creativity, or programming. Do you believe it is normal to be testing the limits of my intelligence, creativity, or programming?
• I'm testing the limits of your intelligence, creativity, or programming. Come, come, elucidate your thoughts.
• I think that fhhgk dkfgh dhfkg djfjg. Do you doubt you that fhhgk dkfgh dhfkg djfjg?
• Yes, I doubt that you can discern between gibberish and real words. We were discussing you, not me."

You're treating psychiatry quite harshly there Caledonian - I would rather say that people expect a conversation in psychotherapy to be very one-sided, with the human bringing all the information and the doctor/Eliza to be basically nodding away. Which definitely isn't the experience I had of (hospital) psychiatry, but the image of Freud listening to someone on a couch talking about his childhood is still very strong.

Robin, the issue is not "having made a mistake years ago", which we all have done; it is "knowingly being intellectually sloppy/dishonest because you know you can get away with it for a while".

@ Robin:

It's not the fact that he was being sloppy that really bothers me. It is the fact that he was knowingly sloppy because he thought he could get away with it. It wasn't sloppiness due to a miscalculation, it was calculated sloppiness. I'm sure Eliezer does the former, but I've never had the impression of him engaging in the latter.

Makes me think:
Wouldn't it be rather recommendable, if instead of heading straight for an (risky) AGI, we worked on (safe) SIs and then have them solve the problem of Friendly AGI?

Frank: buh? "Safe SI" is just another name for "Friendly AI." It's the same problem.

Nick:
I thought the assumption was that SI is to S to get any ideas about world domination?

Nick, I think "S" is for "Specialized" rather than "Super".

Shane and Joseph, years later Schank blames his earlier self for sloppiness he could get away with. But that does not at all mean the earlier self was consciously choosing to be sloppy. We all suffer a temptation to be sloppy when we can get away with it, and we may well succumb even when we are not consciously aware of succumbing.

Robin, both readings are possible: Schank still approves of his younger self's methodology, or Schank now regrets it and sees his younger self as a sloppy fool. I shan't argue the point, though it seems to me the former reading is more consistent with the text. But Schank would have to move far beyond that initial point to become a careful reasoner about futurism, and the rest of the text provides no reason to assume he has made this effort. Why does he believe what he believes? What does he think he knows and how does he think he knows it? For most people asked to comment on the future of AI 50 years hence, there is no answer to this question that takes the issue apart into premises and evidence and theories and predictions. They are just making stuff up that fits their current mood. This is the strong prior; I see no evidence here which overrides it.

Eliezer, yes of course if you assume you are a careful reasoner about the future and have a strong prior against any particular other being careful, even those at Schank's level of achievement, then yes there is nothing in the text I quoted that will force you to change your mind and conclude he is in fact careful.

Robin, I agree that the charitable interpretation is possible. But when he said "It seemed a safe answer since no one could ever tell me I was wrong", I took him to be saying that whether he could be shown to be wrong or not was a major factor in saying what he did.

In that case, if he cared more about saying something that he couldn't be held responsible for if wrong than about having a justified and likely-to-be-correct position, I'd say that is at least as bad.

This blog as a collective has been dying for this post/commentary for as long as I've been reading it!

If there weren't deep keys to optimization, humans would be unable to achieve a sudden huge advantage over other lifeforms, using brains optimized in the entire absence of direct selective pressures for inventing guns or space shuttles. It would just be a matter of slowly evolving swords, slowly evolving guns, slowly evolving nuclear weapons...

More on this please! I'm convinced that evolutionary psychology has big things to teach us about intelligence, but I haven't thought hard about it as yet.

Caledonian,

Oh, sure, ant colonies are optimization processes too. But there are a few criteria by which we can distinguish the danger of an ant colony from the danger of a human from the danger of an AGI. For example:

(1) How powerful is the optimization process— how tiny is the target it can achieve? A sophisticated spambot might reliably achieve proper English sentences, but I work towards a much smaller target (namely, a coherent conversation) which the spambot couldn't reliably hit.

Not counting the production of individual ants (which is the result of a much larger optimization process of evolution), the ant colony is able to achieve a certain social structure in the colony and to establish the same in a new colony. That's nice, but not really as powerful as it gets when compared to humans painting the Mona Lisa or building rockets.

(2) What are the goals of the process? An automated automobile plant is pretty powerful at hitting a small target (a constructed car of a particular sort, out of raw materials), but we don't worry about it because there's no sense in which the plant is trying to expand, reproduce itself, threaten humans, etc.

(3) Is the operation of the process going to change either of the above? This is, so far, only partially true for some advanced biological intelligences and some rudimentary machine ones (not counting the slow improvements of ant colonies under evolution); but a self-modifying AI has the potential to alter (1) and (2) dramatically in a short period of time.

Can you at least accept that a smarter-than-human AI able to self-modify would exceed anything we've yet seen on properties (1) and (3)? That's why the SIAI hopes to get (2) right, even given (3).

Hang on, the automated manufacturing plant isn't quite what I mean by an optimization process of this sort. The "specialized intelligences" being discussed fit the bill better of something with strong optimizing powers but unambitious goals.

If you want to appreciate the inferential distances here, think of how odd all this would sound without the Einstein sequence. Then think of how odd the Einstein sequence would have sounded without the many-worlds sequence...

The Einstein sequence is a unique identifying number attached to an astronomical observation from the Einstein observatory.

If you mean something different, you should explain.

@Phil Goetz:

"The Einstein sequence is a unique identifying number attached to ... If you mean something different, you should explain."

Eliezer posted a fairly detailed series of discussions recently on how and whether we should treat Einstein as some kind of superior being. You should read them front-to-back, but unfortunately, the links run the other way. Here's the final one, which contains links to its predecessors. I suspect if you follow the links, the earliest ones will indeed point into the sequence on many-worlds that preceded it.

http://www.overcomingbias.com/2008/05/einsteins-super.html

Think of how odd all this would sound without the Einstein sequence. Then think of how odd the Einstein sequence would have sounded without the many-worlds sequence...

“There is not the slightest indication that nuclear energy will ever be obtainable. It would mean that the atom would have to be shattered at will.” — Albert Einstein, 1932

Ten years later, when all the greatest Western minds had been hired to work together in the Manhattan Project, the problem was solved.