...Recursion, Magic

Do you have any evidence that insight is applicable to understanding and creating intelligences? Because without that recursion isn't powerful and magic doesn't start to get off the ground.

You really think an office worker with modern computer tools is only 10% more productive than one with 1950-era non-computer tools? Even at the task of creating better computer tools?

Many important innovations can be thought of as changing the range of things that can be changed, relative to an inheritance that up to that point was not usefully open to focused or conscious development. And each new item added to the list of things we can usefully change increases the possibilities for growing everything else. (While this potentially allows for an increase in the growth rate, rate changes have actually been very rare.) Why aren't all these changes "recursive"? Why reserve that name only for changes to our mental architecture?

Presumably, if office workers were all obliged to concentrate on creating better computer tools - as is the case in an analogous nuclear pile, the neutrons don't really have a choice - then a 10% improvement in productivity would be sufficient.

But an economy is different, surely! An office worker may be potentially better at creating better computer tools. But the equilibrium rate of creating better computer tools is perhaps not sufficiently high?

You speculate about why Eurisko slowed to a halt and then complain that Lenat has wasted his life with CYC, but you ignore that Lenat has his own theory which he gives as the reason he's been pursuing CYC. You should at least explain why you think his theory wrong; I find his theory quite plausible.

"You should at least explain why you think his theory wrong"

Please don't ask that... I've heard the parables too many times to count, about how you can't build an AI by putting in individual pieces of knowledge, you need something that generates these pieces of knowledge itself, and so on.

CYC has become a cliche absurdity...

If CYC can be largely brute-forced with enough info and researcher hours, then shouldn't we expect a million sped-up instances of Doug Lenat to be able to reach hard-coded AI to input enough info rather quickly?

Can someone give a brief definition of FOOM or the link that explains it?

@Dan

FOOM (actually, it should be FWOOM!) is onomatopoeia for the sound of flash ignition - imagine filling your oven with gas and then tossing in a match - FWOOM! Also in video games it's often the sound your head makes when it explodes.

I believe it's Robin's humor to designate the "feeling" of the hard takeoff, and humanity's social reaction to such major sudden change.

I have been an office worker using easily documented approximate solutions generated via algorythms to do what my grandparents generation would have done with provably correct logical solutions to the same problems. They would have taken less time in some respects and more in others. On net, I'd guess that we weren't even 10% more productive. We generated many so-called "solutions" in the time they would have taken to generate one solution, but their solution would have been the correct solution while our procedure would be to then choose one of our many solutions for political reasons. We didn't take less time per project. We had more secretarial staff. We accomplished the same sort of work, and we did a worse job. Maybe 1/3 as productive over all?

Obviously, the tools we used could have been used to increase productivity, but could have != did. This phenomenon, as well as very rough measures, may explain the supposed uniformity of growth rates. An AGI with shared goal content and closed loops of self-improvement would not have the same difficulties as an economy in this respect and might plausibly be expected to show significant growth rate increases from insights on the level of the mouse.

Perhaps, in analogy with Fermi's pile, there is a certain critical mass of intelligence that is necessary for an AI to go FOOM. Can we figure out how much intelligence is needed? Is it reasonable to assume that it is more than the effective intelligence of all of the AI researchers working in AI? Or more conservatively, the intelligence of one AI researcher?

"You speculate about why Eurisko slowed to a halt and then complain that Lenat has wasted his life with CYC, but you ignore that Lenat has his own theory which he gives as the reason he's been pursuing CYC. You should at least explain why you think his theory wrong; I find his theory quite plausible."

• Around 1990, Lenat predicted that Cyc would go FOOM by 2000. In 1999, he told me he expected it to go FOOM within a couple of years. Where's the FOOM?
• Cyc has no cognitive architecture. It's a database. You can ask it questions. It has templates for answering specific types of questions. It has (last I checked, about 10 years ago) no notion of goals, actions, plans, learning, or its own agenthood.

I think those prophets of circa 1900 would still get some bounds correct...

If someone points out to you that your pet theory of the universe is in disagreement with Maxwell's equations — then so much the worse for Maxwell's equations. If it is found to be contradicted by observation — well, these experimentalists do bungle things sometimes. But if your theory is found to be against the second law of thermodynamics I can give you no hope; there is nothing for it but to collapse in deepest humiliation.

- Sir Arthur Stanley Eddington, The Nature of the Physical World (1915), chapter 4

You speculate about why Eurisko slowed to a halt and then complain that Lenat has wasted his life with CYC, but you ignore that Lenat has his own theory which he gives as the reason he's been pursuing CYC. You should at least explain why you think his theory wrong; I find his theory quite plausible.

Artificial Addition, The Nature of Logic, Truly Part of You, Words as Mental Paintbrush Handles, Detached Lever Fallacy...

You really think an office worker with modern computer tools is only 10% more productive than one with 1950-era non-computer tools? Even at the task of creating better computer tools?

I'd started to read Engelbart's vast proposal-paper, and he was talking about computers as a tool of intelligence enhancement. It's this that I had in mind when, trying to be generous, I said "10%". Obviously there are various object-level problems at which someone with a computer is a lot more productive, like doing complicated integrals with no analytic solution.

But what concerns us is the degree of reinvestable improvement, the sort of improvement that will go into better tools that can be used to make still better tools. Office work isn't a candidate for this.

And yes, we use programming languages to write better programming languages - but there are some people out there who still swear by Emacs; would the state of computer science be so terribly far behind where it is now, after who knows how many cycles of reinvestment, if the mouse had still not been invented?

I don't know, but to the extent such an effect existed, I would expect it to be more due to less popular uptake leading to less investment - and not a whole lot due to losing out on the compound interest from a mouse making you, allegedly, 10% smarter, including 10% smarter at the kind of computer science that helps you do further computer science.

Robin Hanson: You really think an office worker with modern computer tools is only 10% more productive than one with 1950-era non-computer tools? Even at the task of creating better computer tools?

Not in the general case. However, I've known people who use these modern computer tools to take ages typesetting a simple document, playing with fonts, colours and styles, and have managed to get very little work done. I've even known people who've managed to get negative work done due to their incompetence in using computers.

Eliezer Yudkowsky: EURISKO may still be the most sophisticated self-improving AI ever built

Why is that, do you think? Eurisko was written a quarter-century ago, and is not exactly obscure. Have people not tried to build a better Eurisko, or have they tried and failed?

you just need to choose the right invariant which is orthogonal to intelligence and then optimise intelligence

yes, we use programming languages to write better programming languages - but there are some people out there who still swear by Emacs [...]

It's not just computer languages. Computers help build better computers, editors help build better editors - and so on.

Plus there are synergetic loops: computers help build better editors, and editors help build better computers.

Look as the whole system and you see that the whole man-machine symbiosis builds the next generation of man-machine symbiosis.

The result is the cumulative snowballing of technological progress: technology making better technology. That has actually been going on for billions of years, if you include "natural technologies" - such as photosyntesis, cellulose, sexual reproduction - and so on.

The result is what I refer to as the ongoing "technology explosion". From such a perspective, superintelligent machines are the latest move in an ancient dance.

All "insights" eventually bottom out in the same way that Eurisko bottomed out; the notion of ever-increasing gain by applying some rule or metarule is a fantasy. You make the same sort of mistake about "insight" as do people like Roger Penrose, who believes that humans can "see" things that no computer could, except that you think that a computer can too, whereas in reality neither humans nor computers have access to any such magical "insight" sauce.

The future has a reputation for accomplishing feats which the past thought impossible.

Yes, but our ability to predict which seemingly impossible feats will actually be accomplished is quite poor, so this fact is neither here nor there, but it is appealed to by crackpots everywhere as an ad hoc defense of their own claims.

editors help build better editors

But not by much, and not much better editors. It takes something else -- better concepts -- to build significantly better editors, and one only needs a moderately good editor in the process. And regardless of things like Eurisko, no one has the faintest idea how to automate having significantly better concepts.

So if you run an optimizing compiler on its own source code, and then use the product to do the same again, it should produce the same output on both occasions - at most, the first-order product will run faster than the original compiler.

Now if you are one of those annoying nitpicky types, like me, you will notice a flaw in this logic:

As one of those annoying nitpicky types I think it is perhaps interesting to note that the current highest level of optimization available with the Microsoft C++ compilers is PGO or Profile Guided Optimization. The basic idea is to collect data from actual runs of the program and feed this back into the optimizer to guide its optimization decisions. Adaptive Optimization is the same basic idea applied to optimize code while it is running based on live performance data.

As you might predict, the speedups achievable by such techniques are worthwhile but modest and not FOOM-worthy.

FYI : the link to the "Traveller wargame" (http://www.aliciapatterson.org/APF0704/Johnson/Johnson.html) is broken.

Unless, y'know, you're counting becoming world champion at Traveller without ever previously playing a human, as some sort of accomplishment.

It certainly is interesting, especially given the weirdly extreme or suicidal tactics it used. I recently into this quote on Slate and was struck by the similarity:

40K may not be a true simulation of armed conflict, but it’s part of a centuries-long tradition of war games. After World War II, U.S. Navy Adm. Chester W. Nimitz credited gaming for helping the Allies prepare. “The war with Japan had been re-enacted in the game rooms here by so many people and in so many different ways,” he said, “that nothing that happened during the war was a surprise—absolutely nothing except the kamikaze tactics towards the end of the war; we had not visualized those.”

I find this difficult to follow. Is there a concrete mathematical definition of 'recursion' in this sense available anywhere?