Let's reimplement EURISKO!

Exactly.

I mean, if you kept on making it marginally cooler and cooler, it'd go to "oh, shit" one day after a sequence of "ooh, cools" and I don't know how long that sequence is.

This means we should feel pretty safe, since AI does not appear to be making even incremental progress.

Really, it's hard for anyone who is well-versed in the "state of the art" of AI to feel any kind of alarm about the possibility of an imminent FOOM. Take a look at this paper. Skim through the intro, note the long and complicated reinforcement learning algorithm, and check out the empirical results section. The test domain involves a monkey in a 5x5 playroom. There are some fun little complications, like a light switch and a bell. Note that these guys are top-class (Andrew Barto basically invented RL), and the paper was published at one of the top-tier machine learning conferences (NIPS), in 2005.

Call me a denier, but I just don't think the monkey is going to bust out of his playroom and take over the world. At least, not anytime soon.

Seeing the recent thread necromancy, it looks like this is a much more important question than I realized at the time, since it bears on AI-related existential risk.

The question, to summarize, was, "How exactly do you keep a good AGI prospect from being developed into unFriendly AGI, other than Security by Obscurity?"

My answer is that SbO (Security by Obscurity, not Antimony(II) Oxide, which doesn't even exist) is not a solution here for the same reason it's criticized everywhere else (which I assume is that it increases the probability of a rogue outsmarting mainstream researchers). Better to let the good guys be as well informed as the bad guys so they can deploy countermeasures (their own AGI) when the bad guys develop theirs.

But then, I haven't researched this 24/7 for the last several years, so this may be too trite a dismissal.

Not exactly, Thom. Roughly, for FAI you need precise self-modification. For precise self-modification, you need a precise theory of the intelligence doing the self-modification. To get to FAI you have to walk the road that leads to precise theories of intelligence - something like our present-day probability theory and decision theory, but more powerful and general and addressing issues these present theories don't.

Eurisko is the road of self-modification done in an imprecise way, ad-hoc, throwing together whatever works until it gets smart enough to FOOM. This is a path that leads to shattered planets, if it were followed far enough. No, I'm not saying that Eurisko in particular is far enough, I'm saying that it's a first step along that path, not the FAI path.

The AI can be adapted for other, less restricted, domains

That the ideas from a safe AI can be used to build an unsafe AI is a general argument against working on (or even talking about) any kind of AI whatsoever.

The AI adds code that will evolve into another AI into it's output

The output is to contain only proofs of theorems. Specifically, a proof (or refutation) of the theorem in the input. The state of the system is to be reset after each run so as to not accumulate information.

The AI could self-modify incorrectly and result in unfriendly AI

Any correct or incorrect self-modification is still restricted to the math domain, and so cannot result in an unsafe AI.

bug in the environment itself

Guarding against software bugs is easy in this case. You design an abstract virtual machine environment for the AI, then design the software that implements this environment, then formally prove that the software is correct. Guarding against errors caused by cosmic rays is also easy. You estimate the probability of such an error, and then add redundancy until the probability is so low that it won't happen until the end of the universe.

Look up how difficult it is to sandbox untrusted code

Sandboxing untrusted code is easy. The difficult thing is sandboxing it while making it think that it runs normally. This is irrelevant here.

I don't believe that a system can work only in formal proofs

It is quite probable that a "pure math Oracle" system cannot work. The point was, it can be made safe to try.

If I'm right, the effect of widespread propagation of such memes will be to snuff out what chance of survival and success humanity might have had. Unlike UFAI which is pure science fiction, the strangling of progress is something that occurs - has occurred before - in real life.

What would you consider to be the minimum estimate of the probability that I'm right, necessary to "reasonably" motivate concern or action?

Eliezer,

I am rather surprised that you accept all of the claimed achievements of Eurisko and even regard it as "dangerous", despite the fact that no one save the author has ever seen even a fragment of its source code. I firmly believe that we are dealing with a "mechanical Turk."

I am also curious why you believe that meaningful research on Friendly AI is at all possible without prior exposure to a working AGI. To me it seems a bit like trying to invent the ground fault interrupter before having discovered electricity.

Aside from that: If I had been following your writings more carefully, I might already have learned the answer to this, but: just why do you prioritize formalizing Friendly AI over achieving AI in the fist place? You seem to side with humanity over a hypothetical Paperclip Optimizer. Why is that? It seems to me that unaugmented human intelligence is itself an "unfriendly (non-A)I", quite efficient at laying waste to whatever it touches.

There is every reason to believe that if an AGI does not appear before the demise of cheap petroleum, our species is doomed to "go out with a whimper." I for one prefer the "bang" as a matter of principle.

I would gladly accept taking a chance at conversion to paperclips (or some similarly perverse fate at the hands of an unfriendly AGI) when the alternative appears to be the artificial squelching of the human urge to discover and invent, with the inevitable harvest of stagnation and eventually oblivion.

I accept Paperclip Optimization (and other AGI failure modes) as an honorable death, far superior to being eaten away by old age or being killed by fellow humans in a war over dwindling resources. I want to live in interesting times. Bring on the AGI. It seems to me that if any intelligence, regardless of its origin, is capable of wrenching the universe out of our control, it deserves it.

Why is the continued hegemony of Neolithic flesh-bags so precious to you?

I'd like to echo cousin_it's thanks, I downloaded them as well.

I haven't gotten to read much yet, but I've also run into the problem he's mentioned with theoretical computer science papers being too vague to write code, let alone include it. (Marcus Hutter and Juergen Schmidhuber, I'm looking in your general direction here.)

Richard, thanks a lot! Downloaded all three articles. (No, ScienceDirect doesn't let me download them - the link says "Purchase PDF".)

First surface impression: none of the three papers are as specific as I'd like. After a skim I still have no idea how EURISKO's main loop would look in pseudocode. Will try reading closer.

That's a funny parallel to my own situation. Most hits on the first few pages of Google for "Thom Blake" are me, but the runner-up is a historian from Australia.

I think "vocabulary" in this sense refers to the vocabulary of the bits doing the actual processing. Humans don't have access to the "vocabulary" of their fusiform gyruses, only the result of its computations.

Lenat was also careful to note that AM's success, such as it was, was very much due to the fact that LISP's "vocabulary" started with a strong relation to mathematics. EURISKO didn't show anything like reasonable performance until he realized that the vocabulary it was manipulating needed to be "close" to the modeled domain, in the sense that interesting (to Lenat) statements about the domain needed to be short, and therefore easy for EURISKO to come across.

Yeah, that was basically what I meant. My hypothesis was that if you gave AM to someone with good mathematical aptitude but little prior knowledge, they would discover a lot more interesting mathematical statements than they would have without AM's help, by analogy to Lenat discovering more interesting logical consequences of the wargaming rules with EURISKO's help than any of the experienced players discovered themselves.

This is a very serious point and has been worrying me for some time. This problem connects to continuity of consciousness and reference classes.

It's useful evidence that EURISKO was doing something. There were some extremely dedicated and obsessive people involved in Traveller, back then. The idea that someone unused to starship combat design of that type could come and develop fleets that won decisively two years in a row seems very unlikely.

It might be that EURISKO acted merely as a generic simulator of strategy and design, and Lenat did all the evaluating, and no one else in the contest had access to simulations of similar utility, which would negate much of the interest in EURISKO, I think.

If you had a program that might or might not be on a track to self-improve and initiate an Intelligence explosion you'd better be sure enough that it would remain friendly to, at the very least, give it a robot body, a scalpel, and stand with your throat exposed before it.

Surrounding it with a sandboxed environment couldn't be guaranteed to add any meaningful amount of security. Maybe the few bits of information you provide through your communications channel would be enough for this particular agent to reverse-engineer your psychology and find that correct combination to unlock you, maybe not. Maybe the extra layer(s) between the agent and the physical world would be enough to delay it slightly or stall it completely, maybe not. The point is you shouldn't rely on it.

I think my other reply applies here too, if you read "communications channel" as all the information that might be inferred from the universe the AI finds itself in. Either the AI is not smart enough to be a worry without any sandboxing at all, or you have enough to worry about that you should not be relying on the sandbox to protect you.

Your point about our own simulation (if it is one) lacking a simple communications channel actually works against you - In our universe the simulation hypothesis has been proposed, despite the fact that we have only human intelligence to work with.