What Evidence Is AlphaGo Zero Re AGI Complexity?

I appreciate your posting this here, and I do agree that any information from AlphaGo Zero is limited in our ability to apply it to forecasting things like AGI.

That said, this whole article is very defensive, coming up with ways in which the evidence might not apply, not coming up with ways in which it isn't evidence.

I don't think Eliezer's article was a knock-down argument, and I don't think anyone including him believes that. But I do think the situation is some weak evidence in favor for his position over yours.

I also think it's stronger evidence than you seem to think according to the framework you lay down here!

For example, a previous feature of AI for playing games like Chess or Go was to capture information about the structure of the game via some complex combination. However in AlphaGo Zero, very little specific information about Go is required. The change in architecture actually subsumes some amount of the combination of tools needed.

Again I don't think this is a knockdown argument or very strong or compelling evidence--but it looks as though you are treating it as essentially zero evidence which seems unjustified to me.

I feel like this and many other arguments for AI-skepticism are implicitly assuming AGI that is amazingly dumb and then proving that there is no need to worry about this dumb superintelligence.

Remember the old "AI will never beat humans at every task because there isn't one architecture that is optimal at every task. An AI optimised to play chess won't be great at trading stocks (or whatever) and vice versa"? Well, I'm capable of running a different program on my computer depending on the task at hand. If your AGI can't do the same as a random idiot with a PC, it's not really AGI.

I am emphatically not saying that Robin Hanson has ever made this particular blunder but I think he's making a more subtle one in the same vein.

Sure, if you think of AGI as a collection of image recognisers and go engines etc. then there is no ironclad argument for FOOM. But the moment (and probably sooner) that it becomes capable of actual general problem solving on par with it's creators (i.e. actual AGI) and turns its powers to recursive self-improvement - how can that result in anything but FOOM? Doesn't matter if further improvements require more complexity or less complexity or a different kind of complexity or whatever. If human researchers can do it then AGI can do it faster and better because it scales better, doesn't sleep, doesn't eat and doesn't waste time arguing with people on facebook.

This must have been said a million times already. Is this not obvious? What am I missing?

Robin, or anyone who agrees with Robin:

What evidence can you imagine would convince you that AGI would go FOOM?

Disagreements here are largely going to revolve around how this observation and similar ones are interpreted. This kind of evidence must push us in some direction. We all agree that what we saw was surprising - a difficult task was solved by a system with no prior knowledge or specific information to this task baked in. Surprise implies a model update. The question seems to be which model.

The debate referenced above is about the likelihood of AGI "FOOM". The Hansonian position seems to be that a FOOM is unlikely because obtaining generality across many different domains at once is unlikely. Is AlphaGo evidence for or against this position?

There is definitely room for more than one interpretation. On the one hand, AG0 did not require any human games to learn from. It was trained via a variety of methods that were not specific to Go itself. It used neural net components that were proven to work well on very different domains such as Atari. This is evidence that the components and techniques used to create a narrow AI system can also be used on a wide variety of domains.

On the other hand, it's not clear whether the "AI system" itself should be considered as only the trained neural network, or the entire apparatus including using MCTS to simulate self play in order to generate supervised training data. The network by itself plays one game, the apparatus learns to play games. You could choose to see this observation instead as "humans tinkered for years to create a narrow system that only plays Go." AG0, once trained, cannot go train on an entirely different game and then know how to play both at a superhuman level (as far as I know, anyway. There are some results that suggest it's possible for some models to learn different tasks in sequence without forgetting). So one hypothesis to update in favor of is "there is a tool that allows a system to learn to do one task, this tool can be applied to many different tasks, but only one task at a time."

But would future, more general, AI systems do something similar to human researchers, in order to train narrow AI subcomponents used for more specific tasks? Could another AI do the "tinkering" that humans do, trained via similar methods? Perhaps not with AG0's training method specifically. But maybe there are other similar, general training algorithms that could do it, and we want to know if this one method that proves to be more general than expected suggests the existence of even more general methods.

It's hard to see how this observation can be evidence against this, but there are also no good ways to determine how strongly it is for it, either. So I don't see how this can favor Hanson's position at all, but how much it favors EY's is open to debate.

Promoted to Featured for engaging substantively with a previous Featured post, about an important topic (Foom).

For those of us without a lot of background on AlphaGo Zero, does anyone care to summarize how general the tool used to create it is likely to be?

Honest question - is there some specific technical sense in which you are using "complex"? Colloquially, complex just means a thing consisting of many parts. Any neural network is "a thing consisting of many parts", and I can generally add arbitrarily many "parts" by changing the number of layers or neurons-per-layer or whatever at initialization time.

I don't think this is what you mean, though. You mean something like architectural complexity, though I think the word "architectural" is a weasel word here that lets you avoid explaining what exactly is missing from, e.g., AlphaGo Zero. I think by "complex" you mean something like "a thing consisting of many distinct sub-things, with the critical functional details spread across many levels of components and their combinations". Or perhaps "the design instructions for the thing cannot be efficiently compressed". This is the sense in which the brain is more "complex" than the kidneys.

(Although, the design instructions for the brain can be efficiently compressed, and indeed brains are made from surprisingly simple instructions. A developed, adult brain can't be efficiently compressed, true, but that's not a fair comparison. A blank-slate initialized AlphaGo network, and that same network after training on ten million games of Go, are not the same artifact.)

Other words aside from "complex" and "architecture" that I think you could afford to taboo for the sake of clarity are "simple" and "general". Is the idea of a neural network "simple"? Is a convnet "general"? Is MCTS a "simple, general" algorithm or a "complex, narrow" one? These are bad questions because all those words must be defined relative to some standard that is not provided. What problem are you trying to solve, what class of techniques are you regarding as relevant for comparison? A convnet is definitely "complex and narrow" compared to linear regression, as a mathematical technique. AlphaGo Zero is highly "complex and narrow" relative to a vanilla convnet.

If your answer to "how complex and how specific a technique do you think we're missing?" is always "more complex and more specific than whatever Deepmind just implemented", then we should definitely stop using those words.

I would take this more seriously if I'd seen any evidence that Robin's position had updated at all since the first Yudkowsky-Hanson FOOM debate. Which, despite seeing many discussions, between them and otherwise, I have not.

As it is, updating on this post would be double-counting.

Consider the example of whether a big terror attack indicates that there has been an increase in the average rate or harm of terror attacks. You could easily say "You can't possibly claim that big terror attack yesterday is no evidence; and if it is evidence it is surely in the direction of the ave rate/harm having increased." Technically correct, but then every other day without such a big attack is also evidence for a slow decrease in the rate/harm of attacks. Even if the rate/harm didn't change, every once in a while you should expect a big attack. This in the sense in which I'd say that finding one more big tool isn't much evidence that big tools will matter more in the future. Sure the day when you find such a tool is itself weak evidence in that direction. But the whole recent history of that day and all the days before it may be an evidential wash.

I think it's reasonable to question how relevant this achievement is to our estimates of the speed of GAI, and I appreciate the specification of what evidence Robin thinks would be useful for estimating this speed - I just don't see its relevance.
"The best evidence regarding the need for complexity in strong broad systems is the actual complexity observed in such systems."
There is an underlying question of how complex a system needs to be to exhibit GAI; a single example of evolved intelligence sets an upper limit, but is very weak evidence about the minimum. So my question for Robin is; what evidence can we look for about such a minimum?

(My suggestion: The success of "Narrow AI" at tasks like translation and writing seem like clear but weak evidence that many products of a moderately capable mind is achievable by relatively low complexity AI with large datasets. Robin: do you agree? If not, what short term successes or failures do you predict on the basis of your assumption that NAI successes aren't leading to better GAI?)

Arguing about the mostly likely outcome is missing the point: when the stakes are as high as survival of the human race, even a 1% probability of an adverse outcome is very worrisome. So my question to Robin Hanson is this: are you 99% certain that the FOOM scenario is wrong?

Let's assume you are correct, or at least "less wrong". Then there will eventually exsist a set of simple tools {T} that can be combined to solve many AGI-level problems. A given problem P will require a particular "complex" solver S(P). But then we achieve AGI by creating a system S(S(P)) that can produce S(P) given P and the rules for use of the elements in {T}. It seems to me that S(S(P)) may be amenable to an AlphaGo Zero approach.

It seems to me that systems of the AlphaGo Zero type should be able to derive the rules for subatomic particles from the rules of QED, the rules of nuclear physics from the rules of subatomic particles, and the rules of physical chemistry from the rules of nuclear physics. I'm not sure why anyone would do this, except that if if it fails, it might tell us something about missing rules, or possible find new rules. For instance, the first step might identify ro exclude some new subatomic particles, or identify or definitively exclude new particles at the quark level. The leap from physcal chemistry to the whole of biochemistry is more of a challenge: I suspect the some human guidance to select intermediate goals would simplify things (protein folding has been mentioned.)