Selling Nonapples

I think that the reason that I find Brooks' ideas interesting is because it seems to mirror the way that natural intelligences came about.

Biological evolution seems to amount to nothing more than local systems adapting to survive in and environment, and then aggregating into more complex systems. We know that this strategy has produced intelligence at least once in the history of the universe, and thus is seems to me a productive example to follow in attempting to create artificial intelligence as well.

Now, I don't know what the state of the art is for the emergent AI school of thought is at the moment, but isn't it possible that the challenge isn't solving each of the little problems that feedback loops can help overcome, but rather enfolding the lessons learned by these simple systems into more complex aggregate systems?

That being said, you may be right, it may be easier (at this point) to program AI systems to narrow their search field with information about probability distributions and so forth, but could it not be that this strategy is fundamentally limited in the same way that expert systems are limited? That is, the system is only as "smart" as the knowledge base (or probability distributions) allow it to become, and they fail as "general" AI?

I hope I live to see a world where synchronous computing is considered a quaint artifact of the dawn of computers. cognitive bias has prevented us from seeing the full of extent of what can be done with this computing thing. a limit on feasible computability (limited by our own brain capacity) that has existed for all the millions of years, shaping the way we assume we can solve problems in our world, is suddenly gone. we've made remarkable progress in a short time, I can't wait to see what happens next.

Was the darpa grand challenge winner written using CES or a successor? I see no mention of it in the Darpa paper.

If not, why not? Perhaps neither of these approaches are good in the real world.

I am also guilty of wanting to toss people back to the Turing Tarpit to get to AI, but I don't advocate staying there for long. I just think we have the wrong foundation for resource management and have to redo security and resource allocation at the architectural level. Then rebuild in a more adaptive system from there. I have a few ideas and they do have a fair amount of centralized modeling. But those methods of centralized modeling should be able to be changed in extremis, if what we thought about the world was wrong.

Think about it this way, you advocate sometimes radically rethinking your fundamental ways of conceiving the world, should we not allow our AI systems to be able to sometimes do the same, rather than constrain them with our preconceptions for eternity?

Economists have to face this in spades. So many people say standard econ has failed and the solution is to do the opposite - non-equilibrium instead of equilibrium, non-selfish instead of selfish, non-individual instead of individual, etc. And of course as you point out the problem is that just doesn't say very much.

Tilden is another roboticist who's gotten rich and famous off of unintelligent robots: BEAM robotics

Robin, if you say that's true in economics too, then this is probably a full-blown Standard Iconoclast Failure Mode.

I wonder if the situation in computer programming is just an especially good illustration of it, because the programmer actually does have to reimpose order somehow afterward - you get to see the structure lost, the tarpit, and the effort. Brooks wrote real programs in his expanded design space and even made a buck off it, so we should much more strongly criticize someone who merely advocates "non-equilibrium economics" without saying which kind of disequilibrium.

My understanding is that, while there are still people in the world who speak with reverence of Brooks's subsumption architecture, it's not used much in commercial systems on account of being nearly impossible to program.

I once asked one of the robotics guys at IDSIA about subsumption architecture (he ran the German team that won the robo-soccer world cup a few years back) and his reply was that people like it because it works really well and is the simplist way to program many things. At the time, all of the top teams used it as far as he knew.

(p.s. don't expect follow up replies on this topic from me as I'm current in the middle of nowhere using semi-functional dial-up...)

blinks at Shane

Okay, I'll place my data in a state of expected-instability-pending-further-evidence. This doesn't match what I've heard/found in my own meager investigations. Or maybe it works for a Roomba but not automated vehicles.

I don't get this post. There is no big mystery to asynchronous communication - a process looks for messages whenever it is convenient for it to do so, very much like we check our mail-boxes when it is convenient for us. Although it is not clear to me how asynchronous communication helps in building an AI, I don't see any underspecification here. And if people (including Brooks) have actually used the architecture for building robots, that at least must be clear proof that there is a real architecture here.

Btw, from my understanding, Thrun's team made heavy use of supervised learning - the same paradigm that Eliezer knocked down as being unFriendly in his AI risks paper.

Demanding nonapples is the standard response of voters to the failure of the governing Apple party.

I think subsumption is still popular amongst students and hobbyists.

It does raise an interesting "mini Friendliness" issue... I'm not really comfortable with the idea of subsumption software systems driving cars around on the road. Robots taking over the world may seem silly to most of the public but there are definite decisions to be made soon about what criteria we should use to trust complex software systems that make potentially deadly decisions. So far I think there's a sense that the rules robots should use are completely clear, the situations enumerable -- because of the extreme narrowness of the task domains. As tasks become more open ended, that may not be true for much longer.

@ Venu: Modern AI efforts are so far from human-level competence, that Friendly vs. Unfriendly doesn't really matter yet. Eliezer is concerned about having a Friendly foundation for the coming Singularity, which starts with human-level AIs. A fairly stupid program (compared to humans) that merely drives a car, just doesn't have the power to be a risk in the sense that Eliezer worries about.

@Don: Eliezer says in his AI risks paper , criticising Bill Hibbard, that one cannot use supervised learning to specify the goal system for an AI. And although he doesn't say this in the AI risks paper (contra what I said in my previous comment), I remember him saying somewhere (was it in a mailing list?) that supervised learning as such is not a reliable component to include in a Friendly AI. (I may be wrong in attributing this to him however.) I feel this criticism is misguided as any viable proposal for (Friendly or not) AI will have to be built out of modules which are not smart enough to be Friendly themselves. And supervised learning sure seems like a handy module to have - it clusters highly variable lower level sensory input into more stable higher level objects, and its usefulness has been demonstrated by the heavy use of it by Thrun's team.

"A fairly stupid program (compared to humans) that merely drives a car, just doesn't have the power to be a risk in the sense that Eliezer worries about."

Well, not a significant one, anyway. Perhaps a very, very tiny one. Programs powering robot maids such as are being developed in Japan are a higher risk, as the required forms of intelligence are closer to human forms, and thus probably closer to general intelligence.

The market has no central processing unit - do your arguments against asynchronous parallel decentralized programs work equally well regarding the market? True, the market doesn't always lift its leg when I'd like it to or where I'd like it to, but it does seem to get along in a decephalic fashion.

I think the strength of Brook's method of robotics is that it allows for another option. Some functions of robotics might be much better to be reflexive: encounter object, lift leg. Some functions might be better if they followed the old model: encounter object, take a picture, analyze picture, take sample from object, analyse sample, etc. I forsee more of both in the future. A rocket lands on a distant world and a swarm of small, reflexive robots pour out. They find interesting things for the more 'smart' robots to look at. Or imagine a robot that rolls along mostly on reflex (nice when you have to suddenly pull away from a cliff edge) that can also do analysis of moving targets or disarm land mines or whatever.

Trevor Blake misses the point. Of course there are some good design spots in asynchronous space, it's a huge space, there are bound to be. If you can find them, more power to you.

Venu: And if people (including Brooks) have actually used the architecture for building robots, that at least must be clear proof that there is a real architecture here.

The problem is, almost any AI idea you can think of can be made to work on a few examples, so working on a few examples isn't evidence that there's anything there. The deadly words are "now we just have to scale it up".

derekz: I think subsumption is still popular amongst students and hobbyists.

That is, popular amongst people repeating for themselves the toy examples that other people have used it for already.

I'm surprised nobody put this problem in terms of optimization and "steering the future" (including Eliezer, though I suppose he might have tried to make a different point in his post).

As I see it, robots are a special case of machines intended to steer things in their immediate vicinity towards some preferred future. (The special case is that their acting parts and steering parts are housed in the same object, which is not terribly important, except that the subsumption architecture implies it.)

"Smart" robots have a component analogue to a cortex, which gathers sensory info, models the world, makes plans and directs the acting parts to do things. (This is true even if this component is just 50 lines of code compiled by a smart compiler.) The subsumption-based robots just contain many parts that are hard-wired with "reflexes" for every little connection between them, in such a way that "it just happens" that the ensemble acts intelligently, i.e. steers the future towards some intended subspace.

The fallacy I think is that the "just happens" part is not true. Some process has to generate "optimization rules"; even the very simple reflexes between the components have to be correct (meaning that in the real world steer the future towards something). Subsumption architecture fans will look at something like ants, notice that each element has very simple rules but that the colony works extremely well, and will say "Hey, those are simple rules, I can understand them, I bet I could create another set of simple rules to do what I want. I don't need to build a complex brain."

The problem is that "set of simple rules" is not the same as "simple set of rules": in the ants' case, millions of years of evolution led to a very complicated set of very simple rules that work well. A subsumtion-oriented programmer can only build a simple set of simple rules; a complex one won't work.

To give an example, take the "move leg higher" case:

*) In a few millions of years of evolution, creatures whose legs went higher up when hitting obstacles (say, because of random interactions between unrelated cells) got to eat more and be eaten less; so genetic programs for legged creature tend to contain rules like that.

*) A subsumtion-oriented programmer may notice that a leg that hits something should lift higher (either because (a) he thought about it or because (2) he noticed that's what ants or caterpillars do).

*) A general AI researcher might think about a brain smart enough to decide it should lift a leg over obstacles it wants to cross.

Of course the first programmer will notice that his solution to the "going over obstacles" problem is simpler than building a brain, and it would seem more efficient.

But it's not more efficient, because (in case a) his very complex, general Natural Intelligence brain thought about it, or (in case b) millions of years of evolution caused this particular adaptation (and his general NI noticed it was useful). There's also the problem that there are thousands of other things even a simple organism does, and thus thousands (or more) reflexes to add. Either he'll try to program them (case a). This is steering the future via a Rube Goldberg machine; it can work if you put enormous resources into it, but most likely it will blow in your face (or drop an anvil on you).

Or he'll turn to evolving algorithms (simulating case b). That would probably work (it did until now), but he probably doesn't have enough resources (natural evolution is a very inefficient optimizer). And even when he does, he won't understand how it works.

Or he might notice that even nature found central-processing useful, several times (pretty much everything above worms has a brain of some kind, and brains of arthropods, cephalopods and vertebrates evolved separately). So he'll turn back to centralized programming: hard as it is, it's a more efficient investment than the alternative. Note that this doesn't mean that you need fully-general AI to drive a car. You need all tools you can get your hands on. Saying "reflexes are enough" is like saying "I don't need no tools, rocks are good enough".

The idea is similar to swarm logic or artificial life where the concept there is to program a single agent who behaves naturally then put a bunch of them together and you get natural behavior. The idea of emergent behavior from smaller parts is of great interest in the defense industry.

Stanley used a Markov model.

[ad hominems / noncontent deleted]

vendor a: selling apples on wood carts isn't making as much money as I hoped.

vendor b: maybe we should sell nonapples on nonwood carts.

a: that's just silly. Which convinces me that we should continue selling non-nonapples on non-nonwood.

... ie, the opposite of stupidity is rarely intelligence, but the opposite of the opposite of stupidity never is.

Human intelligence arose out of a neurological Turing tarpit. It is reasonable to imagine that designing intelligence in less time will take tricks - ones which Mother Nature didn't use - to get out of the tarpit. It is not reasonable to imagine that there is one magic trick which gets us all the way there. So saying "my trick is better than your trick" is interesting, but don't mean that the final answer won't need both or neither.

In fact, as soon as it comes to actual wagon design, "nonwood" can be a useful step. If you are building wagons, and there are 5 wagon stores in town, one of which is the "nonwood" store and has the generally lowest-quality - but still functioning - wagons, and you only have time to visit 2 stores for inspiration, which would you visit? Would you be happy to have the nonwood designs all in one store, or would you rather use some classification system (number of spokes) which split them across several stores? (That is, until metal is the new wood.) Any actual "nonwood" wagon is in fact made of something.

Classic... delete the context and throw wild accusations. Good to know your so mature.

Eliezer, you'd have done better to ignore ReadABook's trash. Hir ignorance of your arguments and expertise was obvious.

I don't know anything about the specific AI architectures in this post, but I'll defend non-apples. If one area of design-space is very high in search ordering but very low in preference ordering (ie a very attractive looking but in fact useless idea), then telling people to avoid it is helpful beyond the seemingly low level of optimization power it gives.

A metaphor: religious beliefs constitute a very small and specific area of beliefspace, but that area originally looks very attractive. You could spend your whole life searching within that area and never getting anywhere. Saying "be atheist!" provides an trivial amount of optimization power. But that doesn't mean it's of trivial importance in the search for correct beliefs. Another metaphor: if you're stuck in a ditch, the majority of the effort it takes to journey a mile will be the ten vertical meters it takes to climb to the top.

Saying "not X" doesn't make people go for all non-X equally. It makes them apply their intelligence to the problem again, ignoring the trap at X that they would otherwise fall into. If the problem is pretty easy once you stop trying to sell apples, then "sell non-apples" might provide most of the effective optimization power you need.

Yvain: It might be a good idea to start looking for marketable non-apples, but you can't sell generalized non-apples right away. The situation in question is the opposite of what you highlighted: non-apples can act as semantic stopsigns, a Wise and mysterious answer rather than a direction for specific research. Traps of the former domain aren't balanced out by bewilderment of its negation; it might be easier, or it might be much worse.

Yvain, that's a fair point. And to the extent that you've just got specific bad beliefs infesting your head, "Be atheist!" is operationalizable in a way that "sell nonapples!" is not.

So are you claiming that Brooks' whole plan was, on a whim, to just do the opposite of what the neats were doing up till then? I thought his inspiration for the subsumption architecture was nature, the embodied intelligence of evolved biological organisms, the only existence proof of higher intelligence we have so far. To me it seems like the neats are the ones searching a larger design space, not the other way around. The scruffies have identified some kind of solution to creating intelligent machines in nature and are targeting a constrained design space inspired by this--the neats on the other hand are trying to create intelligence seemingly out of the platonic world of forms.

"When you say to build a wagon using "wood", you're giving much more concrete advice then when you say "not wood". There are different kinds of wood, of course - but even so, when you say "wood", you've narrowed down the range of possible building materials a whole lot more then when you say "not wood"."

"When you say to build a wagon using "wood", you're giving much more concrete advice THAN when you say "not wood". There are different kinds of wood, of course - but even so, when you say "wood", you've narrowed down the range of possible building materials a whole lot more THAN when you say "not wood"."

Returning to the post, I suspect that there is a lack of relevant mathematical theorems in this area. What is needed for example is a theorem which says something like:

"In a sufficiently general environment E the AI planning system must incorporate probabilistic reasoning to satisfy its goals."

Likewise a theorem characterising which environments require "asynchronous" planning architectures is probably a holy grail in this field.

I realize I am way late for this party, but I would like to make a specific theoretical point about synchronous vs. asynchronous communication. It turns out that, given some components or modules or threads or what-have-you communicating via sending messages to one another, synchronous communication is actually more general than asynchronous in the following technical sense. Once can always use synchronous communication to implement asynchronous communication by throwing another module in the middle to act as a mailbox. On the other hand, given only asynchronous communication, it is impossible to implement synchronous communication. Thus, I dearly hope, unlike nazgulnarsil3, that I never see the day where we forget that synchronous communication is strictly more powerful than asynchronous. Moreover, this is not analogous to the wood/nonwood situation where they are entirely different--in fact, in the synchronous/asynchronous situation, one entirely subsumes the other.

This says nothing about that fact that it requires more computing power to encode asynchrony as synchrony (although only linearly more in the number of communication channels). Moreover, it certainly says nothing about central vs. distributed models, which is an orthogonal issue to synchronous vs. asynchronous communication.

I just came across this image and thought it was hilarious in this context: nonaspirin

I was with you until that last part about economics. Behavioral economists ARE working on actual detailed models of human economic decisions that use assumptions other than economic rationality. They use things like hyperbolic discounting (as opposed to just "nonexponential" discounting) and distance-mediated altruism (rather than just "nonselfishness"). So they're not making nonwagons out of nonwood; they're trying to make cars out of steel. They haven't finished yet; but it's really not fair to expect a new paradigm to achieve in 10 years what the old paradigm did in 200.

Maybe this was meant to go against popular accounts of economics... but frankly most popular accounts of economics are so bad they don't even rise to this level. They are things like "the Mexicans are stealing our jobs" "the bourgeoisie oppress the prolerariat" and "taxation is slavery", which are just so blatantly wrong they can't even be formulated in terms of serious economic models.

Apologies for commenting almost a decade after most of the comments here, but this is the exact same reason why "using nonlinear models is harder but more realistic".

The way we were taught math led us to believe that linear models form this space of tractable math, and nonlinear models form this somewhat larger space of mostly intractable math. This is mostly right, but the space of nonlinear models is almost infinitely larger than that of linear models. And that is the reason linear models are mathematically tractable : they form such a small space of possible models. Of course nonlinear models don't have general formulae that always work : they're just defined as what is NOT linear. In other words, linear models are severely restricted in the form they can have. When we define another subset of models suitable to the specific thing being modelled, then we will just as easily be able to come up with a set of explicit symbolic formulae. Then it will be just as "tractable" as linear models, even though it's nonlinear : simply because it has different special properties belonging to its own class of models obeying something just like the law of linearity