The metaphor/myth of general intelligence

I suspect that the closest thing to a "general intelligence module" might be a "skill acquisition module" - e.g. not something that would be generally intelligent by itself, but rather something that could generate more specialized modules that were optimized for some specific domain.

E.g. humans are capable of acquiring a wide variety of very different and specialized skills with sufficient practice and instruction, but we're probably constrained by whether the domain can be easily mapped into one of our native architectures. My hunch is that if you could give a baseline human access to a few more domains that they could natively think in and also enabled them to map new concepts into those domains (and vice versa), they could easily come across as superintelligent by being capable of coming up with modes of thought that were completely unfamiliar to us and applying them to problems that weren't easily handled with normal modes of thought. (On the other hand, they would have to come up with those mappings by themselves, since there was nobody around to give them instructions that was communicated in terms of those domains.)

This is a really nice and useful article. I particularly like the list of problems AI experts assumed would be AI-complete, but turned out not to be.

I'd add that if we are trying to reach the conclusion that "we should be more worried about non-general intelligences than we currently are", then you don't need it to be true that general intelligences are really difficult. It would be enough that "there is a reasonable chance we will encounter a dangerous non-general one before a dangerous general one". I'd be inclined to believe that even without any of the theorising about possibility.

I think one reason for the focus on 'general' in the AI Safety community is that it is a stand in for the observation that we are not worried about path planners or chess programs or self-driving cars etc. One way to say this is that these are specialised systems, not general ones. But you rightly point out that it doesn't follow that we should only be worried about completely general systems.

Most of your post is well reasoned, but I disagree with the opening paragraph:

It's agreed on this list that general intelligences - those that are capable of displaying high cognitive performance across a whole range of domains - are those that we need to be worrying about. This is rational: the most worrying AIs are those with truly general intelligences, and so those should be the focus of our worries and work.

A dangerous intelligence doesn't have to be smart in all domains, it can just be smart in a single dangerous domain. For example, I'd say that a group of uploaded minds running at 1000x speed is an existential risk.

This seems to imply that general intelligences are more powerful, as it basically bakes in diminishing returns - but we haven't included effort yet. Imagine that the following three intelligences require equal effort: (10,10,10), (20,20,5), (100,5,5). Then the specialised intelligence is definitely the one you need to build.

The (100,5,5) AI seems kind of like a Hitler-AI, very good at manipulating people and taking power over human societies, but stupid about what to do once it takes over. We can imagine lots of narrow intelligences that are better at destruction than helping us reach a positive Singularity (or any kind of Singularity). We already know that FAI is harder than AGI, and if such narrow intelligences are easier than AGI, then we're even more screwed.

So let's caveat the proposition above: the most effective and dangerous type of AI might be one with a bare minimum amount of general intelligence, but an overwhelming advantage in one type of narrow intelligence.

I want to point out that a purely narrow intelligence (without even a bare minimum amount of general intelligence, i.e., a Tool-AI), becomes this type of intelligence if you combine it with a human. This is why I don't think Tool-AIs are safe.

So I would summarize my current position as this: General intelligence is possible and what we need in order to reach a positive Singularity. Narrow intelligences that are powerful and dangerous may well be easier to build than general intelligence, so yes, we should be quite concerned about them.

Related paper: How Intelligible is Intelligence?

Abstract: If human-level AI is eventually created, it may have unprecedented positive or negative consequences for humanity. It is therefore worth constructing the best possible forecasts of policy-relevant aspects of AI development trajectories—even though, at this early stage, the unknowns must remain very large.

We propose that one factor that can usefully constrain models of AI development is the “intelligibility of intelligence”—the extent to which efficient algorithms for general intelligence follow from simple general principles, as in physics, as opposed to being necessarily a conglomerate of special case solutions. Specifically, we argue that estimates of the “intelligibility of intelligence” bear on:

• Whether human-level AI will come about through a conceptual breakthrough, rather than through either the gradual addition of hardware, or the gradual accumulation of special-case hacks;

• Whether the invention of human-level AI will, therefore, come without much warning;

• Whether, if AI progress comes through neuroscience, neuroscientific knowledge will enable brain-inspired human-level intelligences (as researchers “see why the brain works”) before it enables whole brain emulation;

• Whether superhuman AIs, once created, will have a large advantage over humans in designing still more powerful AI algorithms;

• Whether AI intelligence growth may therefore be rather sudden past the human level; and

• Whether it may be humanly possible to understand intelligence well enough, and to design powerful AI architectures that are sufficiently transparent, to create demonstrably safe AIs far above the human level.

The intelligibility of intelligence thus provides a means of constraining long-term AI forecasting by suggesting relationships between several unknowns in AI development trajectories. Also, we can improve our estimates of the intelligibility of intelligence, e.g. by examining the evolution of animal intelligence, and the track record of AI research to date.

If there is only specialized intelligence, then what would one call an intelligence that specializes in creating other specialized intelligences? Such an intelligence might be even more dangerous than a general intelligence or some other specialized intelligence if, for instance, it's really good at making lots of different X-maximizers (each of which is more efficient than a general intelligence) and terrible at deciding which Xs it should choose. Humanity might have a chance against a non-generally-intelligent paperclip maximizer, but probably less of a chance against a hoard of different maximizers.

But I'd prefer to look at another assumption: the idea that a general intelligence will really be formidable in multiple domains, and that one of the best ways of accomplishing a goal in a particular domain is to construct a general intelligence and let it specialise.

This assumption explains why evolution hasn't already filled all the economic and ecological niches with species of specialized intelligence, that humans are able to specialize into. If the assumption is false, then there would have to be another explanation for this fact. Are there any proposals on the table?

What if it was very hard to produce an intelligence that was of high performance across many domains?... There are a few strong counters to this - for instance, you could construct good generalists by networking together specialists...

In fact, we already know the minimax optimal algorithm for combining "expert" predictions (here "expert" denotes an online sequence prediction algorithm of any variety); it's the weighted majority algorithm.

I note also KnaveOfAllTrades' recent post about the analogous concept of a "sports quotient".

Brains are like cars. Some are trucks made for heavy hauling, but travel slow. Some are sedans: economical, fuel efficient, but not very exciting. Some are ferraris sleek, fast, and sexy but burn through resources like a mother. I'm sure you can come up with more of your own analogies.

First of all, humans are very far from being general intelligences. But if we picked a random complicated Turing machine from the space of such machines, we'd probably be pretty hopeless at predicting its behaviour. We would probably score very low on the scale of intelligence used to construct the AIXI.

I wonder about this. It sounds plausible, but getting reasonable scores also seems plausible -- perhaps even more plausible to me if you allow a human with a computer. It is probably quite sensitive to permitted thinking time. (I'm assuming that the 'scale of intelligence' you talk about is Legg's AIQ.)

It is the kind of thing we could test empirically, but it's not clear that this would be a good use of resources. How decision-relevant is it for us whether humans are general intelligences?

There are no free lunch theorems that show that no computable intelligences can perform well in all environments. As far as they go, these theorems are uninteresting, as we don't need intelligences that perform well in all environments, just in almost all/most.

They are also, in an important sense, false: the No Free Lunch theorem for statistical learning assumes that any underlying reality is as likely as any other (uniform distribution). Marcus Hutter published a paper years ago showing that when you make a simple Occam's Razor assumption, using the Solomonoff Measure over reality functions instead of a uniform distribution, you do, in fact, achieve a free lunch.

And of course, the Occam's Razor assumption is well-justified by the whole line of thought going from entropy in statistical mechanics through to both information-theoretic entropy and Kolmogorov complexity, viz: a simpler macrostate ("reality function" for classification/concept learning) can "be implemented by", emerge from, many microstates, so Occam's Razor and the Solomonoff Measure work in reductionist ontologies.