Magnus Carlsen is closer in ELO to Stockfish than median human.

Chess is a bad example.

Here's a useful rule of thumb: Every 100 Elo is supposed to give you a 30% edge. Or play around with this: https://wismuth.com/elo/calculator.html

This means that if a 1400 plays a 1500, the 1500 should win about 30% more than the 1400. Totally normal thing that happens all the time.

It also means that if a one-million Elo AI plays a one-million-one-hundred Elo AI, the one-million-one-hundred should win 30% more than the one-million. This is completely absurd, because actual superintelligences are just going to draw each other 100% of the time. Ergo, there can never be a one-million Elo chess engine.

It's like chess has a ceiling, where as you get close to that ceiling all the games become draws and you can't rise further. The ceiling is where all the superintelligences play, but the location of the ceiling is just a function of the rules of chess, not a function of how smart the superintelligences are. Magnus Carlsen is closer to the ceiling than he is to the median human's level, which can be taken as merely a statement about how good he is at chess relative to its rules.

In the game "reality," there's probably still a ceiling, but that ceiling is so high that we don't expect any AIs that haven't turned the Earth into computronium to be anywhere near it.

I agree with most of these claims.  However, I disagree about the level of intelligence required to take over the world, which makes me overall much more scared of AI/doomy than it seems like you are.  I think there is at least a 20% chance that a superintelligence with +12 SD capabilities across all relevant domains (esp. planning and social manipulation) could take over the world.  

I think human history provides mixed evidence for the ability of such agents to take over the world.  While almost every human in history has failed to accumulate massive amounts of power, relatively few have tried.  Moreover, when people have succeeded at quickly accumulating lots of power/taking over societies, they often did so with surprisingly small strategic advantages.  See e. g. this post [LW(p) · GW(p)]; I think that an AI that was both +12 SD at planning/general intelligence and social manipulation could, like the conquistadors, achieve a decisive strategic advantage without having to have some kind of crazy OP military technology/direct force advantage.  Consider also Hitler's rise to power and the French Revolution as cases where one actor/a small group of actors was able to surprisingly rapidly take over a country. 

While these examples provide some evidence in favor of it being easier than expected to take over the world, overall, I would not be too scared of a +12 SD human taking over the world.  However, I think that the AI would have some major advantages over an equivalently capable human.  Most importantly, the AI could download itself onto other computers.  This seems like a massive advantage, allowing the AI to do basically everything much faster and more effectively.  While individually extremely capable humans would probably greatly struggle to achieve a decisive strategic advantage, large groups of extremely intelligent, motivated, and competent humans seem obviously much scarier.  Moreover, as compared to an equivalently sized group of equivalently capable humans, a group of AIs sharing their source code would be able to coordinate among themselves far better, making them even more capable than the humans.

Finally, it is much easier for AIs to self modify/self improve than it is for humans to do so.  While I am skeptical of foom for the same reasons you are, I suspect that over a period of years, a group of AIs could accumulate enough financial and other resources that they could translate these resources into significant cognitive improvements, if only by acquiring more compute.  

While the AI has the disadvantage relative to an equivalently capable human of not immediately having access to a direct way to affect the "external" world, I think this is much less important than the AIs advantages in self replication, coordination, an self improvement.

Among humans +6 SD g factor humans do not seem in general as more capable than +3 SD g factor humans as +3 SD g factor humans are compared to median humans.

I'm sceptical of this. Can you say more about why you think this is true?

Assuming a Gaussian distribution, +6 SD is much rarer than +3 SD, which is already quite rare. There's probably less than 10 +6 SD people alive on the earth today, wheras there are ~10 million +3 SD people. Given the role of things like luck, ambition, practical knowledge, etc., it's not surprising that we see several of the +3 SD people accomplishing things far greater than any of the +6 SD g-factor people, purely on the basis of their much greater abundance.

And that's ignoring potential trade-off effects. Among humans, increased intelligence often seems to come at the cost of lowered social skills and practical nature- there are certainly many intelligent people who are good at sociality and practicality, but there is an inverse correlation (though of course, being intelligent also helps directly to make up for those shortcomings). There's no reason to expect that these same trade-offs will be present in an artificial system, who take completely different physical forms, both in size / form-factor, and in the materials and architectures used to build them. And the incentive gradients that govern the development and construction of artificial systems are also quite different from those that shape humans.

Among humans +6 SD g factor humans do not seem in general as more capable than +3 SD g factor humans as +3 SD g factor humans are compared to median humans.

The smartest humans might be more likely to be mentally ill, the smartest monkeys may be more likely to be mentally ill. 

There is no reason to think human intelligence is normally distributed. But if we go with that assumption, +3 SD is 1 in a thousand. Fairly common in intellectual circles. +6 is 1 in a billion. There should be 8 people worldwide that are +6 SD. Even if you knew who those people were (it isn't like the world has a standardized global g factor test that is applied to all humans), you would still have only 8 datapoints, leaving a lot of room for chance. Subjective impressiveness is a subjective measure. 

Discovering something like relitivity takes around +3SD, plus a fair bit of "luck". Some of that luck is being born in a position where you get maths lessons and time to think. Some is deciding to focus your intelligence on physics. Some is all over the place. Like whether or not you read a particular textbook, whether your thinking style is more geometric or formal symbolic. Whether an apple falls on your head. Neurochemical noise. 

When general relativity was discovered, there was probably 1 +6SD human, (smaller population) And statistically they were a peasant farmer never taught anything beyond arthritic, who may well have been conscripted into some war and shot at. Saying "this +6 human failed to invent relativity" is putting an extremely weak upper bound on the capabilities of +6 SD humans. 

If we go into the logic of evolution, selection pressure applies mostly near median. +3SD is already getting to the region where selection pressure is minimal.  Anything beyond that is just getting lucky on the genetic dice. Consider a model where there are 650 genes, each 99% likely to have the correct version, and with a 1% chance of having a deleterious mutation. A +3SD human has all the genes correct. Thus the only way to go beyond that is to not only have all the genes correct, but for randomness to produce a new gene that's even better. The average human has 6.5 bad genes, so in this model, you would need 6.5 beneficial mutations (and no deleterious ones) to get the +6SD human. This is far too unlikely to ever happen.

So without hard data, we can't put reasonable upper bounds on what a +6SD human is capable of. Even with any data you might gather, I don't think it would be easy to learn much about the limits of intelligence. Any signal is mostly about the upper tail produced by evolution. And possibly about who is best at goodhearting your g test. 

FWIW thank you for posting this! It's good to see where different people are coming from on this, and I like several of your other writings

On many useful cognitive tasks(chess, theoretical research, invention, mathematics, etc.), beginner/dumb/unskilled humans are closer to a chimpanzee/rock than peak humans (for some fields, only a small minority of humans are able to perform the task at all, or perform the task in a useful manner

This seems due to the fact that most tasks are "all or nothing", or at least have a really steep learning curve. I don't think that humans differ that much in intelligence, but rather that small differences result in hugely different abilities. This is part of why I expect foom. Small improvements to an AI's cognition seem likely to deliver massive payoffs in terms of their ability to affect the world.
 

On many useful cognitive tasks(chess, theoretical research, invention, mathematics, etc.), beginner/dumb/unskilled humans are closer to a chimpanzee/rock than peak humans

All of these tasks require some amount of learning. AIXI can't play chess if it has never been told the rules or seen any other info about chess ever.

So a more reasonable comparison would probably involve comparing people of different IQ's who have made comparable effort to learn a topic. 

Intelligence often doesn't look like solving the same problems better, but solving new problems. In many cases, problems are almost boolean, either you can solve them or you can't. The problems you mentioned are all within the range of human variation. Not so trivial any human can do them, nor so advanced no human can do them. 

Among humans +6 SD g factor humans do not seem in general as more capable than +3 SD g factor humans as +3 SD g factor humans are compared to median humans.

This is a highly subjective judgement. But there is no particularly strong reason to think that human intelligence has a Gaussian distribution. The more you select for humans with extremely high g factors, the more you goodheart to the specifics of the g factor tests. This goodhearting is relitively limited, but still there at +6SD. 

3.0. I believe that for similar levels of cognitive investment narrow optimisers outperform general optimisers on narrow domains.

I think this is both trivially true, and pragmatically false. Suppose some self modifying superintelligence needs to play chess. It will probably largely just write a chess algorithm and put most of it's compute into that. This will be near equal to the same algorithm without the general AI attached. (probably slightly worse at chess, the superintelligence is keeping an eye out just in case something else happens, a pure chess algorithm can't notice a riot in the spectator stands, a superintelligence probably would devote a little compute to checking for such possibilities.)

However, this is an algorithm written by a superintelligence, and it is likely to beat the pants off any human written algorithm.

4.1. I expect it to be much more difficult for any single agent to attain decisive cognitive superiority to civilisation, or to a relevant subset of civilisation.

Being smarter than civilization is not a high bar at all. The government often makes utterly dumb decisions. The average person often believes a load of nonsense. Some processes in civilization seem to run on the soft minimum of the intelligences of the individuals contributing to them. Others run on the mean. Some processes, like the stock market, are hard for most humans to beat, but still beaten a little by the experts. 

My intuition is that the level of cognitive power required to achieve absolute strategic dominance is crazily high.

My intuition is that the comparison to a +12SD human is about as useful as comparing heavy construction equipment to top athletes. Machines usually operate on a different scale to humans. The +12 SD runner isn't that much faster than the +6SD runner, Especially because, as you reach into the peaks of athletic performance, the humans are running close to biological limits, and the gap between top competitors narrows. 

This is reasonably close to my beliefs. An additional argument I'd like to add is:

• Even if superintelligence is possible, the economic path towards it might be impossible.

There needs to be an economically viable entity pushing AI development forward every step of the way. It doesn't matter if AI can "eventually" produce 30% worldwide GPD growth. Maybe diminishing returns kick in around GPT-4, or we run out of useful training data to feed to the models (We have very few examples of +6 SD human reasoning, as MikkW points out in a sibling comment).

Analogy: It's not the same to say that a given species with X,Y,Z traits can survive in an ecosystem, than to say it can evolve from its ancestor in that same ecosystem.

Fascinating insight with the chimpanzee. I believe this connects to the argument that is sometimes made: an ASI takeover is unlikely because how widely distributed the physical levers of power are in the real world. Ok... but chimpanzees don't have access to massively scaleable industrial weapon technology, such as nuclear, biological etc. They don't live in an increasingly electronically connected world. They don't rely on an electricity grid, among others, for their many daily needs. Also they live in groups or troops - not tribes, but that was more for the pedantic kick of it.

I think it's a mistake to think of current chess or go engines as being at maximum capability. If we would throw a few billion dollar worth of compute at them they would likely get significantly better. 

Narrow Optimisers Outperform General Optimisers on Narrow Domains

That's true sometimes but not always. Notably, GATO is better at controlling a Sawyer arm than more specialized optimizers. Given that the company that sells the Sawyer arm spent a lot of time developing software to control it, that's impressive. 

I expect Magnus Carlsen to be closer in ELO to a bounded superintelligence than to a median human.

Seems like this sort of claim could be something tractable that would qualify as material progress on understanding bounds to superintelligence? I'm thinking about results such as this [LW · GW].

However I think that post's title oversells the result-- from the paper:

This paper has demonstrated that even superhuman agents can be vulnerable to adversarial policies. However, our results do not establish how common such vulnerabilities are: it is possible Go-playing AI systems are unusually vulnerable.

There may be superhuman go playing models that are more robust.

I'm also just noting my thoughts here, as I'm also very interested in foom dynamics and wondering how the topic can be approached.

Also - I don't really get "the general intelligence is composite anyway" argument. Ok - I also believe that it is. But what would prevent an ASI from being developped as a well-coordinated set of many narrow optimizers?

Also - why the fixation on 12 SD? It's not that high really. It sounds high to a human evaluating another human. Bostrom made a good point on this - the need to step out of the antropomorphic scale. This thing could very well reach 120 SD (the fact that we wouldn't even know how to measure and recognize 120 SD is just an indication of our own limitations, nothing more), and make every human look like a clam.