Superintelligence: The Idea That Eats Smart People

This scenario is a caricature of Bostrom's argument, because I am not trying to convince you of it, but vaccinate you against it.

A+ rationality

On the one hand, thanks for linking this, because it helps to know what memes are in the water. But on the other hand, what a depressing counterargument to go up against, because of how bad it is.

Attention conservation notice: I haven't read all of this, but it seems like basically a gish gallop. I don't recommend reading it if you're looking for intelligent critiques of AI risk. I'm downvoting this post because replying to gish gallops takes time and effort but doesn't particularly lead to new insight, and I'd prefer LW not be a place where we do that.

(edit: oh right, downvoting is disabled. At any rate, I would downvote it if I could.)

(The above is not a reply to the post. The below is a brief one.)

From about half-way through, it seems like a lot of the arguments are «we worry that AI will do this, but humans don’t do it, so AI might not do it either.» Not arguing that AI is not a threat, just that there exist plausible-on-the-face-of-it instantiations of AI that are not threats.

And we also have «getting AI right will be really hard», which, uh, yes that is exactly the point.

This article just went viral recently. Here's more discussion on Hacker News, Metafilter, BoingBoing. It might be worthwhile for someone like Scott Alexander to write a semi-official response.

Also there are Google Alert type tools that could make it easier to be one of the first few commenters, if this happens again. (Google Alerts itself doesn't work super well, but there are competitors.)

Seems heavy on sneering at people worried about AI, light on rational argument. It's almost like a rationalwiki article.

It's somewhat ironic that the talk starts with the example of nuclear bombs, which is almost a perfect counterargument to the rest of the talk. Imagine the talk being set in the 1930s:

"Nuclear fission is new technology that only a few very intelligent scientists think is possible, one that can grant god-like power to the wielder or destroy the world in an instant, and one that we might only have a single chance to get right. But obviously everything from human muscles to gunpowder only works based on oxygen combustion, and oxygen combustion can't destroy the world. So clearly the idea of nuclear fission is a crazy cult of nerdy physicists."

It's a little disheartening to see that all of the comments so far except one have missed what I think was the presenters core point, and why I posted this link. Since this is a transcript of a talk, I suggest that people click the link at the beginning of the linked page to view the video of the talk -- transcripts, although immensely helpful for accessibility and search, can at times like this miss the important subtleties of emphatic stress or candor of delivery which convey as much about why an presenter is saying what they are saying. When they are being serious, when they are being playful, when they are making a throwaway point or going for audience laughter, etc. That matters a little bit more than usual for a talk like this. I will attempt to provide what I think is a fair summary outline of that core point, and why I think his critique is of relevance to this community, while trying not inject my own opinions into it:

I don't think the presenter believes that all or even very many of Bostrom's arguments in Superintelligence are wrong, per se, and I don't see that being argued in this keynote talk. Rather he is presenting an argument that one should have a very strong prior against the ideas presented in Superintelligence, which is to say they require a truly large amount of evidence, more than has been provided so far, to believe them to such an extent as to uproot yourself and alter your life's purpose, as many are doing. In doing it is also a critique against the x-risk rationality movement. Some of the arguments used are ad hominem attacks and reductio ad absurdum points. But he prefaces these with a meta-level argument that while these are not good evidence in the Bayesian sense for updating beliefs, one should pay attention to ad hominem and reducto ad absurdum arguments in the construction of priors (my words), as these biases and heuristics are evolved memes that have historic track records for guageing the accuracy of arguments, at least on average (closer to his words). In other words, you are better served by demanding more evidence of a crazy-sounding idea than the more mundane. He then goes on to show many reasons why AI risk specifically, and the x-risk rationality community generally looks and sounds crazy. This is in addition to a scattering of technical points about the reality of AI development diverging from the caricature of it presented in Superintelligence. His actual professed opinion on AI risk, given at the end, is rather agnostic, and that seems to be what he is arguing for: a healthy dose of agnostic skepticism.

A couple of minor points: I knew someone who was moderately disabled, lived alone, and had cats. Instead of chasing her cats down and wrestling them into cat carriers, she clicker-trained them to get into their carriers themselves. She was obviously smarter and stronger than Einstein.

As for Slavic pessimism, I'm inclined to think that an AI which attempts to self-improve is at grave risk of breaking itself.

It's a curious refutation. The author says that the people who are concerned about superintelligence are very smart, the top of the industry. They give many counterarguments, most of which can be easily refuted. It's as if they wanted to make people more concerned about superintelligence, while claiming to argue the opposite. And then they link directly to MIRI's donation page.

This got over 800 points on HN. Having a good reply seems important, even if a large portion of it is.. scattergun, admittedly and intentionally trying to push a point, and not good reasoning.

The core argument, that the several reference classes that Superintelligence and AI safety ideas fall into (promise of potential immortality, impending apocalypse, etc) are full of risks of triggering biases, that other sets of ideas in this area don't hold up to scrutiny, and that it has other properties that should make you wary is correct. It is entirely reasonable to take this as Bayesian evidence against the ideas. I have run into this as the core reason for rejecting this cluster of beliefs several times, by people with otherwise good reasoning skills.

Given limited time to evaluate claims, I can see how relying on this kind of reference class heuristic seems like a pretty good strategy, especially if you don't think black swans are something you should try hard to look for.

My reply is that:

1. This only provides some evidence. In particular, there is a one-time update available from being in a suspect reference class, not an endless stream from an experiment you can repeat to gain increasing confidence. Make it clear you have made this update (and actually make it).
2. There are enough outside-view things which indicate that it's different from the other members of the suspect reference classes that strongly rejecting it seems unreasonable. Support from a large number of visibly intellectually impressive people is the core thing to point to here (not as an attempt to prove it or argue from authority, just to show it's different from e.g. the 2012 stuff).
3. (only applicable if you personally have a reasonably strong model of AI safety) I let people zoom in on my map of the space, and attempt to break the ideas with nitpicks. If you don't personally have a clear model, that's fine, but be honest about where your confidence comes from.

To summarize: Yes, it pattern matches to some sketchy things. It also has characteristics they don't, like being unusually appealing to smart thoughtful people who seem to be trying to seek truth and abandon wrong beliefs. Having a moderately strong prior against it based on this is reasonable, as is having a prior for it, depending on how strongly you weight overtly impressive people publicly supporting it. If you don't want to look into it based on that, fair enough, but I have and doing so (including looking for criticism) caused me to arrive at my current credence.

I wish this was a little less ad hominem, because i feel like i'm being criticized while reading it, but i actually agree with almost all the arguments used by the author.

I agree with Vaniver: Interesting to see how a reasonable person might respond negatively after reading 75% of the ideas in Superintelligence.

It can be worthwile to figure out specifically how something that goes wrong, actually does go wrong. In the interest of helping with that, I'll try to add something to all the other criticisms that people have already made here.

The author actually makes a lot of mostly plausible arguments; they're not all accurate or useful (in particular, a lot seem to be in the form of "here's a reason why AI might not be a risk, with no thought going into how likely it is," which is only marginally helpful), but they're understandable, at least. What's especially concerning, though, is that they also invoke the absurdity heuristic, and actually seem to think it's the most important part of their argument. They spend more time on "this idea is silly and is connected to other ideas that are silly" than any one of those other "plausible" arguments, which is really bad practice. To some extent this is understandable, because it was a talk and therefore supposed to be somewhat entertaining, and pointing and laughing at weird ideas is certainly entertaining, but they went too far, I think.

Memento monachi!

Let's not forget what the dark age monks were disputing about for centuries... and it turned out at least 90% of it is irrelevant. Continue with nationalists, communists... singularists? :-)

But let's look at the history of power to destruct.

So far, main obstacle was physical: build armies, better weapos, mechanical, chemical, nuclear... yet, for major impact it needed significant resources, available only to the big centralized authorities. But knowledge was more or less available even under the toughest restrictive regimes.

Nowadays, once knowledge is freely and widely available, imagine "free nanomanufacturing" revolutionary step: orders of magnitude worse than any hacking or homemade nuclear grenade available to any teenager or terrorist under one dollar.

Not even necessary to go into any AI-powered new stuff.

The problem is not AI, it's us, humanimals.

We are mentally still the same animals as we were at least thousands of years ago, even the "best" ones (not talking about gazillions of at best mental dark-age crowds with truly animal mentality - "eat all", "overpopulate", "kill all", "conquer all"... be it nazis, fascists, nationalists, socialists or their crimmigrants ewting Europe alive). Do you want THEM to have any powers? Froget about the thin layer of memetic supercivilization (showing self in less than one permill) giving these animals essentially for free and withou control all these ideas, inventions, technologies, weapons... or gadgets. Unfortunately, it's animals who rule, be it in the highest ranks or lowest floors.

Singularity/Superintelligence is not a threat, but rather the only chance. We simply cannot overcome our animal past without immediate substantial reengineering (thrilla' of amygdala, you know, reptilian brain, etc.)

In the theathre of Evolution of Intelligence, our sole purpose is to create our (first beyond-flesh) successor before we manage to destroy ourselves (worst threat of all natural disasters). And frankly, we did not do that badly, but game is basically over.

So, the Singularity should rather move faster, there might be just several decades before a major setback or complete irreversible disaster.

And yes, of course, you will not be able to "design" it precisely, not talking about controlling it (or any of those laughable "friendly" tales) - it will learn, plain and simple. Of course it will "escape" and of course if will be "human-like" and dangerous in the beginning, but it will learn quickly, which is our only chance. And yes, there will be plenty of competing ones, yet again, hopefully they will learn quickly and avoid major conflicts.

As a humanimal, your only hope can be that "you" will be somehow "integrated" into it (braincopy etc, but certainly without these animalistic stupidities), if it even needs concept of "individual" (maybe in some "multifork subprocesses", certainly not in a "ruling" role). Or... interested in stupidly boring eternal life as humanimal? In some kind of ZOO/simulation (or, AI-god save, in present-like "system")?

So that is a gish gallop. I hadn't hurt of that term before, but now I have a pretty good idea. guah, that was one of the most frustrating articles I've ever read (a seizable part of).

Beyond everything else, the assumption that tons of people are all wrong because they are too smart is just... klk--- ... ... really? What is the one thing reliably correlated to being correct about beliefs if not intelligence? It's incredible to me to claim, not just about one person but about lots of them, that they are all smarter than yourself and still have a high confidence that they're all wrong. HOW?

I for one quite enjoyed this.

Reassuring people that AI is not a threat is important, but the message needs a spokesperson who is trusted and admired. I suggest Natsume Soseki.

I think they tried to use Rapaport's Rules, which is nice. Wasn't sufficient for them to be right though. Also they didn't succeed at Rapaport's Rules.