Charles Stross: Three arguments against the singularity

I sometimes update in favor of a controversial claim when I see a smart opponent of that claim put forth their best argument against it, and the argument just isn't that good.

I'm not sure which parts of the essay are supposed to be more convincing than others, so I made a standing offer on HN to argue against any individual point.

It seems like his best points are tangential to the supposed subject of argument.

He strangely seems to take embryonic personhood for granted while in the same paragraph noting that the relevant metric is consciousness. His assertion that general intelligence requires emotional instability or large amounts of sleep seems unsupported in principle. Also, his suggestion that all programs will become intelligent once any AI exists is absurd. The idea of reorienting an AI's sense of self is an interesting one, though.

He seems to have a very primitive concept of uploading - if we can upload a brain, we can simulate a body for it. If uploaded humans want to be able to compete with AI, they could participate in specialized training environments to rewire their senses toward the pure digital instead of analog-to-digital. Rewiring senses is something we already know how to do and in a digital upload, there are no biological limits on neuroplasticity. For those who would rather remain in a human-esque form, set up digital reservations, problem solved.

By the way, I do not think that it is a coincidence that Robin Hanson also wrote a blog post on the Singularity yesterday evening. I wrote both of them at the weekend ;-)

Summary:

• Human-level or above AI is impossible because either they're going to be people, which would be bad (we don't want to have to give them rights, and it'd be wrong to kill them), or they're going to refuse to self-improve because they don't care about themselves.
• Uploading is possible but will cause a religious war. Also, if there are sentient AIs around, they'll beat us.
• It's unlikely we're in a simulation because why would anyone want to simulate us?

Pretty reasonable for someone who says "rapture of the nerds". The main problem is anthropomorphism; Stross should read up on optimization processes. There's no reason AIs have to care about themselves to value becoming smarter.

(I've never found a good argument for "AGI is unlikely in theory". It makes me sad, because Stross is looking at practical aspects of uploading, and I need more arguments for/against "AGI is unlikely in practice".)

Stross makes a few different claims, but they don't seem to add up to any kind of substantial critique of singularity-belief. The simulation argument is just a tangent: it doesn't actually have anything to do with the probability of a technological singularity. He basically says he doesn't think uploading is likely and then describes why it is undesirable anyway.

The argument that religious opposition to uploading will prevent it from being introduced is parochial, since there are huge countries like Korea, China, and Japan which are overwhelmingly secular, and where the technology could very plausibly be developed. In addition, secularism is increasingly dominant in North America and Europe, so that viewpoint is short-sighted as well. More to the point, I sincerely doubt religious people would really have a problem with (other people) uploading in the first place. Stross's view that people would launch holy wars against uploaders seems like paranoia to me.

On further reflection, this piece and the comments on his site make me skeptical about ever achieving human intelligence.

The post by Stross triggered a reaction by Alex Knapp, 'What’s the Likelihood of the Singularity? Part One: Artificial Intelligence', which in turn caused Michael Anissimov to respond, 'Responding to Alex Knapp at Forbes'.

I also think that human-level AI is unlikely for a number of reasons – more than a few of them related to the differences between biological and machine intelligence. For one thing, we’re approaching the end of Moore’s Law in about a decade and a half or so, and generalized quantum computing isn’t likely to be with us anytime soon yet. For example, D-Wave’s adiabatic quantum computer isn’t a general computer – it’s focused on optimization problems. But even with that, the differences between human, animal and machine intelligence are profound.

...

At the present time, a lot of the ways that human beings think is simply unknown. To argue that we can simply “workaround” the issue misses the underlying point that we can’t yet quantify the difference between human intelligence and machine intelligence.

...

...it’s definitely worth noting that computers don’t play “human-level” chess. Although computers are competitive with grandmasters, they aren’t truly intelligent in a general sense – they are, basically, chess-solving machines. And while they’re superior at tactics, they are woefully deficient at strategy, which is why grandmasters still win against/draw against computers.

...computers lose to humans because they are simply no match for humans at creating long-term chess strategy.

...

As for Watson’s ability to play Jeopardy!, it’s important to note that while Watson did win the Jeopardy tournament he played in, it’s worth noting that Watson’s primary advantage was being able to beat humans to the buzzer (electric relays are faster than chemical ones). Moreover, as many who watched the tournament (such as myself) noted, Watson got worse the more abstract the questions got.

...

What computers are smart at are brute-force calculations and memory retrieval. They’re not nearly as good at pattern recognition or the ability to parse meaning and ambiguity, nor are they good at learning.

...

...there are more than a few limitations on human-level AI, not the least of which are the actual physical limitations coming with the end of Moore’s Law and the simple fact that, in the realm of science, we’re only just beginning to understand what intelligence, consciousness, and sentience even are, and that’s going to be a fundamental limitation on artificial intelligence for a long time to come. Personally, I think that’s going to be the case for centuries.

The content of the arguments has been effectively refuted already, but I have one thing to add. The piece, and especially the comments section on Stross' site, really need to sort out their terms. They all seem to be equivocating between intelligence and consciousness. Something can be a really powerful optimization process without having humanlike emotions or thought processes.

He seems to be pretty optimistic about an artificial intelligence architecture requiring lots of custom-crafted crunchy bits to go from being able to autonomously clean and fix your apartment, buy you groceries, make you lunch, drive you around town and figure out it needs to save your life and do so when you get a cardiac arrest to being able to do significant runaway learning and reasoning that will have serious and cumulative unintended consequences. And per Omohundro's Basic AI Drives paper, needing to have efficient real-world intelligence might count as a potentially dangerous motivation on its own, so not giving the things a goal system cribbed from humans won't help much.

I like how he goes from:

Uploading ... is not obviously impossible unless you are a crude mind/body dualist.

to:

But even if mind uploading is possible

To be sure, it continues as an "if ... and ...", but way to suggest two arguments where you've made just one.

This article seems to have sparked many comments in the blogosphere - for instance here:

• http://blogs.forbes.com/alexknapp/2011/06/23/whats-the-likelihood-of-the-singularity-part-one-artificial-intelligence/
• http://blogs.forbes.com/alexknapp/2011/06/24/whats-the-likelihood-of-the-singularity-part-two-uploading-the-brain/
• http://blogs.forbes.com/alexknapp/2011/06/25/whats-the-likelihood-of-the-singularity-part-three-a-simulated-world/

...with many replies from all over the place.

Was this material really worth treating seriously? It seemed rather low-grade to me.

I was hoping for something similar to Nick Szabo's claim that specialized AI will pose existential threats before we ever have to worry about self improvers given his (Stross) fictional works.

I'm not sure exactly what arguments 2 and 3 are.

If the subject of consciousness is not intrinsically pinned to the conscious platform, but can be arbitrarily re-targeted, then we may want AIs that focus reflexively on the needs of the humans they are assigned to — in other words, their sense of self is focussed on us, rather than internally. They perceive our needs as being their needs, with no internal sense of self to compete with our requirements. While such an AI might accidentally jeopardize its human's well-being, it's no more likely to deliberately turn on it's external "self" than you or I are to shoot ourselves in the head. And it's no more likely to try to bootstrap itself to a higher level of intelligence that has different motivational parameters than your right hand is likely to grow a motorcycle and go zooming off to explore the world around it without you.

Interesting take on friendly AI.

don't get your hopes up, this is pretty weak stuff.

It is not on the same page - and not in a good way.

I'm disappointed. Essays like this are supposed to contain at least an off-hand reference to Ayn Rand somewhere.

ETA: This comment being at -1 is making me worried that people are interpreting it as something else than a hopefully uncontroversial gripe about a rhetorical style that takes cheap shots against complicated technical claims by vaguely associating them with disliked political/cultural currents.

The exponential function has no singularities (unlike the hyperbola).

More complete failure to get it from Stross: http://www.orbitbooks.net/2011/07/08/artificial-stupids/