comment by Yvain2 ·
2008-11-04T00:08:00.000Z · LW(p) · GW(p)
Disappointing. I kept on waiting for Eliezer to say some sort of amazingly witty thing that would cause everything Jaron was saying to collapse like a house of cards, but either he was too polite to interrupt or the format wasn't his style.
At first I thought Jaron was talking nonsense, but after thinking it over for a while, I'm prepared to give him the benefit of the doubt. He said that whether a computer can be intelligent makes no difference and isn't worth talking about. That's obviously wrong if he's using a normal definition of intelligent, but if by intelligent he means "conscious", it makes a lot of sense and he's probably even right - there's not a lot of practical value in worrying about whether an intelligent computer would be conscious (as opposed to a zombie) at this point. He wouldn't be the first person to use those two words in weird ways.
I am also at least a little sympathetic to his "consciousness can't be reduced" argument. It made more sense once he said that consciousness wasn't a phenomenon. Still not perfect sense, but trying to raise something stronger from its corpse I would argue something sort of Kantian like the following:
Goldbach's conjecture says that every number is the sum of three primes. It hasn't been proven but there's a lot of inductive evidence for it. If I give you a difficult large number, like 20145, you may not be capable of figuring out the three primes, but you should still guess they exist. Even if you work quite hard to find them and can't, it's still more likely that it's a failure on your part than that the primes don't exist.
However, North Dakota is clearly not the sum of three primes. Even someone with no mathematical knowledge can figure this out. This statement is immune to all of the inductive evidence that Goldbach's conjecture is true, immune to the criticism that you simply aren't smart enough to find the primes, and doesn't require extensive knowledge of the history and geography of North Dakota to make. It's just a simple category error.
Likewise, we have good inductive evidence that all objects follow simple scientific/reductionist laws. A difficult-to-explain object, like ball lightning, probably still follows scientific/reductionist laws, even if we haven't figured out what they are yet. But consciousness is not an object; it's the subject, that by which objects are perceived. Trying to apply rules about objects to it is a category error, and his refusal to do so is immune to the normal scientific/reductionist criticisms you would level against someone who tried that on ball lightning or homeopathy or something.
I'm not sure if I agree with this argument, but I think it's coherent and doesn't violate any laws of rationality.
I agree with everyone who found his constant appeal to "I make algorithms, so you have to believe me!" and his weird nervous laughter irritating.