Bloggingheads: Robert Wright and Eliezer Yudkowsky

It reminds me of when Richard Dawkins was doing a bunch of interviews and discussions to promote his then-latest book The God Delusion. It was kind of irritating to hear the people he was talking with failing again and again in the same predictable ways, raising the same dumb points every time. And you could tell that Dawkins was sick of it, too. The few times when someone said something surprising, something that might force him to change his mind about something (even a minor point), his face lit up and his voice took on an excited tone. And when he was particularly uncertain about something, he said so.

People accused him of being arrogant and unwilling to change his mind; the problem is that the people he was arguing with were just so piteously wrong that of course he's not going to change his mind from talking with them. It's funny, because one of the things I really like about Dawkins is that he's genuinely respectful in discussions with other people. Sometimes barbed, but always fundamentally respectful. When the other person says something, he won't ignore it or talk past them, and he assumes (often wrongly) that whoever he's speaking with is intelligent enough and sane enough to handle a lack of sugarcoating.

And of course, all this led to accusations of cultishness, for exactly the same reasons that are making you uncomfortable.

Maybe it's because his brain is so large that my mirror neurons have to fire three times faster to compensate, but I always get so frustrated when watching Eliezer discussing things with non-SIAI people.

Start with a bit of LW's own "specialized cult jargon" (I kid, really!)... specifically the idea of inferential distance.

Now imagine formalizing this concept more concretely than you get with story-based hand waving, so that it was more quantitative -- with parametrized shades of grey instead of simply being "relevant" or "not relevant" to a given situation. Perhaps it could work as a quantitative comparison between two people who could potentially Aumann update with each other, so that "ID(Alice,Bob) == 0 bits" when Alice knows everything Bob knowsand they already believe exactly the same thing, and can't improve their maps by updating about anything with each other. If its 1 bit then perhaps a single "yes/no Q&A" will be sufficient to bring them into alignment. Larger and larger values imply that they have more evidence (and/or more surprising evidence) to share.

(A simple real world proxy for ID(P1,P2) might be words read or heard by P1 that P2 wrote or spoke. The naive conversion from words to bits would then be to multiply words by ~10 to get bits of information while crossing your fingers and hoping that every word was a novel report of evidence rather than a re-summarization of basically the same information that might let evidential double-counting sneak in the back door. So maybe "ID(Alice,Bob) == 50 bits" means there are five perfectly chosen words that Bob could say to let Alice sync with him?)

Now consider naively (IE imagine that everyone is a baseline human operating mostly on folk wisdom) that Alice and Bob are in a debate being judged by Jim where Jim is forced to judge in favor of one or the other debater, but not both or neither. Given this background information, H, what do you think of the specific probability estimate:

PROB ( J judges for A | H and ID(J,A) < ID(J,B) )

If this is 0.5 then the concept of inferential distance gives no special predictive power about about how Jim will judge. I think this is unlikely, however, given what I suspect about the kinds of mistakes Alice and Bob will make (assuming things intelligible to themselves are intelligible to everyone) and the kinds of mistakes that Jim will make (thinking that if something isn't transparently obvious then whatever was said is just wrong). My guess would be that Jim would judge in favor of Alice more often, simply because he already deeply understands more of what she says in the course of the debate.

So... I think the critical question to ask is what evidence from the world might Robert Wright have talked about if he hadn't been wrongfooted when he was pulled into Eliezer's unfamiliar frameworks for describing optimization processes and for doing expectation-based-argumentation (that you're already familiar with but that Robert presumably hasn't read up on).

In point of fact, Robert has published several books with lots of evidence even if he isn't good at defending himself from Eliezer's rhetorical jujitsu. Basically none of the contents of his books came out because, although Robert offered helpfully leading questions about Eliezer's area of specialization (which Eliezer complimented him on -- I think maybe misunderstanding his basic-conversational-generosity for agreement-and-hence-intelligence) Eliezer didn't reciprocate which meant that the video audience didn't get to see Robert's specialist knowledge.

Here is a bit from Amazon's quote of Publisher's Weekly review of Robert's book "Nonzero" describing the kinds of things Robert could have been talking about if Eliezer had "played along for the sake of argument" before going into attack mode:

The non-zero-sum dynamic, Wright says, is the driving force that has shaped history from the very beginnings of life, giving rise to increasing social complexity, technological innovation and, eventually, the Internet. From Polynesian chiefdoms and North America's Shoshone culture to the depths of the Mongol Empire, Wright plunders world history for evidence to show that the so-called Information Age is simply part of a long-term trend. Globalization, he points out, has been around since Assyrian traders opened for business in the second millennium B.C. Even the newfangled phenomenon of "narrowcasting" was anticipated, he claims, when the costs of print publishing dropped in the 15th century and spawned a flurry of niche-oriented publications. Occasionally, Wright's use of modish terminology can seem glib: feudal societies benefited from a "fractal" structure of nested polities, world culture has always been "fault-tolerant" and today's societies are like a "giant multicultural brain." Despite the game-theory jargon, however, this book sends an important message that, as human beings make moral progress, history, in its broadest outlines, is getting better all the time.

This sounds to me like a lot of non-fictional evidence. My guess is that Wright is ultimately just more interested in the Invisible Hand than in Azathoth and sees the one "deity" as being more benevolent than the other. If I generously misinterpret him as claiming this, I notice that I'm already willing to believe this because Azathoth seems kind of scary and horrifying to me. If I imagine more evidence this way I'm more inclined to believe it...

So expect that if the conversation in the video had been more about "cooperative truth seeking" than about "debate winning", then Robert's would have said something and justified it in a way that improved my thinking.

I think a lot of what's scary about many real world epistemic failure modes is not that they are full of gross logical fallacies, or involve wearing silly clothes, or get you to work on truly positive "public goods", but that that they deflect you from acquiring certain kinds of evidence without your even noticing it.

Bear in mind that, like many good works of pop science, the vast majority of what the Sequences present is other people's ideas; I'm much more confident of the value of those ideas than of the parts that are original to Eliezer.

There's not much he could be proven wrong about. What EY mainly accomplished is to put the right pieces, that have already been out there before him, together and create a coherent framework.

But since I've only read maybe 5% of LW I might be wrong. Is there something unique that stems from EY?

Another problem is that what EY is saying is sufficiently vague so that you cannot argue with it if you do not doubt some fundamental attributes of reality.

I'm not trying to discredit EY. I actually don't know of any other person that comes even close to his mesh of beliefs. To the extent that I'm much more relaxed since I know about him for that if I was going to die there's everything and much more I ever came up with contained inside EY' mind :-)

Anyway, I can't help and often muse about the possibility that EY is so much smarter that he actually created the biggest scam ever around the likelihood of uFAI to live of donations by a bunch of nonconformists. - "Let's do what the Raelians do! Let's add some nonsense to this meme!"

Of course I'm joking, hail to the king! :-)

Do you know how epistemically distressing it is to have learned half the things you know from one person who keeps on getting proven right?

Yeah, huge red flag. I'll also note that reading Eliezer's stuff made me feel like I got to extend my beliefs in the same direction away from mainstream that they were already skewed, which is probably why I was extremely receptive to it.

Even though I've learned a lot, I don't get to congratulate myself for a real Mind Change.

(Runs off to read the Two Cult Koans.)

Or this.

There are not very many seriously written-up position statements from Eliezer. So, it probably doesn't represent a very attractive target for "academics" to attack.

There are a couple of papers about the possibility of THE END OF THE WORLD. That is an unconvential academic subject - partly because no instances of this have ever been observed.

Actually, do you have a citation for this datum?

Edit: The author has commented downthread.

That doesn't seem like a stable equilibrium -- too much incentive for the rabbits to be "over-cautious" for foxes at the expense of running ability. If they can figure that "being able to notice and turn toward a fox" is just as good as having the energy to escape a fox, then they'll over-invest in being good at this signal until the foxes realize it's not a reliable signal of a failed hunt.

Elaboration of my skepticism on this claim here.

I'm willing to bet that such software would be very hard to develop.

Requiring 5 second pauses after speaking to allow for thought would be an interesting experiment.

Eliezer's repertoire is higher-status because it's less reactive.

I agree that Eliezer maintained his calm better, but I don't believe that Wright is the simpleton you seem to be painting him to be. I've watched a lot of his videos, and I would say there are very rarely moments of "red-faced rage," and certainly none in this video. He was at times frustrated, but he really is working to understand what Eliezer is saying.

The reason Wright got bent out of shape (my theory): Eliezer seemed to imply the communal mind theory is Wright's wishful thinking. This seems a little simplistic. I do believe Wright is a little disingenuous, but it is a little more subtle than that. It appears to me he thinks he has an idea that can be used to wean millions of the religious faithful to a more sensible position, and he is trying to market it. And he would sort of like to have it both ways. With hard edged science folk he can say all that with a wink because we are sophisticated and we get it. And the rubes can all swallow it hook line sinker.

I forget the exact term Eliezer used that seemed to set him off. It was something like wishing or hoping or rooting-for. Then Wright's speech got loud and fast and confused and his blood pressure went up. He seemed to feel like he was being accused of acting in bad faith when he was claiming to try to be helpful.

Maybe Wright's friends thought he did great under fire?

I wouldn't say the evasiveness was purposeful. Robert misunderstood something Eliezer said fairly early, taking it as an attack when Eliezer was trying to make a point about normative implications. This probably switched Robert out of curiosity-mode and into adversarial-mode. Things were going fine after Eliezer saw what was happening and dropped the subject. But later, when Robert didn't understand Eliezer's argument, adversarial-mode was active and interpreted it as Eliezer continuing (in Robert's mind) to be a hostile debate partner. I doubt Robert thought he was in trouble; more likely he thought Eliezer was in trouble and was being disingenuous.

I do not know his position or view to see that. But i got the impression he was badly prepared. Severe misunderstandings, and a lesson in staying calm.

This implies that we exist in a universe that tries out many possibilities (many worlds interpretation)

Big World rather. Many-worlds doesn't give different laws of physics in the way that the string theory landscape or Tegmark's mathematical universe hypothesis do.

The main problem in the discussion that appeared to me is the fact that the present state of the universe is really unlikely, and you would never get it by chance.

Any hypothesis that assigns a really low probability to the present state of the universe is probably wrong.

Have you read Sean Carroll's "From Eternity to Here"? It's a fairly layman-friendly take on that problem (or I suppose more accurately, the problem of why the past was in such an improbable state of low entropy). I think his explanation would fall under Carl Schulman's "Big World" category.

I think this argument is mostly about whether purpose is there - not about where it comes from.

Designoid entities as a result of anthropic selection effects seem quite possible in theory - and it would be equally appropriate to describe them as being purposeful [standard teleology terminology disclaimers apply, of course].

what is the LW policy on commenting on old threads? All good? Frowned upon?

It's pretty much okay. If there is a recent "Sequence rerun" thread about in in Discussion, then the discussion should happen there instead, but otherwise there's no particular issues.

"‘Teleology is like a mistress to a biologist: he cannot live without her but he’s unwilling to be seen with her in public.’ Today the mistress has become a lawfully wedded wife. Biologists no longer feel obligated to apologize for their use of teleological language; they flaunt it. The only concession which they make to its disreputable past is to rename it ‘teleonomy’."

So when unmarried, biology and teleogy happened to have the same last name. But after the marriage, teleology changed her surname to be different? No wonder ordinary people don't understand science!

"‘Teleology is like a mistress to a biologist: he cannot live without her but he’s unwilling to be seen with her in public.’ Today the mistress has become a lawfully wedded wife. Biologists no longer feel obligated to apologize for their use of teleological language; they flaunt it.

Sure, so long as you recognize that "purpose" in

"The purpose of the heart is to pump blood."

cashes out as something different from

"The purpose of the silicon CPU is to implement a truth table."

In my experience, there are about zero philosophers of science who both understand this distinction, and harp on this point about teleology in biology. Here is one I read recently.

As Wright says, organisms are not that harmonious anyway. At every polymorphic locus, two alleles are engaged in a battle-to-the-death.

There is still enough cooperation for people to describe the resulting behaviour as "purposeful" - despite all the underlying replicator-level conflicts that produce it.

Why do you like it? How does it maximize paperclips?

It seems to me your task would be easier if people didn't acknowledge paperclip maximisers. Humans aren't going to start maximising paperclips, so your best odds are to do and end run around us. That's easier to do if we aren't aware of your existence.

Our universe does seem to have infinitely many observers in it but that doesn't necessarily mean it has to have a particularly high density of them. It instead indicates that particularly densely populated universes are unlikely for some other reason (e.g. uFAI or other planet-wide or lightcone-wide existential risks). Alternatively, it could be that for some reason the computation 'Earth around roughly 2010' includes a disproportionately large amount of the measure of agents in the timtyler reference class. Perhaps we third millennium human beings are a particularly fun bunch to simulate and stimulate.

The purpose of natural selection, fine-tuning of physical constants in our universe, and of countless other detailed coincidences (1) was to create me. (Or, for the readers of this comment, to create you)

The optimization process that optimized all these things is called anthropics. Its principle of operation is absurdly simple: you can't find yourself in a part of the universe that can't create you.

That's not what "purpose" means.

I don't think this is much of an insight, to be honest. The "anthropic" interpretation is a statement that the universe requires self-consistency. Which is, let's say, not surprising.

The purpose of natural selection, fine-tuning of physical constants in our universe, and of countless other detailed coincidences (1) was to create me. (Or, for the readers of this comment, to create you)

My feeling is that this is a statement about the English language. This is not a statement about the universe.

Once you take away all the meta-purpose that is caused by anthropics, then I really do think there is no more purpose left.

There's also the possibility of "the adapted universe" idea - as laid out by Lee Smolin in "The Life of the Cosmos" and James Gardner in "Biocosm" and "Intelligent-Universe".

Those ideas may face some Occam pruning - but they seem reasonably sensible. The laws of the universe show signs of being a complex adaptive system - and anthropic selection is not the only possible kind of selection effect that could be responsible for that. There could fairly easily be more to it than anthropic selection.

Then there's Simulism...

I go into the various possibilites in my "Viable Intelligent Design Hypotheses" essay:

http://originoflife.net/intelligent_design/

Robert Wright has produced a broadly similar analysis elsewhere.