Eliezer's Sequences and Mainstream Academia

The kind of AI cooperation discussed by Eliezer is not the type discussed as "program equilibrium".

Fixed, thanks.

In your research, did you fail to find previous academic work for some elements of the sequences?

I didn't look very hard. I merely thought about the stuff I already knew about, and then picked a subset of those things to list here.

Because the distinction between main and discussion is really confusing.

Because it's only a discussion.

HI WE'RE AN INTERNET MEETUP GROUP!

The internet is low status due to the low barriers to entry. Mention higher status things than the internet.
Future of Humanity Institute at Oxford (Hey I've heard of Oxford)
Vinge (a published author many have heard of)
Center for Applied Rationality which does real things in real life with real people

Talk about cognitive science. Talk about economics. Talk about anything but the internet. LessWrong? Oh it's just for coordinating all the interesting people who are interested in these interesting things.

I'd like to see at least some work on how to talk about LW without implying insularity.

Name-drop like a motha...

• "Did you read Predictably Irrational by Dan Ariely or Thinking Fast and Slow by Nobel laureate Daniel Kahneman? We study their field of predictable human thinking errors and try to figure out how best to apply those lessons to everyday human life so that we can learn how to make decisions that are more likely to achieve our goals.

• "We talk some about Alan Turing's idea that machines could one day become smarter than humans, and how shortly thereafter we might expect them to become more powerful than humans. One of the mathematicians who worked with him to crack the German Enigma Code, I.J. Good, explained that a smarter-than-human machine could use its intelligence to improve its own inteligence. And since neuroscientists like Paul Glimcher at NYU and Kent Berridge at U Michigan are learning that what humans care about is incredibly complex, it's unlikely that we'll be able to figure out how to program smarter-than-human machines to respect every little detail of what we care about."

I don't think there's much value in pretending that issues like God (and the absence thereof) or the compatibility between determinism and (any logically coherent view of) free will haven't been decisively answered.

Seriously now, the compatibility between free will and determinism is something that I was figuring out by myself back in junior high. Eliezer with his "Thou Art Physics" expressed it better and more compactly than I ever did to myself (I was instead using imagery of the style "we're the stories that write themselves", and this was largely inspired by Tolkien's Ainulindale, where the various gods sing a creation song that predicts all their future behaviour), but the gist is really obvious once you get rid of the assumption that determinism and free will must somehow be opposed.

In every discussion I've had since, in any forum, nobody who thinks them to be incompatible can describe even vaguely what "free will" would be supposed to look like if it does not contain determinism inside it.

I

Thanks for bothering to give "lowly" positive feedback! :)

Not a source, but definitely a parallel.

We have talked about Newcomb's problem with transparent boxes on here a few times - I'm pretty sure that's originally from Good and Real.

Yep. Gloriously lucid and quite readable book.
Encapsulates good chunks of the sequences.

Much more accessible than I had anticipated.

Hayakawa's Language In Thought and Action

...is not what I'd call "mainstream academia." Its program of "general semantics" is instead what Martin Gardner labeled as "cultism and pseudo-science" in one chapter of Fads and Fallacies in the Name of Science. Despite this, Language in Thought and Action is pretty good.

Could you connect them to the sequences like Luke did please? To the extent that I am familiar with your list, I'm having a hard time seeing it.

I'm actually moderately alarmed by the level at which their comments are upvoted (with all the bold face and whatnot). This event potentially adds motivation to the "Don't let users with low Karma upvote things" change.

Owing to my low karma, I can't respond directly to DevilWorm's comment. But this is for you, my friend.

DevilWorm, I don't like to judge a person too harshly. I acknowledge that in principle your trolling has the potential to make a positive contribution to the community. It would be cruel to just say "go back to troll school until you measure up". But to be honest, I do have my doubts about whether you have what it takes. To be brutally honest, the world only needs good trolls and bad trolls, and you are neither.

Bad trolls (in the good sense of bad) are the ones who are born to trolldom. They are the griefers, the naturally vindictive spirits. But among genuinely evil trolls, you wouldn't last a day. If you enrolled in the Sith Academy's trolling masterclass, they would end up using you for target practice.

Good trolls have trolldom thrust upon them. Through a mysterious mutant strain in their makeup, they see something that others cannot, and they are driven to the status of troll just by bearing witness to truth. But you are that sad case of someone who mistakes their own blind spot for superior vision: you think you can see that there is nothing to see here.

Finally we have the ugly trolls, the wannabes, the ones who aspire to trolldom. My advice is, give it up. Don't use boldface, don't use italics, don't call people semiliterate when you don't know where to place an apostrophe yourself. Be humble, accept your station in life as just another commenter who occasionally makes a valid point. It's better to be an ordinary person who makes a small but genuine contribution, than a blowhard who contributes nothing at all.

Your concerned friend, metatroll

the OP is vastly overstating how much of the Sequences are similar to the standard stuff out there... I think Luke is being extremely charitable in his construal of what's "already" been done in academia

Do you have a Greasemonkey script that rips all the qualifying words out of my post, or something? I said things like:

• "Eliezer's posts on evolution mostly cover material you can find in any good evolutionary biology textbook"
• "much of the Quantum Physics sequence can be found in quantum physics textbooks"
• "Eliezer's metaethics sequences includes dozens of lemmas previously discussed by philosophers"
• "Eliezer's free will mini-sequence includes coverage of topics not usually mentioned when philosophers discuss free will (e.g. Judea Pearl's work on causality), but the conclusion is standard compatibilism."
• "[Eliezer's posts] suggest that many philosophical problems can be dissolved into inquiries into the cognitive mechanisms that produce them, as also discussed in"
• "[Eliezer's posts] make the point that value is complex, a topic explored in more detail in..."

Your comment above seems to be reacting to a different post that I didn't write, one that includes (false) claims like: "The motivations, the arguments by which things are pinned down, the exact form of the conclusions are mostly the same between The Sequences and previous work in mainstream academia."

I have yet to encounter anyone who thinks the Sequences are more original than they are.

Really? This is the default reaction I encounter. Notice that when the user 'Thomas' below tried to name just two things he thought were original with you, he got both of them wrong.

Here's a report of my experiences:

• People have been talking about TDT for years but nobody seems to have noticed Spohn until HamletHenna and I independently stumbled on him this summer.

• I do find it hard to interpret the metaethics sequence, so I'm not sure I grok everything you're trying to say there. Maybe you can explain it to me sometime. In any case, when it comes to the pieces of it that can be found elsewhere, I almost never encounter anyone who knows their earlier counterparts in (e.g.) Railton & Jackson — unless I'm speaking to someone who has studied metaethics before, like Carl.

• A sizable minority of people I talk to about dissolving questions are familiar with the logical positivists, but almost none of them are familiar with the recent cogsci-informed stuff, like Shafir (1998) or Talbot (2009).

• As I recall, Less Wrong had never mentioned the field of "Bayesian epistemology" until my first post, The Neglected Virtue of Scholarship.

• Here's a specific story. I once told Anna that once I read about intelligence explosion I understood right away that it would be disastrous by default, because human values are incredibly complex. She seemed surprised and a bit suspicious and said "Why, had you read Joshua Greene?" I said "Sure, but he's just one tip of a very large iceberg of philosophical and scientific work demonstrating the complexity of value. I was convinced of the complexity of value long ago by metaethics and moral psychology in general."

Several of these citations are from after the originals were written! Why not (falsely) claim that academia is just agreeing with the Sequences, instead?

Let's look at them more closely:

• Lots of cited textbooks were written after the Sequences, because I wanted to point people to up-to-date sources, but of course they mostly summarize results that are a decade old or older. This includes books like Glimcher (2010) and Dolan & Sharot (2011).

• Batson (2011) is a summary of Batson's life's work on altruism in humans, almost all of which was published prior to the Sequences.

• Spohn (2012) is just an update to Spohn's pre-Sequences on work on his TDT-ish decision theory, included for completeness.

• Talbot (2009) is the only one I see that is almost entirely composed of content that originates after the Sequences, and it too was included for completeness immediately after another work written before the Sequences: Sharif (1998).

I don't understand what the purpose of this post was supposed to be - what positive consequence it was supposed to have.

That's too bad, since I answered this question at the top of the post. I am trying to counteract these three effects:

1. Some readers will mistakenly think that common Less Wrong views are more parochial than they really are.
2. Some readers will mistakenly think Eliezer's Sequences are more original than they really are.
3. If readers want to know more about the topic of a given article, it will be more difficult for them to find the related works in academia than if those works had been cited in Eliezer's article.

I find problem #1 to be very common, and a contributor to the harmful, false, and popular idea that Less Wrong is a phyg. I've been in many conversations in which (1) someone starts out talking as though Less Wrong views are parochial and weird, and then (2) I explain the mainstream work behind or similar to every point they raise as parochial and weird, and then (3) after this happens 5 times in a row they seem kind of embarrassed and try to pretend like they never said things suggesting that Less Wrong views are parochial and weird, and ask me to email them some non-LW works on these subjects.

Problem #2 is common (see the first part of this comment), and seems to lead to phygish hero worship, as has been pointed out before.

Problem #3, I should think, is uncontroversial. Many of your posts have citations to related work, most of them do not (as is standard practice in the blogosphere), and like I said I don't think it would have been a good idea for you to spend time digging up citations instead of writing the next blog post.

writing something that predictably causes some readers to get the impression that ideas presented within the Sequences are just redoing the work of other academics, so that they predictably tweet ...I do not think the creation of this misunderstanding benefits anyone

Predictable misunderstandings are the default outcome of almost anything 100+ people read. There's always a trade-off between maximal clarity, readability, and other factors. But, I'm happy to tweak my original post to try to counteract this specific misunderstanding. I've added the line: "(edit: probably most of their content is original)".

[Further reading, I would guess] gave Luke an epiphany he's trying to share - there's a whole world out there, not just LW the way I first thought.

Remember that I came to LW with a philosophy and cogsci (especially rationality) background, and had been blogging about biases and metaethics and probability theory and so on at CommonSenseAtheism.com for years prior to encountering LW.

I get what this is trying to do. There's a spirit in LW which really is a spirit that exists in many other places, you can get it from Feynman, Hofstadter, the better class of science fiction, Tooby and Cosmides, many beautiful papers that were truly written to explain things as simply as possible, the same place I got it.

That is definitely not the spirit of my post. If you'll recall, I once told you that if all human writing were about to be destroyed except for one book of our choosing, I'd go with The Sequences. You can't get the kind of thing that CFAR is doing solely from Feynman, Kahneman, Stanovich, etc. And you can't get FAI solely from Good, Minsky, and Wallach — not even close. Again, I get the sense you're reacting to a post with different phrasing than the one I actually wrote.

So they won't actually read the literature and find out for themselves that it's not what they've already read.

Most people won't read the literature either you or I link to. But many people will, like Wei Dai.

Case in point: Remember Benja's recent post on UDT that you praised as "Original scientific research on saving the world"? Benja himself wrote that the idea for that post clicked for him as a result of reading one of the papers on logical uncertainty I linked to from So You Want to Save the World.

Most people won't read my references. But some of those who do will go on to make a sizable difference as a result. And that is one of the reasons I cite so many related works, even if they're not perfectly identical to the thing me or somebody else is doing.

With both your comment here and your comments on the troll-fee issue I've found you coming across as arrogant. This perception seems to roughly match the response that other people have had to those comments as well, since most people disagreed with you in both areas (judging by number of upvotes). I hadn't perceived you that way before now, so I'm wondering if something happened to you recently that's altered the way you post or the way you think. This change is for the worse; I want my old model of Eliezer Yudkowsky back!

Frankly, I have found the sequences to be primarily useful for condensing concepts that I already had inside my head. The ideas expressed in almost all of the sequences are blatantly obvious, but they come across as catchy and often are reducible to a quick phrase. Their value lies in the fact that they make it easy to internalize certain ideas so that they're more readily accessible to me. They also helped clarify the boundaries of some concepts, to a certain extent. The sequences have provided me with a useful terminology, but I don't think they've offered me much else.

What ideas do you believe to be original that you've produced?

Is there a reason that defending the originality of the sequences is so important to you?

One anecdote given the 'PR' worries raised:

I have never read the sequences. After reading Luke's post, I am much less likely to: the impression given is the sequences are generally idiosyncratic takes which recapitulate an already existing and better organized literature. I also think it is more likely the sequences are overrated, either through readers being unaware their (or similar) insights have already been made, or lacking the technical background to critique them.

It also downgraded my estimate of the value of EY's work. Although I was pretty sceptical, I knew there was at least some chance that the sequences really were bursting with new insights and that LW really was streets ahead of mainstream academia. This now seems much less likely - although I don't think EY is a plagiarist, it seems most of the sequences aren't breaking new ground, but summarizing/unwittingly recapitulating insights that have already been made and taken further elsewhere.

So I can see the motivation for EY to defend that their originality: his stock goes down if the sequences are neat summaries but nothing that new rather than bursting with new and important insights, and EY's stock is important for things like donations, public perception of him and the SI, etc. (Both my likelihood of donating and my regard for SI has been lowered a bit by this post and comments). However, EY's way of responding to (weakly implied) criticism with catty arrogance compounds the harm.

I hadn't expected you to disagree with that tweet, so I'm clearly getting something wrong. I wrote that in the hope that it would encourage people to read the Sequences, not put them off - I think people imagine it as this million-word work of revelation, but a very large part of what it is is a work of popular science - turning people on to good existing ideas in psychology and philosophy and biology and physics and suchlike. There is a great deal that is original and valuable in there, but I don't think of it as the majority of the material.

I agree that Luke's post might cause some people to update too much in the direction of "the Sequences aren't original". He was wrong or overstated things in the couple of bullet points that I checked out (and pointed out in my earlier comment). He probably should have showed it to you for error-checking and making sure it's being fair before posting it.

I do think having an index of related works is very valuable, for people wanting to do further readings, or figuring out exactly which parts of the Sequences are original.

So they won't actually read the literature and find out for themselves that it's not what they've already read.

I read Spohn right away, and I'm at least planning to read some of the other references. But I'm not sure how typical I am in this regard.

There's literature out there which is written in the same spirit as LW, but with different content. Now that's an exciting message. It might even get people to read things.

Maybe we can start to build up a repository of those things, too. So far, you've recommended:

• Language in Thought and Action
• Psychological Foundations of Culture
• Good and Real
• Rational Choice in an Uncertain World

Unfortunately, those works seem incredibly different to me, so it's hard for me to guess which other works you would also endorse as being in the "LW spirit." I'll try anyway:

• Many of the readings for Aaronson's class Philosophy and Theoretical Computer Science
• Several works by Paul Almond
• Much of Overcoming Bias
• Several blogs of other LWers, e.g. Yvain's blog

There's a spirit in LW which really is a spirit that exists in many other places,

Yes, and pointing out those other places here serves two purposes.

It serves to brand LW, so that people passing by can quickly see the kind of spirit here. Yes, there's a whole world out there, and many of us have spent some time in it, so seeing references to that world here serves to quickly communicate some of what LW is about.

References also server to point people here to other expositions of similar material.

For example, you say:

The only real way for people to learn better is to go out and read Language in Thought and Action

I'd recommend people at some point move on from Hayakawa to Korzybski, Science and Sanity, and the whole General Semantics literature. People have spent decades discussing these issues and organizing

It doesn't get people to read the literature. Why should they? From what they can see, it's already been presented to them on LW, after all.

That's not my reaction to references. When I first came here, the references to Jaynes didn't make me think "I've already covered this stuff, no need to read this web site." On the contrary, it made me want to read more. Similarly, seeing a reference to other work associated with a sequence wouldn't make me think "no need to bother reading that, EY has already regurgitated it for me", it would make me want to read the original.

Yes, there's a whole world out there, making it easier to navigate that world with links is a good thing.

TDT doesn't artifically sever decision nodes from anything upstream; the idea is that observing your algorithm, but not its output, is supposed to screen off things upstream.

Pardon me; I'm not yet much of an expert with LW decision theories. When you explained TDT on the whiteboard to Alex (with me listening), you kept talking about "severing" rather than "screening off." I'll try to find a way to modify the OP.

ETA: I remembered I have a recording of that tutorial, and I when checked the recording, and it turns out my memory was wrong. You did talk about how TDT "screened off" the information whereas CDT "severs" the causal diagram.

I don't understand what the purpose of this post was supposed to be - what positive consequence it was supposed to have.

I took the post to be Luke writing notes to himself, in public so as to recruit others' help, toward the kind of bibliography that might be included in an academically acceptable version of the Sequences, or of some parts of them.

The intention being, I gathered, to publish these bibliographies as an adjunct to the Sequences - perhaps in the "wall of references" style of Luke's early posts. (If so, I hope a more user-friendly way of displaying those is worked out first!)

(ETA: the specific positive consequence of that would be to help the reader "find the related works in academia" as per Luke's third numbered point in the OP.)

Thankyou for clearing that up. Given your occupational affiliation with Luke I had been overestimating the extent to which you would endorse his position. That is, I wouldn't have expected Luke to write this without checking with you first so thought you must have agreed.

Regardless of whether it's original, you're the one making rationality popular. Inspiring this many people to take more interest in rationality is a profoundly worthwhile accomplishment. The world needs teachers who can motivate them to think more clearly. I'm heartened to see your progress.

The only real way for people to learn better is to go out and read Language in Thought and Action or Rational Choice in an Uncertain World.

Added to my list!

Do you have any more reading suggestions for people who have read the sequences? I read a few books recommended in the book recommendation open threads (or on irc), but was sometimes disappointed ("Thinking in Systems" is not very rigorous and formal, Nassim Taleb's "Fooled by Randomness" takes too much liberty interpreting various concepts, and I'm not a fan of books that start by telling me "don't worry I won't hurt your little brain with equations").

Great minds think alike?

They're compiling a book-format edition of the Sequences, and there's quite a bit of work into an alternate pop-sci edition.

Assuming we disregard ownership of multiple accounts, the consequent of your conditional is impossible.

It is easy to see why DevilWorm wouldn't disregard ownership of multiple accounts as a possibility.

Eliezer credits Nick Bostrom with coming up with the idea of Friendly AI first (and indeed while Eliezer was indifferent to AI risk on the assumption that either superintelligences would be automatically supermoral or it didn't matter what happened). FOOM probably goes to I.J. Good, or SF (Eliezer found out about the idea of a technological singularity by reading Vernor Vinge's science fiction, and closely related ideas are decades older in SF).

FOOM (AKA Intelligence Explosion) was formulated by I.J. Good about 50 years ago.