A cynical explanation for why rationalists worry about FAI

Let's take the outside view for a second. After all, if you want to save the planet from AIs, you have to do a lot of thinking! You have to learn all sorts of stuff and prove it and just generally solve a lot of eye-crossing philosophy problems which just read like slippery bullshit. But if you want to save the planet from asteroids, you can conveniently do the whole thing without ever leaving your own field and applying all the existing engineering and astronomy techniques. Why, you even found a justification for NASA continuing to exist (and larding out pork all over the country) and better yet, for the nuclear weapons program to be funded even more (after all, what do you think you'll be doing when the Shuttle gets there?).

Obviously, this isn't any sort of proof that anti-asteroid programs are worthless self-interested rent-seeking government pork.

But it sure does seem suspicious that continuing business as usual to the tune of billions can save the entire species from certain doom.

I agree with the gist of this (Robin Hanson expressed similar worries), though it's a bit of a caricature. For example:

people who really like to spend their time arguing about ideas on the Internet have managed to persuade themselves that they can save the entire species from certain doom just by arguing about ideas on the Internet

... is a bit unfair, I don't think most SIAI folk consider "arguing about ideas on the Internet" to be of much help except for recruitment, raising funds, and occasionally solving specific technical problems (like some decision theory stuff). It's just that the "arguing about ideas on the Internet" is a bit more prominent because, well, it's on the Internet :)

Eliezer, specifically, doesn't seem to do much arguing on the internet, though he did do a good deal of explaining his ideas on the Internet, which more thinkers should do. And I don't think many of us folks who chat about interesting things on LessWrong are under any illusion that doing so is Helping Save Mankind From Impending Doom.

Aaron, I currently place you in the category of "unconstructive critic of SI" (there are constructive critics). Unlike some unconstructive critics, I think you're capable of more, but I'm finding it a little hard to pin down what your criticisms are, even though you've now made three top-level posts and every one of them has contained some criticism of SI or Eliezer for not being fully rational.

Something else that they have in common is that none of them just says "SI is doing this wrong". The current post says "Here is my cynical explanation for why SI is doing this thing that I say is wrong". (Robin Hanson sometimes does this - introduces a new idea, then jumps to "cynical" conclusions about humanity because they haven't already thought of the idea and adopted it - and it's very annoying.) The other two posts introduce the criticisms in the guise of offering general advice on how to be rational: "Here is a rationality mistake that people make; by coincidence, my major example involves the founder of the rationality website where I'm posting this advice."

I suggest, first of all, that if your objective on this site is to give advice about how to be rational, then you need to find a broader range of examples. People here respect Eliezer, for very good reasons. If you do want to make a concentrated critique of how he has lived his life, then make a post about that, don't disguise it as a series of generic reflections on rationality which just happen to be all about him.

Personally I would be much more interested in what you have to say about the issue of AI. Do you even think AI is a threat to the human race? If so, what do you think we should do about it?

Obviously this isn't any sort of proof that working on FAI is irrational, but it does seem awfully suspicious that people who really like to spend their time thinking about ideas have managed to persuade themselves that they can save the entire species from certain doom just by thinking about ideas.

For what it's worth, I would personally be much happier if I didn't have to worry about FAI and could just do stuff that I found the most enjoyable. I also don't think that the work I do for SI has a very high chance of actually saving the world, though it's better than doing nothing.

I do consider the Singularity Institute a great employer, though, and it provided me a source of income at a time when I was desperately starting to need one. But that happened long after I'd already developed an interest in these matters.

Obviously one possibility (the inside view) is simply that rationality compels you to focus on FAI. But if we take the outside view for a second, it does seem like FAI has a special attraction for armchair rationalists: it's the rare heroic act that can be accomplished without ever confronting reality.

After all, if you want to save the planet from an asteroid, you have to do a lot of work! You have to build stuff and test it and just generally solve a lot of gritty engineering problems. But if you want to save the planet from AI, you can conveniently do the whole thing without getting out of bed.

What is the "outside view" on how much of an existential risk asteroids are? You know, the one you get when you look at how often asteroid impacts at or near the level that can cause mass extinctions happen? Answer: very damn low.

"The Outside View" isn't just a slogan you can chant to automatically win an argument. Despite the observational evidence from common usage the phrase doesn't mean "Wow! You guys who disagree with me are nerds. Sophisticated people think like I do. If you want to be cool you should agree with me to". No, you actually have to look at what the outside view suggests and apply it consistently to your own thinking. In this post you are clearly not doing so.

After all, if you want to save the planet from an asteroid, you have to do a lot of work!

Something being difficult (or implausible) is actually a good reason not to do it (on the margin).

You have to build stuff and test it and just generally solve a lot of gritty engineering problems. But if you want to save the planet from AI, you can conveniently do the whole thing without getting out of bed.

What the? Where on earth are you getting the idea that building an FAI isn't hard work? Or that it doesn't require building stuff and solving gritty engineering problems?

Indeed, as the Tool AI debate as shown, SIAI types have withdrawn from reality even further.

This is an unsubstantiated assertion presented with the form of something that should be conclusive. This is bizarre since the SIAI position on Tool AI is not a particular weak point in the SIAI position and the referenced conversation doesn't indicate any withdrawal from reality.

The actual people at SIAI are much less prone to this than the community.

When I was living in San Francisco, people would regularly discuss various experiments that they were running on themselves, or skills that they were practicing. If I tried to assert something without concrete examples or predictions, people would be skeptical.

OK, I believe we have more than enough information to consider him identified now:

• Dmytry
• private_messaging
• JaneQ
• Comment
• Shrink
• All_work_and_no_play

Those are the currently known sockpuppets of Dmytry. This one warrants no further benefit of the doubt. It is a known troll wilfully abusing the system. To put it mildly, this is something I would prefer not to see encouraged.

You are using testosterone to boost performance, it also clouds social judgment severely, and in so much as I know it, I can use it to dismiss literally anything you say (hard to resist temptation to, at times).

Counter incremented.

People at the upper end of the IQ spectrum get lonely for someone smarter to talk to. It's an emotional need for an intellectual peer.

My cynical take on FAI is that it's a goal to organize one's life around which isn't already in the hands of professionals. I have no idea whether this is fair.

But if we take the outside view for a second, it does seem like FAI has a special attraction for armchair rationalists: it's the rare heroic act that can be accomplished without ever confronting reality.

IMO, a signalling explanation makes more sense. Publicly-expressed concern about moral issues signals to others what a fine fellow you are. In that context the more far-out things you care about the better. Trees, whales, simulated people, distant descendants - they all signal how much you care.

Obviously one possibility (the inside view) is simply that rationality compels you to focus on FAI. But if we take the outside view for a second, it does seem like FAI has a special attraction for armchair rationalists: it's the rare heroic act that can be accomplished without ever confronting reality.

I think the last sentence here is a big leap. Why is this a more plausible explanation then the idea that aspiring rationalist simply find AI-risk and FAI compelling. Furthermore, since this community was founded by someone who is deeply interested in both topics, members who are attracted to the rationality side of this community get a lot of exposure to the AI-risk side. As such, if we accept the premiss that AI-risk is a topic that aspiring rationalists are more likely to find interesting than a random member of the general public, then it's not surprising that many end up thinking/caring about it after being exposed to this community.

You seem to attempt to justify this last sentence of the quoted text with the following:

After all, if you want to save the planet from an asteroid, you have to do a lot of work! You have to build stuff and test it and just generally solve a lot of gritty engineering problems. But if you want to save the planet from AI, you can conveniently do the whole thing without getting out of bed.

I would respond to this by saying that thinking/caring about AI-risk ≠ working on AI-risk. I imagine there are also lots of people who think about the risks of asteroid impacts, but aren't working on solving them, and wouldn't claim they are. Also, this paragraph could be interpreted a saying that, people who claim to be doing work on AI-risk (e.g., SI) aren't actually doing any work. It would be one thing to claim the work is misdirected, but to claim they aren't working hard (to me) seems misinformed or disingenuous.

Which then leads into the following:

Indeed, as the Tool AI debate as shown, SIAI types have withdrawn from reality even further. There are a lot of AI researchers who spend a lot of time building models, analyzing data, and generally solving a lot of gritty engineering problems all day. But the SIAI view conveniently says this is all very dangerous and that one shouldn't even begin to try implementing anything like an AI until one has perfectly solved all of the theoretical problems first.

I think a more accurate characterization of SI's stance would be that there are lots of important philosophical and mathematical problems that if solved will increase the likely hood of a positive Singularity, and that those doing what you call the “gritty engineering” haven't properly considered the risks. Your statement seems to trivialize this work, and you state Holden's criticism as evidence. What specifically in this “debate”—including the responses from SI—lead you to believe that SI's approach is “withdrawn from reality.”

Scarcely the most cynical conceivable explanation. Here, try this one:

"Yes," declaimed Deep Thought, "I said I'd have to think about it, didn't I? And it occurs to me that running a programme like this is bound to create an enormous amount of popular publicity for the whole area of philosophy in general. Everyone's going to have their own theories about what answer I'm eventually to come up with, and who better to capitalize on that media market than you yourself? So long as you can keep disagreeing with each other violently enough and slagging each other off in the popular press, you can keep yourself on the gravy train for life. How does that sound?"

The two philosophers gaped at him.

"Bloody hell," said Majikthise, "now that is what I call thinking. Here Vroomfondel, why do we never think of things like that?"

"Dunno," said Vroomfondel in an awed whisper, "think our brains must be too highly trained, Majikthise."

So saying, they turned on their heels and walked out of the door and into a lifestyle beyond their wildest dreams.

— Douglas Adams, The Hitchhiker's Guide to the Galaxy, ch. 25

Not sure about your sampling method, but a lot of LWers I know (in NY area) are pretty busy "doing stuff". Propensity to doing stuff does not seem to negatively correlate with FAI concerns as far as I can tell.

That said this is a bit of a concern for me as a donor, which is why I think the recent increase in transparency and spinning off CFAR is a big positive sign: either the organization is going to be doing stuff in the FAI area (I consider verifiable research doing stuff, and I don't think you can do it all in bed) or not, it's going to be clear either way.

Keep in mind that real sock puppeteering is about making a strawman sock puppet, or a sock puppet that disagrees cleverly using existing argument you got but tentatively changes the view, or the like.

Sock puppets, both here and on Reddit or Wikipedia, can be used for multiple purposes, not just that.

One reason I have respect for Eliezer is HPMOR-- there's a huge amount of fan fiction, and writing something which impresses both a lot of people who like fan fiction and a lot of people who don't like fan fiction is no small achievement.

Also, it's the only story I know of which gets away with such huge shifts in emotional tone. (This may be considered a request recommendations of other comparable works.)

Furthermore, Eliezer has done a good bit to convince people to think clearly about what they're doing, and sometimes even to make useful changes in their lives as a result.

I'm less sure that he's right about FAI, but those two alone are enough to make for respect.

After all, if you want to save the planet from an asteroid, you have to do a lot of work! You have to build stuff and test it and just generally solve a lot of gritty engineering problems. But if you want to save the planet from AI, you can conveniently do the whole thing without getting out of bed.

If you want to do either, you have to do work, get paid, and donate to the appropriate charity. The only difference is where you donate.

I can't seem to find a charity for asteroid impact avoidance. As far as I can tell, everything going into that is done by governments. Even if it is, I would still expect to find lobbyists involved. Also, I wonder if it's possible to donate directly to that stuff.

it does seem like FAI has a special attraction for armchair rationalists: it's the rare heroic act that can be accomplished without ever confronting reality.

After all, if you want to save the planet from an asteroid, you have to do a lot of work! You have to build stuff and test it and just generally solve a lot of gritty engineering problems. But if you want to save the planet from AI, you can conveniently do the whole thing without getting out of bed.

What? really? I happen to have drunk the kool-aid and think this is an important problem. So it is very important to me that you tell me how it is I could solve FAI without doing any real work.

I was all prepared to vote this up from '"If Eliezer had simply been obsessed by saving the world from asteroids, would they all be focused on that?'." But then you had to go and be wrong - suggesting some sort of "lazy ideas only" search process that makes no sense historically, and conflating LW and SI.

actually, let it be the last post here, I get dragged out any time I resolve to leave then check if anyone messaged me.

A two hour self-imposed exile! I think that beats even XiXiDu's record.

LessWrong rationality nerds cared so much about creating Friendly AI

I don't!

btw, if you're aaronsw in my Twitter feed, welcome to LessWrong^2

What do you think of contemporary theoretical physics? That is also mostly "arguing on the Internet".

Ok, if what you're saying is not "SI concludes this" but just that we don't really know what even the theoretical SI concludes, then I don't disagree with that and in fact have made similar points before. (See here and here.) I guess I give Eliezer and Luke more of a pass (i.e. don't criticize them heavily based on this) because it doesn't seem like any other proponent of algorithmic information theory (for example Schmidhuber or Hutter) realizes that Solomonoff Induction may not assign most posterior probability mass to "physics sim + location" type programs, or if they do realize it, choose not to point it out. That presentation you linked to earlier is a good example of this.

I believe that Hutter et all were rightfully careful not to expect something specific, i.e. not to expect it to not kill him, not to expect it to kill him, etc etc.

You would think that if Hutter thought there's a significant chance that AIXI would kill him, he would point that out prominently so people would prioritize working on this problem or at least keep it in mind as they try to build AIXI approximations. But instead he immediately encourages people to use AIXI as a model to build AIs (in A Monte Carlo AIXI Approximation for example) without mentioning any potential dangers.

Those are questions to be, at last, formally approached.

Before you formally approach a problem (by that I assume you mean try to formally prove it one way or another), you have to think that the problem is important enough. How can we decide that, except by using intuition and heuristic/informal arguments? And in this case it seems likely that a proof would be too hard to do (AIXI is uncomputable after all) so intuition and heuristic/informal arguments may be the only things we're left with.

But an interest in rationality pulls in expertise transferable to all manner of fields, e.g. the 2011 survey result showing 56.5% agreeing with the MWI. (I certainly hope the next survey will ask how many of those saying they agree or disagree with it can solve the Schroedinger equatlon for a hydrogen atom, and also how many of those expressing a position would understand the solution to the Schroedinger equatlon for a hydrogen atom if they saw it written out.) So acquiring meaningful knowledge of artificial intelligence is par for the course.

I'm not sure about the community at large, but as for some people, like Eliezer, they have very good reasons for why working on FAI actually makes the most sense for them to do, and they've gone to great lengths to explain this. So if you want to limit your cynicism to "armchair rationalists" in the community, fine, but I certainly don't think this extends to the actual pros.

Also, I only thought of that when you would go on how I must feel so humiliated by some comment of yours

(I did not make such a claim, nor would I make one.)

SIAI types have withdrawn from reality even further. There are a lot of AI researchers who spend a lot of time building models, analyzing data, and generally solving a lot of gritty engineering problems all day. But the SIAI view conveniently says this is all very dangerous and that one shouldn't even begin to try implementing anything like an AI until one has perfectly solved all of the theoretical problems first.

I think it is more that you can't build a genuine General Intelligence before you have solved some intractable mathematical problems, like practical algorithms for solomonoff induction. The fact that you would like goal stability and human-values is just two of many difficult theoretical problems that are not going to be solved by using black-box models, nor using simple models.

Practical AI researchers do a lot of good work. But remember that a neural network is a statistical classification tool.

(I am aware that neural networks isn't the bleeding edge, but i cannot come to mind of a more resent trend. My point stands that AI is fiendishly hard and you are not going to be able to build anything concise and efficient by solving gritty engineering problems. This is even visible in simple programming where the mathematically elegant functional programming languages are gaining on the mainstream 'gritty engineering' imperative languages.)

I would expect it to also attract unusually high percentage of narcissists .

Suppose there was a suspicion 2..3 people with particularly strong view just decided to pick on your account to downvote? (from back when it was Dmytry) How do you actually check that?

Or, you know, get over it. It's just karma!

A small number of people just downvote what ever I post, and this is especially damaging to any of the good posts.

Assuming the latter exist.