Self-skepticism: the first principle of rationality

I agree- the answer given in the FAQ isn't a complete and valid response to the critics of the Singularity. But it was never meant to be; it was meant to be "short answers to common questions." The SI's longer responses to critics of the Singularity are mostly in peer-reviewed research; for example, in:

Luke Muehlhauser and Anna Salamon (2012). Intelligence Explosion: Evidence and Import. In The Singularity Hypothesis, Springer. (http://singularity.org/files/IE-EI.pdf)

Carl Shulman and Nick Bostrom (2012). How Hard is Artificial Intellience?. In Journal of Consciousness Studies, Imprint Academic. (http://www.nickbostrom.com/aievolution.pdf)

Chalmers, D. (2010). "The Singularity: A Philosophical Analysis." Journal of Consciousness Studies 17:7-65. (http://consc.net/papers/singularity.pdf)

Sotala, Kaj (2012) Advantages of Artificial Intelligences, Uploads, and Digital Minds. International Journal of Machine Consciousness 4 (1), 275-291. ( http://kajsotala.fi/Papers/DigitalAdvantages.pdf )

Of course, now I feel pretty bad for linking you to several hundred pages of arguments- which are often overlapping and repetitive, and which still don't represent everything the SI has written on the subject (or even a majority, I think). If you have any specific criticisms of the SI's ideas, it might be faster for you to post them here.

I feel even (slightly) worse, because none of the SI's arguments have reached the level of evidence you're comparing it to- eg Givewell's analyses, and the disproofs of spoon-bending and new reading methods. But the SI isn't capable of providing evidence that strong, because whether or not its claims were accurate, they would still be predictions of an abrupt future change, as opposed to claims about the efficacy of past actions. I do think that, where possible, the SI has tried to be very transparent- for example, in my opinion the SI's last yearly progress report was around as thorough as Givewell's last yearly progress reports- part 1, part 2, part 3, part 4.

(On a side note, it might interest you that Holden Karnofsky, co-founder of Givewell, also analyzed the SI and came to a very negative conclusion- posted here. His post is currently the most upvoted post of all time on LessWrong.)

Was the main post edited? The comments seem entirely disconnected from the article.

It might be hard at first to tell the difference, so I'm going to have to use some examples. I'd ask that you try and suspend any emotional reactions you have to the examples I chose and just look at which approach seems more rational.

Bullshit. You aren't providing an example because it is "hard to tell the difference at first". You started with an intent to associate SIAI with self delusion and then tried to find a way to package it as some kind of rationality related general point.

Contrast this with the Singularity Institute. A skeptic might well ask whether the Singularity is actually going to occur. Well, the SIAI FAQ addresses this, but only to summarily dismiss a couple objections in a cursory paragraph (that evades most of the force of the objections). And that's the closest the FAQ gets to any sort of skepticism, the rest of it is just a straight and confident summary that tries to persuade you of SIAI beliefs.

The FAQ on the website is not the place to signal humility and argue against your own conclusions. All that would demonstrate is naivety and incompetence. You are demanding something that should not exist. This isn't to say that there aren't valid criticisms to be made of SIAI and their FAQ. You just haven't made them.

Which attitude seems more like a serious scientist? Which seems more like Uri Geller?

Am I the only person who is outright nauseated by the quality of reasoning in these recent mud-slinging posts by aaronsw? What I see is a hastily selected bottom line along the lines of "SingInst sux" or perhaps "SingInst folks are too arrogant" then whatever hastily conceived rhetoric he can think of to support it. The problem isn't in the conclusions---it is that the arguments used either don't support or outright undermine the conclusion.

Competent criticism is encouraged. But the mere fact that a post is intended to be critical or 'cynical' isn't sufficient. It needs to meet some kind of minimum intellectual standard too. If it did not represent an appeal to the second-order-contrarians and was evaluated based on actual content this post would probably end up mildly negative, even in the discussion section.

Reply to revised OP: Policy debates should not appear one-sided (because there's no systematic reason why good policies should have only positive consequences) but epistemic debates are frequently normatively one-sided (because if your picture of likelihood ratios is at all well-calibrated, your updates should trend in a particular direction and you should almost never find strong evidence on more than one side of an epistemic debate; "strong evidence" by Bayesian definition is just that sort of evidence which we almost never see when the hypothesis is wrong).

In my own experience, self skepticism isn't sufficient. It's bloody useful of course, but it's also an exceptional time sink -- occasionally to the point where I'll forget to actually think of solutions to the problem.

Does anyone have any algorithms they use to balance self-skepticism with actually solving the problem?

Responding to the new version of this article, I'll observe that the intent and competence of the Cochrane vs. Heritage folks seems to suggest:

CC signal intellectual honesty and are willing to sacrifice their impact on the beliefs of the masses, expecting in compensation to have better academic reputations, and perhaps influence a few more rationalists.

Heritage signal "don't oppose us or we'll embarrass you" and "our side is going to win. don't even think about switching" and perhaps "we all understand that politics is war. we'll be happy to share our true beliefs with you at the victory orgy, once you've proven your loyalty".

You can tell almost nothing about their competence or rationality without checking the work of the Cochrane analysis (this is straightforward but costly), or by judging the effectiveness of Heritage propaganda, compared to what you'd expect given their goals and a given level of hidden rationality and intelligence (this is tricky).

outside tests showed the new techniques only caused reading scores to go down.

The Feynman article you link says they didn't test them, not that they tested them and failed.

Related posts: The Proper Use of Humility and How To Be More Confident... That You're Wrong.

BTW, what justifies calling self-skepticism "the first principle of rationality"? Feynman called it that, but Eliezer doesn't seem to think that self-skepticism is as important as, for example, having something to protect.

I agree with the general principle that self-skepticism is vital, as a personality trait, for rationalism. However, I don't think that a single written work necessarily gives very much evidence of its author's self-skepticism; it's common to make lots of objections and fail to put them in writing, or to put them in different places. I have also noticed that objections to futurist and singularitarian topics, moreso than other things, tend to lead into clouds of ambiguity and uncertainty which cannot be resolved in either direction, which don't suggest such straightforward approaches as surveying people to find out about nets in use, and which take a lot of words to adequately deal with. (This last issue is especially relevant to your particular example; FAQs are a bad medium for long arguments, and while some links would be nice, their absence is not very informative.)

...I really should have listened to that inner voice that said "Don't waste your time!"

The reason I used the term "Dark Arts" is that your post cleverly generalizes from unknown examples, infers this to be true for an overwhelming majority of cases, and then proposes this as a Fully General Counterargument that any LWer attempting to revise themselves is merely plugging in "self-doubt" while generalizing.

Your argument effectively proves by axiom that there are very few LWers if any actually using real epistemic rationality skills, and by hidden connotation also shows that we're all Gray, and thus because Gray isn't White our thoughts and work aren't worth any more than those of any random Internet user.

I'm somewhat impressed at how steeply we were downvoted for this thread, though. Perhaps we were instantly pattern-matched to "Troll" and "Naive guy replying to troll" in some way?

I hope SI will agree that the FAQ answer you linked is inadequate (either overlooking some common objections, or lumping them together dismissively as unspecified obstacles that will be revealed in the future). For example, "building an AI seems hard. no human (even given much longer lifespans) or team of humans will ever be smart enough to build something that leads to an intelligence explosion", or "computing devices that can realistically model an entire human brain (even taking shortcuts on parts that turn out to be irrelevant to intelligence) will be prohibitively expensive and slow" are both plausible.

And yes, even if the answer is improved, it does suggest a possible pattern. It could just be a lack of resources available to create high quality, comprehensive answers to objections. Or it could be that SI is slightly more like Uri Geller in not doubting itself than GiveWell is.

Is GiveWell really doubting itself or its premise - that it's worth spending extra money evaluating where to give money? (actually, I think it is worth it, but that's not my point).

A TED Talk centred on this subject came out today: Dare to Disagree

Please don't use Dark Arts and Fully General Counterarguments while trolling. Someone might mistake you for a serious poster that just didn't read the sequences.

[WARNING: Industrial-strength mind-killing here, tread carefully]

Edit: Well, it was. Post appears to have been heavily edited.