In Wikipedia — reading about Roko's basilisk causing "nervous breakdowns" ...

post by ThurstonBT · 2021-10-13T23:12:09.190Z · LW · GW · 18 comments

I recently came upon the story of "Roko's basilisk".  Heretofore I'd not seen the topic.  Reading about it in Wikipedia[1] I saw "Discussion of Roko's basilisk was banned on LessWrong for several years because Yudkowsky had stated that it caused some readers to have nervous breakdowns".[1]

That Yudkowsky censored mention of Roko's basilisk because it "caused some readers to have nervous breakdowns" (henceforth THE CLAIM) seemed very strange to me.  So I asked in a Berkeley CA 'rationalist community' forum — "This story of censorship of Roko's basilisk and it causing "nervous breakdowns" strikes me me a bizarre;  Is this some sort of joke?".

The answers to this question, some of which were angry and emotional, amounted to:
1)  THE CLAIM is false.
2) No one could provide evidence that THE CLAIM is false sufficient to meet Wikipedia's evidentiary standards so as to rebut or remove the THE CLAIM in Wikipedia..
3) No one could provide a citation to a specific denial from Yudkowsky that THE CLAIM did not figure into the decision to censor mention of Roko's basilisk. 

In the interest of truth-seeking ...
... if the THE CLAIM is false then I'd like to obtain evidence to that effect and correct Wikipedia.  

... if THE CLAIM is true then it brings to mind some potentially unkind questions about the psychological heath of a seemingly significant portion of the 'rationality community'.

Please note that my questions here bears only on THE CLAIM (ie. "caused some readers to have nervous breakdowns") and not any of the issues associated with Roko's basilisk, eg. claims of infohazard.

Even if the "caused some readers to have nervous breakdowns" is a hyperbolic description, with no actual "nervous breakdowns" occurring, the notion that censorship needed to be employed to protect emotionally brittle persons in the 'rationality community' from a thought experiment reflects poorly on the community.




Comments sorted by top scores.

comment by Rob Bensinger (RobbBB) · 2021-10-15T06:54:52.463Z · LW(p) · GW(p)

I've spoken over PM with the person I believe Roko was talking about when he said "one person at SIAI was severely worried by this, to the point of having terrible nightmares". They say:

I'm the person Roko was referring to in his original post. I saw this thread talking about "nervous breakdowns" and wanted to post a basic clarification to make sure this isn't because people are reading something into that line that's more sensational than the reality. There have been times when I was anxious about possibilities in the space of TDT, acausal blackmail, and simulationism, not "the Basilisk" particularly, but that general sort of thing. This happened both when I was a visiting fellow at SIAI in 2009-10 and earlier. None of this amounted to anything I'd consider a "nervous breakdown". Nobody observing my behavior or internet posting, say, would have noticed much out of the ordinary. These anxieties have faded over time and haven't been an issue for the last decade or so.

Replies from: Viliam
comment by Viliam · 2021-10-17T11:39:59.561Z · LW(p) · GW(p)

Funny how one person becomes "some" becomes "a seemingly significant portion of the 'rationality community'".

(On the other hand, something in my heart rejoices in the thought that in our community, every single person is considered a significant portion.)

comment by habryka (habryka4) · 2021-10-14T00:00:38.376Z · LW(p) · GW(p)

I think the claim is false, though it's hard to prove an absence. I have never heard of anyone having a particularly strong emotional reaction to Roko's basilisk (outside of a "sneering" reaction towards rationalists or something), and almost everyone I've met in the community thinks the thought experiment is not particularly interesting or important or dangerous. I do like the general policy of not talking a ton about infohazards of this type, but that's just a broad reference class, of which Roko's basilisk seems like a non-central example.

The Roko's Basilisk [? · GW] wiki article has a lot of detail, and I recommend looking through it. It also includes these paragraphs: 

Other sources have repeated the claim that Less Wrong users think Roko's basilisk is a serious concern. However, none of these sources have yet cited supporting evidence on this point, aside from Less Wrong moderation activity itself. (The ban, of course, didn't make it easy to collect good information.)

Less Wrong user Gwern reports that "Only a few LWers seem to take the basilisk very seriously," adding, "It's funny how everyone seems to know all about who is affected by the Basilisk and how exactly, when they don't know any such people and they're talking to counterexamples to their confident claims."

comment by Richard_Kennaway · 2021-10-17T06:08:54.972Z · LW(p) · GW(p)

Given the latest post [LW · GW] about not just Leverage but also CFAR and MIRI, the Wikipedia article now seems to me sufficiently accurate as it stands.

comment by Linch · 2021-10-15T12:14:32.082Z · LW(p) · GW(p)

... if THE CLAIM is true then it brings to mind some potentially unkind questions about the psychological heath of a seemingly significant portion of the 'rationality community'.

So I think we have much stronger evidence of psychological health issues with the rationality community (which I assume is the same thing as the 'rationality community' though I'm uncertain) via things like the LW and SSC surveys. Perhaps you do not trust surveys because of self-report issues? But in that case I'd probably look at proxies like common correlates of mental health issues, or maybe suicide rates within the community and see if it's above baseline. 

I'm personally pretty convinced that psychological issues in the rationalist community is substantially above baseline[1], but I would not want to trust a fairly complex causal assignment ascribed to a specific cause, that likely has politically motivated reporting, above more mundane instruments like surveys and other observational data. 

[1] The main source of hesitation for me is uncertainty about the baseline; it's plausible but unlikely to me that "normal people" are very crazy in the clinical sense but for some reason this is mostly unreported. 

Replies from: ThurstonBT
comment by ThurstonBT · 2021-10-15T21:55:54.437Z · LW(p) · GW(p)

@Linch: My observations (based on an admittedly limited set of observations and my lack of psychological training) agree with your "I'm personally pretty convinced that psychological issues in the rationalist community is substantially above baseline.

I'm surprised, given the claimed truth-seeking and evidentiary rigor values of 'the rationalist community', that there is not a magiteral data-laden LessWrong essay that addresses "psychological issues in the rationalist community" that is cited when discussion turns to this topic.  Can anyone point to such an essay?


Replies from: Linch
comment by Linch · 2021-10-17T05:31:05.651Z · LW(p) · GW(p)

What does "magiteral" mean here? 

At any rate, you're free to be the change you want to see in the world. :)

It was a dirty job, he thought, but somebody had to do it. 

As he walked away, he wondered who that somebody might be.


comment by Richard_Kennaway · 2021-10-15T13:25:25.385Z · LW(p) · GW(p)

There's no way to give a citation to Eliezer not saying it. You might get further querying the citations in the article for the claim that he did say it. There are three, but the first does not mention it and the third calls it a rumour. The second quotes a chunk of Eliezer's immediate response to Roko, the one that began "Listen to me very closely, you idiot" (which I remember reading at the time), and then says "Yudkowsky said that Roko had already given nightmares to several LessWrong users and had brought them to the point of breakdown." My memory is not detailed enough to say whether he said that, but I think it quite likely that someone did say something of the sort back then.

If you search LessWrong for "Listen to me very closely, you idiot", there is a hit on that comment by Eliezer, but the comment itself has been deleted. Since there is that hit, the comment is presumably still in the database. Perhaps the LessWrong admins might be willing to give you sight of it? But the result either way wouldn't be a citable source for Wikipedia.

comment by ChristianKl · 2021-10-15T12:03:58.077Z · LW(p) · GW(p)

The argument for which EY banned the basilisk is a very complex argument and you need to understand TDT to follow it. Newspapers are in the business of dumbing things down for their readers so that their readers get the sense they understand what goes on. As a result you won't find a good explanation of why EY banned it in any newspaper. As a result you won't find a good explanation in anything that's seen as worthy to be a Wikipedia source.

Replies from: RobbBB
comment by Rob Bensinger (RobbBB) · 2021-10-15T17:32:50.884Z · LW(p) · GW(p)

IIRC Tom Chivers' book explains the thing OK, and is citeable on Wikipedia.

Replies from: ChristianKl
comment by ChristianKl · 2021-10-15T18:36:06.224Z · LW(p) · GW(p)

I haven't read it. It might be a good source for someone writing the Wikipedia article.

comment by bn22 · 2021-10-14T00:36:34.989Z · LW(p) · GW(p)

To my knowledge this claim seems to be almost entirely fabricated as the only text that is vaguely reminiscent of this claim in the original thread is a claim from Roko that “One might think that the possibility of CEV punishing people couldn’t possibly be taken seriously enough by anyone to actually motivate them. But in fact one person at SIAI was severely worried by this, to the point of having terrible nightmares, though ve wishes to remain anonymous.” which along with describing the experience of a grand total of one person only refers to anxiety resulting from an idea related to the premises of the thought experiment and not the thought experiment itself.

comment by Shouperfluous · 2021-10-14T00:28:07.424Z · LW(p) · GW(p)

The "Berkeley, CA based Rationalist Community Forum" sent you to this a few times [LW · GW]. In which EY stated that the infohazard itself, not "emotional brittleness" was the cause. This included a direct statement from EY, whereas the links and sources that suggested "emotional brittleness" were sourced as "rumors."

Be careful of DDCT!

Replies from: ThurstonBT
comment by ThurstonBT · 2021-10-14T01:59:23.682Z · LW(p) · GW(p)

@ Shouperfluous [LW · GW]: Would you please reread the post to which you are responding?  If you can in fact provide suitable evidence then would you please quote and cite it?

Please note my prior: "No one could provide evidence that THE CLAIM [ Yudkowsky censored mention of Roko's basilisk because it "caused some readers to have nervous breakdowns"] is false and sufficient to meet Wikipedia's evidentiary standards so as [to allow editing Wikipedia so as] to rebut or remove the THE CLAIM".

"No one could provide a citation to a specific denial from Yudkowsky that THE CLAIM did not figure into the decision to censor mention of Roko's basilisk.

I emphasize that the issue here is evidence "sufficient to meet Wikipedia's evidentiary standards".  You might not like Wikipedia's evidentiary standards.  I might share your dislike.  But that is not the point.  Wikipedia, not us, sets Wikipedia's evidentiary standard.

Replies from: RobbBB, Shouperfluous
comment by Rob Bensinger (RobbBB) · 2021-10-14T03:12:08.157Z · LW(p) · GW(p)

As a former Wikipedia admin, I don't think the current basilisk content on Wikipedia meets Wikipedia's evidential standards (or general site standards):

  1. The page claims that "Yudkowsky had stated that it caused some readers to have nervous breakdowns", but none of the three sources cited for this claim say this. The source instead seems to be this claim in an Observer article cited elsewhere on the Wikipedia page: "It seemed like little more than a harmless thought experiment, but rumor has it that the discussion thread was deemed a danger to susceptible minds and exorcised from the blog after a reader had a nervous breakdown." This whole chain seems to be a game-of-telephone bastardization of Roko's statement "one person at SIAI was severely worried by this, to the point of having terrible nightmares".
  2. The Wikipedia edit claims that Eliezer deleted Roko's post because "the idea [of Roko's basilisk] was 'a genuinely dangerous thought'". Eliezer says that's not true, and that he's being misinterpreted (source).
  3. The only source Wikipedia quotes regarding the basilisk argument itself, what it says about the rationalists or the history of LW, etc. is Auerbach's "Roko's basilisk is a referendum on autism" hit piece from 2014, specifically: "the combination of messianic ambitions, being convinced of your own infallibility, and a lot of cash never works out well, regardless of ideology, and I don't expect Yudkowsky and his cohorts to be an exception. I worry less about Roko's Basilisk than about people who believe themselves to have transcended conventional morality."

Large parts of Wikipedia are inconsistent with the site's standards. This is why the encyclopedia is a work in progress. (Well, that plus the fact that the world is big and keeps changing, and our knowledge isn't exhaustive.)

If one of the editors thinks that no strong evidence is required to include claims like 'some readers of Roko's post had nervous breakdowns', but thinks strong evidence is required in order to remove that claim once it's on a Wikipedia page, then the editor just doesn't understand how Wikipedia works. You're welcome to quote my claims here on the relevant WP Talk page and start a discussion there if you think there's any disagreement on any of these points.

Replies from: RobbBB
comment by Rob Bensinger (RobbBB) · 2021-10-14T03:25:49.101Z · LW(p) · GW(p)

Regarding 2: Eliezer's original reply to Roko's post was ambiguous. The current WP article version claims the correct interpretation is X; Eliezer disputes this, and says he meant Y instead.

The actual meaning, in Eliezer's telling, is roughly: 'Roko's argument is silly, but there seems to be non-negligible probability (here in 2010) that there's some structurally analogous argument out there that would work, and the field hasn't had time to explore the space and confirm that there's no such argument; when we haven't done that basic due diligence, we shouldn't go around posting about this on the public Internet.'

Which is easy to round off via a game of telephone to 'Roko's basilisk is dangerous', and thereby to 'Eliezer thinks Roko's argument is sound'. But Wikipedia isn't supposed to be in the business of contributing to games of journalistic telephone. It aspires to be more like the 'final draft' of humanity's knowledge -- an extremely vetted encyclopedia, more carefully fact-checked than Britannica, with large conservative boundaries in place to ensure it errs on the side of excluding any information it's even slightly uncertain about. It obviously hasn't reached that ideal, since it's hard to wrangle thousands of volunteer editors spread across millions of articles; but that is what the editorial policies are trying to create.

In this case, both interpretations are perhaps consistent with the text (neither is silly or obviously false). I think a case can be made that Eliezer's claim about what he had in mind makes sense, there's no special reason to doubt it, and it can just be reported on, if the topic is noteworthy enough to justify that level of detail.

... But if there is serious published disagreement about whether Eliezer is making stuff up here, then: In cases like this, as a matter of policy, Wikipedia avoids doing original research / detective work into sussing out that Eliezer's lying and his real meaning was something totally different from what he claims it was. Instead, if the topic is important enough to cover in detail at all, you quote the source text and then report on noteworthy claims about what that text means.


Regarding 3: This wouldn't necessarily be out of place in a long Wikipedia article that includes a bunch of other quotes, so long as the article is detailed enough to provide any context for why Auerbach thinks any of those things. (As is, someone reading WP to learn about LW for the first time won't know what any of this has to do with 'transcending conventional morality' or 'messianic ambitions'; bringing all this up without context is just bad writing.)

However, on its own, this is an obviously axe-grindy choice of quote that makes the Wikipedia article look like an op-ed that's trying to support a critique of LW, rather than looking like an encyclopedia.

comment by Shouperfluous · 2021-10-15T00:10:54.545Z · LW(p) · GW(p)

My response is a direct refutation of THE CLAIM, in that Yudkowsky explains the rational for the prohibition. 

Replies from: ThurstonBT
comment by ThurstonBT · 2021-10-15T02:43:06.640Z · LW(p) · GW(p)

@Shouperfluous — You don't understand the meaning of "direct refutation".  

An actual direct refutation of  THE CLAIM would be a quote from Yudkowsky to the effect that "my decision to censor mention/discussion of Roko's basilisk was in no way motivated by emotional reactions from members of 'the rationalist community' including but not limited to nervous breakdowns and nightmares.'

You might infer a refutation to the THE CLAIM from the above-linked post but your inference is not a "direct refutation".