Blog posts as epistemic trust builders

But fortunately, I have a very high level of epistemic trust for the rationalist community.

No! Not fortunately! Speaking from personal experience, succumbing to the delusion that there is any such thing as "the rationalist community" worthy of anyone's trust has caused me an enormous amount of psychological damage, such that I'm still (still?!) not done recovering from the betrayal trauma more than three years later—and I'm probably not the only one.

(Uh, can't say I wasn't warned [LW · GW].)

(I thought I was done recovering as of (specifically) 13 September, but the fact that I still felt motivated to write a "boo 'rationalists'" comment on Friday [LW(p) · GW(p)] and then went into an anxiety spiral for the next 36 hours—and the fact that I'm drafting this comment in a paper notebook when I should be spending a relaxing network-free Sunday studying math—suggest that I'm still (somehow—still?!) not done grieving. I think I'm really close, though!)

There is no authority regulating who's allowed to use the "rationalist" brand name. Trusting "the rationalist community" leaves you open to getting fucked over^[1] by any bad idea that can successfully market itself to high-Openness compsci nerds living in Berkeley, California in the current year. The craft is not the community. The ideology is not the movement. Don't revere the bearer of good info. [LW · GW] Every cause—every cause—wants to be a cult. [LW · GW] At this point, as a guard against my earlier mistakes, I've made a habit of using the pejorative "robot cult" to refer to the social cluster, reserving "rationalist" to describe the methodology set forth in the Sequences—and really, I should probably phase out "rationalist", too. Plain rationality is already a fine word for cognitive algorithms [LW · GW] that create and exploit map–territory correspondences—maybe it doesn't have to be an -ism.

Real trust—trust that won't predictably blow up in your face and take three and a half years to recover from—needs to be to something finer-grained than some amorphous self-recommending "community." You need to model the specific competencies of specific people and institutions, and model their incentives to tell you the truth—or to fuck with you.

(Note: people don't have to be consciously fucking with you in order for modeling them as fucking with you to be useful [LW · GW] for compressing the length of the message needed to describe your observations. I can't speak to what the algorithms of deception [LW · GW] feel from the inside [LW · GW]—just that the systematic production of maps that don't reflect the territory for any reason, even mere "bias", should be enough for you to mark them as hostile.)

COVID-19 is an unusually easy case, where people's interests are, for the most part, aligned. People may use the virus as a prop in their existing political games [LW · GW], but at least no one is actually pro-virus. Under those circumstances, sure, trusting an amorphous cluster of smart people who read each other's blogs can legitimately be a better bet than alternative aggregators of information. As soon as you step away from the unusually easy cases—watch your step!

If you learned a lot from the Sequences, I think that's a good reason to trust what Eliezer Yudkowsky in particular says about AI in particular, even if you can't immediately follow the argument. (There's a prior that any given nonprofit claiming you should give it money in order to prevent the destruction of all value in the universe [LW · GW] is going to just be a scam [LW · GW]—but you see, the Sequences are very good.) That trust does not bleed over (except at a very heavy quantitative discount) to an alleged "community" of people who also trust Yudkowsky—and I don't think it bleeds over to Yudkowsky's off-the-cuff opinions on (let's say, picking an arbitrary example) the relative merits of polyamory, unless you have some more specific reason to trust that he actually thought it through and actually made sure to get that specific question right, rather than happening to pick up that answer through random cultural diffusion from [? · GW] his robot cult. (Most people get most of their beliefs from random cultural diffusion; we can't think fast enough [LW · GW] to do otherwise.) Constant vigilance!

Building reputation by repeated interaction.

But it needs to be the type of interaction where you notice and remember the author. For example, if you go to LessWrong, you are more likely to associate "I read this on LessWrong" with the information, than if you just visited LessWrong articles from links shared on social networks. (And it is probably easier to remember Zvi than an average author at LessWrong, because Zvi recently posted a sequence of articles, which is easier to remember than an equal number of articles on unrelated topics.) You need to notice "articles by Zvi" as a separate category first, and only then your brain can decide to associate trust with this category.

(Slate Star Codex takes this a bit further, because for my brain it is easier to remember "I read this on SSC" than to remember the set of articles written by Scott on LessWrong. This is branding. If your quality is consistently high, making the fact "this was written by me" more noticeable increases your reputation.)

The flip side of the coin is that the culture of sharing hyperlinks on social networks destroys trust. If you read hundred articles from hundred different sources every day, your brain has a problem to keep tabs. Before internet, when you regularly read maybe 10 different journals, you gradually noticed that X is reliable and Y is unreliable. Because sometimes you read ten reliable stories on one day, and ten unreliable stories on a different day, and it felt differently. But on internet, there are hundred websites, and you switch between them, so even if a few of the are notoriously bad, it is hard to notice. Even harder, because the same website can have multiple authors with wildly different quality. A scientist and a crackpot can have a blog on the same domain. With paper sources, the authors within one source were more balanced. (LessWrong is also kinda balanced, especially if you only consider the upvoted articles.)