Salvage Epistemology

• 'force fields' like gravity

AFAIK gravity was indeed considered at least woo-ish back in the day, e.g.:

Newton’s theory of gravity (developed in his Principia), for example, seemed to his contemporaries to assume that bodies could act upon one another across empty space, without touching one another, or without any material connection between them. This so-called action-at-a-distance was held to be impossible in the mechanical philosophy. Similarly, in the Opticks he developed the idea that bodies interacted with one another by means of their attractive and repulsive forces—again an idea which was dismissed by mechanical philosophers as non-mechanical and even occult.

One example, maybe: I think the early 20th century behaviorists mistakenly (to my mind) discarded the idea that e.g. mice are usefully modeled as having something like (beliefs, memories, desires, internal states), because they lumped this in with something like "woo."  (They applied this also to humans, at least sometimes.)

The article Cognition all the way down argues that a similar transition may be useful in biology, where e.g. embryogenesis may be more rapidly modeled if biologists become willing to discuss the "intent" of a given cellular signal or similar.  I found it worth reading.  (HT: Adam Scholl, for showing me the article.)

I think "you should one-box on Newcomb's problem" is probably an example.  By the time it was as formalized as TDT it was probably not all that woo-y looking, but prior to that I think a lot of people had an intuition along the lines of "yes it would be tempting to one-box but that's woo thinking that has me thinking that."

I like this inquiry. Upvoted.

Do you think they often do [look like woo]…?

Well… yes, but not for deep reasons. Just an impression. The cases where I've made shifts from "that's woo" to "that's true" are super salient, as are cases where I try to invite others to make the same update and am accused of fuzzy thinking in response. Or where I've been the "This is woo" accuser and later made the update and slapped my forehead.

Also, "woo" as a term is pretty strongly coded to a particular aesthetic. I don't think you'd ever hear concern about "woo" in, say, Catholicism except to the extent the scientist/atheist/skeptic/etc. cluster is also present. But Catholics still slam into ontology updates that look obviously wrong beforehand and are obviously correct afterwards. Deconversion being an individual-scale example.

(Please don't read me as saying "Deconversion is correct." I could just as well have given the inverse example: Rationalists converting to Catholicism is also an ontological update that's obviously wrong beforehand and obviously correct afterwards. But that update does look like "woo" beforehand, so it's not an example of what I'm trying to name.)

Do you… have salient non-controversial examples?

I like the examples others have been bringing. I like them better than mine. But I'll try to give a few anyway.

Speaking to one of your "maybe never woo" examples: if I remember right, the germ theory of disease was incredibly bizarre and largely laughed at when first proposed. "How could living creatures possibly be that small? And if they're so small, how could they possibly create that much illness?" Prevailing theories for illness were things like bad air and demons. I totally expect lots of people thought the microbes theory was basically woo. So that's maybe an example.

Another example is quantum mechanics. The whole issue Einstein took with it was how absurd it made reality. And it did in fact send people like Bohm into spiritual frenzy. This is actually an incomplete ontology update in that we have the mathematical models but people still don't know what it means — and in physics at least they seem to deal with it by refusing to think about it. "If you do the math, you get the right results." Things like the Copenhagen Interpretation or Many Worlds are mostly ways of talking about how to set up experiments. The LW-rationalist thing of taking Many Worlds deeply morally seriously is, as far as I can tell, pretty fringe and arguably woo.

You might recall that Bishop Berkeley had some very colorful things to say about Newton's infinitesimals. "Are they the ghosts of departed quantities?" If he'd had the word "woo" I'm sure he would have used it. (Although this is an odd example because now mathematicians do a forgivable motte-and-bailey where they say infinitesimal thinking is shorthand for limits when asked. Meaning many of them are using an ontology that includes infinitesimals but quickly hide it when challenged. It's okay because they can still do their formal proofs with limits, but I think most of them are unaware of the various ways to formalize infinitesimals as mathematical objects. So this is a case where many mathematicians are intentionally using an arguably woo fake framework [LW · GW] and translating their conclusions afterwards instead of making the full available ontology update.)

Given that I'm basically naming the Semmelweis reflex, I think Semmelweis's example is a pretty good one. "What?! You're accusing me, an educated professional gentleman, of carrying filth on my hands?! Preposterous! How dare you?!" Obviously absurd and wrong at the time, but later vindicated as obviously correct.

many worlds interpretation

Received such a bad reception that Everett left academic physics.

atomism

Didn't seem crazy to the Greeks, but was controversial when reintroduced by Boltzman.

If you think that some kinds of ideas are probably harmful for some people to hear, is acting on that belief always saviorist fuckery or does there exist a healthy form of it?

I am not Valentine, but I would say: it is “saviorist fuckery” if your view is “these ideas are harmful to hear for people who aren’t me/us (because we’re enlightened/rational/hyper-analytic/educated/etc. and they’re not)”. If instead you’re saying “this is harmful for everyone to hear (indeed I wish I hadn’t heard it!), so I will not disseminate this to anyone”, well, that’s different. (To be clear, I disapprove of both scenarios, but it does seem plausible that the motivations differ between these two cases.)

I think the incentives in science and woo are different.

I agree, though I'm not sure how that observation relates to my comment. But yes, certainly evaluating the incentives and causal history of a claim is an important part of epistemology.

Someone who wants to "salvage" e.g. Buddhism is privileging a source that has a replication rate way below 50%.

I'm not sure if it really makes sense to think in terms of salvaging "Buddhism", or saying that it has a particular replication rate (it seems pretty dubious whether the concept of replication rate is well-defined outside a particular narrow context in the first place). There are various claims associated with Buddhism, some of which are better-supported and potentially valuable than others. 

E.g. my experience is that much of meditation seems to work the way some Buddhists say it works, and some of their claims seem to be supported by compatible models and lines of evidence [? · GW] from personal experience, neuroscience, and cognitive science. Other claims, very much less so. Talking about the "replication rate of Buddhism" seems to suggest taking a claim and believing it merely on the basis of Buddhism having made such a claim, but that would be a weird thing to do. We evaluate any claim on the basis of several factors, such as what we know about the process that generated them, how compatible they are with our other beliefs, how useful they would be for explaining experiences we've had, what success similar claims have shown in helping us get better at something we care about, etc.. And then some other claim that's superficially associated with the same source (e.g. "Buddhism") might end up scoring so differently on those factors that it doesn't really make sense to think of them as even being related. 

Even if you were looking at a scientific field that was known to have a very low replication rate, there might be some paper that seemed more likely to be true and also relevant for things that you cared about. Then it would make sense to take the claims in that paper and use them to draw conclusions that were as strong or weak as warranted, based on that specific paper and everything else you knew.

Someone who wants to "salvage" e.g. Buddhism is privileging a source that has a replication rate way below 50%.

Who has tried to replicate it?

Scientists are rewarded for discovering new things, or finding an error in existing beliefs, so if 100 scientists agree on something, that probably means more than if 100 astrologers agree on something. 

When aspiring rationalists interact with science, it's not just believing whatever 100 scientists agree on. If you take COVID-19 for example, we read a bunch of science, build models in our head about what's happening and then took action based on those models.

Scientists are rewarded for discovering new things, or finding an error in existing beliefs, so if 100 scientists agree on something, that probably means more than if 100 astrologers agree on something.

It's not obvious to me this effect dominates over politician punishments for challenging powerful people's ideas. I definitely think science is more self-correcting than astrology over decades, but don't trust the process on a year to year basis.

It is difficult to distinguish between (1) people who think that there may be some value in a woo, and it is worth exploring it and separating the wheat from the chaff, and (2) people who believe that the woo is useful, and their only question is how to make it more palatable for the rationalist community. Both these groups are together opposed to people who would refuse to touch the woo in principle.

The subtle difference between those two groups is the absence or presence of motivated reasoning. If you are willing to follow evidence wherever it may lead you, you are open to the possibility that the horoscopes may actually correlate with something useful, but you are also open to the possibility that they might not. While the "salvage at all costs" group already knows that the horoscopes are useful, and are useful in more or less the traditional way, the only question is how to convince the others, who are immune to the traditional astrological arguments, but it seems mostly like the question of using the right lingo [LW · GW], so perhaps if we renamed Pisces to "cognitive ichthys", the usual objections would stop and rationalists would finally accept that Pisces might actually be cognitively different from other people; especially if a high-status community member supported it publicly with an anecdote or two.

(The opposite kind of mistake would be refusing to accept in principle that being a Virgo might correlate with your success at school... simply because it makes you one of the oldest kids in the classroom.)

Maybe it's a question of timing. First prove that some part of the woo makes sense; then use it. Do not simply start with the assumption that surely most of the woo will be salvageable somehow; it may not.

Suggestion withdrawn.

I don't have a complete or principled model of what an epistemic immune system is or ought to be, in the area of woo, but I have some fragments.

One way of looking at it is that we look at a cluster of ideas, form an outside view of how much value and how much crazymaking there is inside it, and decide whether to engage. Part of the epistemic immune system is tracking the cost side of the corresponding cost/benefit. But this cost/benefit analysis doesn't generalize well between people; there's a big difference between a well-grounded well-studied practitioner looking at their tenth fake framework, and a newcomer who's still talking about how they vaguely intend to read the Sequences.

Much of the value, in diving into a woo area, is in the possibility that knowledge can be extracted and re-cast into a more solid form. But the people who are still doing social-mimicking instead of cost/benefit are not going to be capable of doing that, and shouldn't copy strategies from people who are.

(I am trying not to make this post a vagueblog about On Intention Research, because I only skimmed it and I don't know the people involved well, so I can't be sure it fits the pattern, but the parts of it I looked at match what I would expect from a group trying to ingest fake frameworks that they weren't skilled enough to handle.)

I think there's something important in the distinction between declarative knowledge and metis, and something particularly odd about the process of extracting metis from an area where you believe all of the declarative knowledge in the vicinity is false. I think when a group does that together, they wind up generating a new thing attached to a dialect full of false rhymes, where they sound like they're talking about the false things but all the jargon is slightly figurative and slightly askew. When I think of what it's like to be next to that group but not in it, I think of Mage: The Ascenscion.

Engaging with woo can be a rung on the countersignaling hierarchy: a way to say, look, my mental integrity is so strong that I can have crystal conversations and still somehow stay sane. This is orthogonal to cost/benefit, but I expect anyone doing it for that reason to tell themself a different story. I'm not sure how much of a thing this is, but would be surprised if it wasn't a thing at all.

Are people here mostly materialists?

Okay, since you seem interested in knowing why people are materialists. I think it's the history of science up until now. The history of science has basically been a constant build-up of materialism.

We started out at prehistoric animism where everything happening except that rock you just threw at another rock was driven by an intangible spirit. The rock wasn't since that was just you throwing it. And then people started figuring out successive compelling narratives about how more complex stuff is just rocks being thrown about. Planets being driven by angels? Nope, just gravitation and inertia. Okay, so comets don't have comet spirits, but surely living things have spirits [LW · GW]. Turns out no, molecular biology is a bit tricky, but it seems to still paint a (very small) rocks thrown about picture that convincingly gets you a living tree or a cat. Human minds looked unique until people started building computers. The same story is repeating again, people point human activities as proofs of the indomitable human spirit, then someone builds an AI to do it. Douglas Hofstadter was still predicting that mastering chess would have to involve encompassing the whole of human cognition in 1979 and had to eat crow in the introduction of the 20th anniversary edition of his book.

So to sum up, simple physics went from spiritual (Aristotle's "rocks want to go down, smoke wants to go up") to materialist, the outer space went from spiritual to materialist, biological life went from spiritual to materialist and mental acts like winning a chess game went from spiritual to materialist.

We're now down to the hard problem of consciousness, and we're also still missing a really comprehensive scientific picture for how you go from neurons to high-level human thought. So which way do you think this is going to go? A discovery that the spiritual world exists after all, and was hiding in the microtubules of the human brain all along, or people looking at the finished blueprint for how the brain works that explains things up to conscious thought and going "oh, so that's how it works" and it's all just rocks thrown about once again. So far we've got a perfect record of everybody clamoring for the first option and then things turning out to be the second one.

Are people here mostly materialists?

Yes.

I reviewed it on LW [LW · GW], almost 10 years ago now.

It's an academic textbook on rationality. It defines rationality as:

The best kind of thinking, which we shall call rational thinking, is whatever kind of thinking best helps people achieve their goals. If it should turn out that following the rules of formal logic leads to eternal happiness, then it is “rational thinking” to follow the laws of logic (assuming that we all want eternal happiness). If it should turn out, on the other hand, that carefully violating the laws of logic at every turn leads to eternal happiness, then it is these violations that we shall call “rational.”

When I argue that certain kinds of thinking are “most rational,” I mean that these help people achieve their goals. Such arguments could be wrong. If so, some other sort of thinking is most rational.

I love your observations here. The quality of grounding in a clear intuition here.

I don't think you can avoid the subculture thing. The discipline doesn't exist in a void the way math kind of does. Unless & until you can actually define the practice of rationality, there's no clear dividing line between the social scene and the set of people who practice the discipline. No clear analogue to "actually playing a guitar".

Like, I think I follow your intuition, but consider:

Meanwhile, I know at least some people definitely meet any reasonable bar for "actually a goddamn rationalist". If you intentionally reflect on and direct your cognitive patterns in ways that are more likely to find true beliefs and accomplish your goals, and you've gone off into the world and solved some difficult problems that depended on you being able to do that... I think you're just plain a rationalist.

I'm reasonably sure a lot of people here would consider me a great example of a non-rationalist. Lots of folk told me that to my face while I worked at CFAR. But the above describes me to an utter T. I'm just doing it in a way that the culture here doesn't approve of and thinks is pretty nutty. Which is fine. I think the culture here is doing its "truth-seeking" in a pretty nutty way too. Y'all are getting great results predicting Covid case numbers, and I'm getting great results guiding people to cure their social anxiety and depression. To each their own.

I think what you're talking about is way, way more of an aesthetic [LW · GW] than you might realize. Like, what are you really using to detect who is and isn't "actually a goddamn rationalist"? My guess is it's more of a gut sense that you then try to examine upon reflection.

Is Elon Musk "actually a goddamn rationalist"? He sure seems to care about what's true and about being effective in the world. But I'm guessing he somehow lands as less of a central example than Oli or Eliezer do. If so, why?

If Elon doesn't do it for you, insert some other successful smart person who mysteriously doesn't gut-ping as "actually a goddamn rationalist".

If I'm way off here, I'd actually be pretty interested in knowing that. Because I'd find that illuminating as to what you mean by rationalism.

But if I'm basically right, then you're not going to separate the discipline from the social scene with a term. You'll keep seeing social status and perception of skill conflated. Not exactly overlapping, but muddled nonetheless.

I don't know; finding a better solution sounds great, but there aren't that many people who talk here, and many of us are fairly reflective and ornery, so if a small group keeps repeatedly requesting this and doing it it'd probably be sufficient to keep "aspiring rationalist" as at least a substantial minority of what's said.

“Effective Altruist” has six syllables, “Aspiring Rationalist” has seven. Not that different.

I will try using it in my writing more for a while.

This was not the usage in the Sequences, however, and otherwise at the time the Sequences were written.

The same model that says aspiring rationalist will self-replace with rationalist, says aspiring beisutsuka will self-rpeplace with beisutsuka. But beisutsuka is a bit better than rationalist on its own terms; it emphasizes being a practitioner more, and presupposes the skill less. And it avoids punning with a dozen past historical movements that each have their own weird connotations and misconceptions. Unfortunately the phonology and spelling of beisutsuka is 99.9th-percentile tricky and that might mean it's also a linguistic invalid move.

I kinda like this

for me it mostly felt like I and my group of closest friends were at the center of the world, with the last hope for the future depending on our ability to hold to principle. there was a lot of prophesy of varying qualities, and a lot of importance placed suddenly on people we barely knew then rapidly withdrawn when those people weren't up for being as crazy as we were.

What you say sounds like it could easily be very reasonable, and yet it has almost nothing in common, results-wise, with what we actually observe among rationalists who take psychedelics.