What Motte and Baileys are rationalists most likely to engage in?

Thinking about it more: I can imagine a group that tries to become unusually good at learning true things in a pretty domain-general way, so they call themselves the Learnies, or the Discoverers, or the Trutheteers.

If a group like that succeeds in advancing its art, then it should end up using that art to discover at least some truths about the world at large that aren't widely known. This should be a reliable consequence of 'getting good at discovering truths'. And in many such worlds, it won't be trivial for the group to then convince the entire rest of the world to believe the things they learned overnight. So five years might pass and the Learnies are still the main group that is aware of, say, the medical usefulness of some Penicillium molds (because they and/or the world they're in is dysfunctional in some way).

It seems natural to me that the Learnies' accumulated learnings get mentally associated with the group, even though penicillin doesn't have any special connection to the art of learning itself. So I don't think there's anything odd about rationalists${}_2$ being associated with 'knowledge we accumulated in the process of applying rationality techniques', or about my wanting a way to figure out to what extent someone at a party has acquired that knowledge. I think this example does illustrate a few issues, though:

• First, obviously, it's important to be able to distinguish 'stuff the Learnies learned that's part of the art of learning' from 'other stuff the Learnies learned'. Possibly it would help here to have more names for the clusters of nonstandard beliefs rationalists${}_2$ tend to have, so I'm somewhat less tempted to think 'is this person familiar with the rationalist${}_2$-ish content on ems?' vs. 'is this person familiar with the techno-reductionist content on ems?' or whatever.
• Second, I've been describing the stuff above as "knowledge". But what if there's a dispute about whether the Learnies' worldly discoveries are true? In that case, maybe it's logically rude to call yourself 'Learnies', because (a) it maybe rhetorically tilts the debate in your favor, and (b) it's just not very informative. (Example: A world where pro-penicillin people call themselves Penicillists and anti-penicillin people call themselves Anti-Penicillists is better than a world where pro-penicillin people call themselves Accuracyists and anti-penicillin people call themselves Realityists.)
  • Considerations like this update me toward using the term 'aspiring rationalist' more, because it hopefully makes it tilts debate less toward the presupposition that 'whoever identifies themselves as a truth-seeker' is correct about whatever thing they think their rationality techniques have helped them learn.
  • (Though I think 'aspiring rationalist' still tilts the debate some. But, thinking about it again, maybe I actually prefer the world where more people make one flavor or another of 'truth-seeking' part of their identity? It might be at least somewhat easier to call people out on bullshit and make them reflect and update, if it were more common for people to take pride in their truth-seeking and knowledgeableness, however unwarranted.)
• Then there's the question of whether the Learnies' commitment to the art of learning obliges them to never Glomarize or stay-out-of-the-fray about politically controversial things they learn. I take it that something along these lines is your main criticism of rationalists${}_2$ calling themselves 'rationalists', but I'm not sure exactly what norm you're advocating. Currently I don't find this compelling -- like, I do think it's epistemically risky for groups to selectively avoid talking about important politicized topics, and I think it's important to try to find ways to counter resultant epistemic distortions in those cases (and to seriously consider whether staying-out-of-the-fray is just a dumb strategy). But I guess I just disagree with some of the examples you've provided, and agree with others but don't think they're as serious or deeply-epistemic-corrupting as you do? Mostly, I just don't think I've read enough of the stuff you've written to understand your full argument; but I can at least give my current epistemic state.

This doesn't seem to be about the term rationalist at all. It seems to be about which rhetorical style different people prefer. Eliezer makes in a much more confident and more polarizing way then Scott. 

One option might be 'do the rationalist-ish thing when you're forced to because it's decision-relevant; but when you're just analyzing an interesting intellectual puzzle for fun, don't do the rationalist-ish thing'

This is the closest to what I was trying to say, but I would scope my criticism even more narrowly. To try and put it bluntly and briefly: Don't choose to suspend disbelief for multiple core hypotheses within your argument, while simultaneously holding that the final conclusion built off of them is objectively likely and has been supported throughout. 

The motte with this argument style, that your conclusion is the best you can do given your limited data, is true and I agree. Because of that this is a genuinely good technique for decision making in a limited space, as you mention. What I see as the bailey though, that your conclusion is actually probable in a real and objective sense, and that you've proven it to be so with supporting logic and data, is what doesn't follow to me. Because you haven't falsified anything in an objective sense, there is no guaranteed probability or likelihood that you are correct, and you are more likely to be incorrect the more times in your argument you've chosen to deliberately suspend disbelief for one of your hypotheses to carry onward. Confidence intervals are a number you're applying to your own feelings, not actual odds of correctness, so can't be objectively used to calculate your chance of being right overall.

Put another way, in science it is totally possible and reasonable for a researcher to have an informed hypothesis that multiple hypothetical mechanisms in the world all exist, and that they combine together to cause some broader behavior that so far has been unexplained. But if this researcher were to jump to asserting that the broader behavior is probably happening because of all these hypothetical mechanisms, without first actively validating all the individual hypotheses with falsifiable experiments, we'd label their proposed broad system of belief as a pseudoscience. The pseudoscience label would still be true even if their final conclusion turned out to be accurate, because the problem here is with the form (assuming multiple mechanisms are real without validating them) rather than the content (the mechanisms themselves). This becomes better or worse the more of these hypothetical but unproven mechanisms need to exist and depend on each other for the researcher's final conclusion to be true.

I hear you on examples, but since I don't like posts that do this I don't have any saved to point at unfortunately. I can go looking for new ones that do this if you think it would still be helpful though.

The implied claim that I took from the passage (perhaps incorrectly) is that motte and bailey is a fallacy inherent to post-modernist thought in general, rather than a bad rhetorical technique that some post-modernists commenters engage in on the internet. From that it should be easier, not harder, to cite real-world examples of it since the rhetorical fallacy is actually widespread and representative of post-modern thought. The government example isn't analogous, as it would have at least been a real-world example and the person in that hypothetical wouldn't be trying to argue that the dysfunctional dynamic is inherent to all government. But the quote chose to make up an absurd post-modernist claim about the sun being socially constructed to try and prove a claim that post-modernism is absurd.

I made my aside because I am a relatively everyday person who is a general fan of post-modernism, or at least the concept of social construction as I've described, and I have a strong suspicion that whatever specific real-world examples the author is pattern-matching as denying objective reality probably have a stronger argument for being a socially constructed than they're aware of. Or at least able to hand-wave as absurd as easily as their sun hypothetical.

This is all just an aside of an aside though, and I somewhat regret putting it in the body of my post and distracting from the rest. People generally do make terrible arguments on the internet, so in terms of sheer volume I do agree that bad arguments abound.

Well, the motte is "I'm very epistemically humble", and the bailey is "that's why I'm always right".

Shouldn't it be the other way round?