Moderation Reference

Yeah, thinking overnight and reading more comments this morning has me updating that I shouldn't have used the word steelmanning there, and I'll update it soon (although I'm not 100% sure what the best term here is and not sure there's a single term that does what I want).

Background:

When discussing this with other mods, I suggested saying that LW should be about collaborative rather than adversarial truthseeking, and other mods noted that there is a time for adversarial truthseeking even on LW and it'd probably be epistemically fraught to try and bake collaboration into the LW discussion DNA.

I ended up writing the three bullet points (address core points, invest interpretive labor, and being kind) as standalone points. It seemed important to distinguish the first two points from the last one. I searched my brain for a term that seemed to encompass the first two terms, generated "Steelman" and then called it a day. But, yeah, steelman has other properties that don't quite make sense for what I'm trying to point at here.

Your criticism of steelmanning is apt. Of “interpretive effort” I have already spoken elsewhere [LW · GW]. As for “addressing core points”, however…

The problem with this criterion is simply that an author and a reader may disagree on what the core points are.

Note that I am not simply talking about cases where a reader misunderstands what the author wrote, and mistakes something for an intended “core point” that is no such thing! Such cases are, in a sense, trivial, insofar as clearing up the misunderstanding results in the original piece standing unmodified (excepting, perhaps, any clarifying modifications that are aimed at preventing exactly such misunderstanding from reoccurring in other readers).

Rather, I’m talking about cases where the reader understands what the author is saying, but considers the author’s claimed “core point” to actually be of peripheral significance, and considers a different point (one which the author either did not mention at all, or devoted only scant attention to) to be central. In this case, no misunderstanding, per se, is occurring (unless you call a deep conceptual or empirical error a “misunderstanding”, which I do not—I reserve the latter term for miscommunication).

This sort of thing can come in many forms. I will describe just one of them here.

It sometimes happens that someone—call him Dave—will write a post describing some (purported) phenomenon P—some pattern, some dynamic, some causal structure, etc.—and, to illustrate P, will provide an example E.

Carol, a reader, then comes along and says: “Actually, Dave, I don’t think E is an example of P! Consider the following…” Dave perhaps defends his example, perhaps not, but agrees, in the end, that E is not really a good example of P after all. “But,” says Dave, “that was just an example! Quit nitpicking, Carol—address my core points!”

What is Carol to say to this? Does Dave accuse her fairly? Is it mere nitpicking to attack a mere example?

But if E is the only example of P that Dave had provided, then—E having now been disqualified from that role—Dave is left with no examples of P. And if Dave cannot provide any other (or, I should simply say, any) examples of P, then perhaps P is simply not real? I can hardly thinking of a more “core” point than the question of whether the thing we’re talking about is even a real thing!

As I say, the “attacking an example” form is just one of the ways in which the “disagreement about what the core points are” dynamic can manifest. But it alone has, in my view, been responsible for a great deal of epistemic trouble among rationalists. As I’ve said elsewhere, we—the types of people who frequent such sites as Less Wrong—are very good at inventing abstract patterns, “crystallizing concepts”, constructing systems of classification, and so on. We are so good at it, in fact, that this talent of ours often gets away from us. It is absolutely critical to keep it reined in. (I do not choose the expression lightly! Reins, after all, are used, not to prevent a horse from moving, but to make it move in the direction you want to go—instead of uselessly running wild.) And those reins are made of real-world examples, they are made of extensions (as vs. intensions), they are made of practice. Without it, deceiving ourselves that we’ve gained knowledge, when in fact we’re building sky castles of abstract nonsense, is all too easy.

Only that a criticism that steelmans is, typically, more valuable than criticism that doesn’t.

I disagree. Steelmanning is nice, but I don't think it necessarily adds value. I think there is real value in engaging the actual arguments that the person made, in the way that they made them. If LessWrong is going to train rationalists to argue for their points persuasively, I think it's imperative that we engage with the actual evidence that is presented, and not the idealized version of the evidence that would have convinced us of the conclusions.

Edit: After thinking about it some more, I have realized that steelmanning poses a danger to the listener as well as to the speaker. Namely, given two arguments of equal strength, one which I am able to steelman, and one which I am not, it's quite possible I will find the argument that I am able to steelman more convincing, even though it has no more evidence behind it than the argument that I am not able to steelman. It seems to me that steelmanning exaggerates our cognitive blindspots, rather than reducing them. Can you show me that steelmanning is not an epistemic hazard?

It doesn't seem like steelmanning is particularly useful for communication or critique. It refines ideas into something different, more interesting, drawing attention away from to the original. This makes it useful for collaborative truth seeking, or just for your own thinking based on what you read.

A useful notion of steelmanning needs to be distinguished from charity and rationalization. Charity is looking into the reasons a person believes or says the things they do. The beliefs themselves may be ridiculous and not useful to understand, but the reasons for arriving at them point to a real process in someone's mind and may clarify the context where the beliefs come up. Ignoring the beliefs as something you won't accept is different from ignoring the process that created them, and charity is about paying attention to the process. The reasons for holding a belief can be different from arguments given for it, and there is also a question about the reasons for arriving at certain arguments. Pursuing charity leads to identifying errors in thinking. It's also the right point of view on weaponized words that turn out not to reflect beliefs in the usual sense, but serve a purpose, even without knowledge of the people utilizing the words.

Steelmanning, on the other hand, acts on the beliefs themselves. It brings to attention improved versions of the beliefs, versions that may be more worthy of discussion than the original, non-steelmanned beliefs. So it's a way of changing the topic to something occasionally more interesting, and in that it's similar to charity, but it changes the topic in a completely different way.

Rationalization is finding a convincing argument for a predefined position. When the position is incorrect, even slightly, the arguments to choose from are flawed, and the task is to find the most convincing of them. The flaws are mostly about ignoring some evidence and giving too much weight to other evidence, although if the audience is not too discerning, other flaws may allow the argument to become even more convincing.

Steelmanning of a belief discards the problem statement for rationalizing it by changing the belief. Steelmanning of an argument for a predefined belief is almost exactly rationalization. But steelmanning an argument without requiring its conclusion to come out the same may be interesting, even as it changes the topic of discussion.