Blegg Mode

your argument seems to rely purely on the intuition of your fictional character

Yes, the dependence on intuition is definitely a weakness of this particular post. (I wish I knew as much math as Jessica Taylor [LW · GW]! If I want to become stronger [LW · GW], I'll have to figure out how fit more studying into my schedule!)

you seem to assume that categories are non-overlapping.
you seem to assume that categories are crisp rather than fuzzy

I don't believe either of those things. If you have any specific wording suggestions on how I can write more clearly so as to better communicate to my readers that I don't believe either of those things, I'm listening.

If you take a table made out of a block of wood, and start to gradually deform its shape until it becomes perfectly spherical, is there an exact point when it is no longer called a "table"?

No, there is no such exact point; like many longtime Less Wrong readers [LW · GW], I, too, am familiar with the Sorities paradox.

But in any case, the categorization would depend on the particular trade-offs that the designers of the production line made (depending on things like, how expensive is it to run the palladium scanner)

Right. Another example of one of the things the particular algorithm-design trade-offs will depend on is the distribution of objects.

We could imagine a slightly altered parable in which the frequency distribution of objects is much more evenly spread out in color–shape–metal-content space: while cubeness has a reasonably strong correlation with redness and palladium yield, and eggness with blueness and vanadium yield, you still have a substantial fraction of non-modal objects: bluish-purple rounded cubes, reddish-purple squarish eggs, &c.

In that scenario, a natural-language summary of the optimal decision algorithm wouldn't talk about discrete categories: you'd probably want some kind of scoring algorithm with thresholds for various tests and decisions as you describe, and no matter where you set the threshold for each decision, you'd still see a lot of objects just on either side of the boundary, with no good "joint" to anchor the placement of a category boundary.

In contrast, my reading of Yudkowsky's original parable posits a much sparser, more tightly-clustered distribution of objects in configuration space. The objects do vary somewhat (some bleggs are purple, some rubes contain vanadium), but there's a very clear cluster-structure [LW · GW]: virtually all objects are close to the center of—and could be said to "belong to"—either the "rube" cluster or the "blegg" cluster, with a lot of empty space in between.

In this scenario, I think it does make sense for a natural-language summary of the optimal decision algorithm to talk about two distinct "categories" where the density in the configuration space [LW · GW] is concentrated. Platonic essences are just the limiting case as the overlap between clusters goes to zero.

In my fanfiction, I imagine that some unknown entity has taken objects that were originally in the "rube" cluster, and modified them so that they appear, at first glance but not on closer inspection, to be members of the "blegg" cluster. At first, the protagonist wishes to respect the apparent intent of the unknown entity by considering the modified objects to be bleggs. But in the process of her sorting work, the protagonist finds herself wanting to mentally distinguish adapted bleggs from regular bleggs, because she can't make the same job-relevant probabilistic inferences with the new "bleggs (either regular or adapted)" concept as she could with the old "bleggs (only standard bleggs)" concept.

To see why, forget about the category labels for a moment and just consider the clusters in the six-dimensional color–shape–texture–firmness–luminesence–metal-content configuration space.

Before the unknown entity's intervention, we had two distinct clusters: one centered at {blue, egg, furry, flexible, luminescent, vanadium}, and another centered at {red, cube, smooth, hard, non-luminescent, palladium}.

After the unknown entity's intervention, we have three distinct clusters: the two previously-existing clusters, and a new cluster centered at {blue, egg, furry, hard, non-luminescent, palladium}. This is a different situation! Workers on the sorting line might want different language in order to describe this new reality!

Now, if we were to project into the three-dimensional color–shape–texture subspace, then we would have two clusters again: with just these attributes, we can't distinguish between bleggs and adapted bleggs. But since workers on the sorting line can observe hardness, and care about metal content, they probably want to use the three-cluster representation, even if they suspect the unknown entity might thereby feel disrespected.

Thanks for trying. I have limited time and got a sense for where we seem to have split from each other about halfway through your comment so I'll mainly respond to that. You brought up a bunch of stuff very specific to gender issues that I don't think is relevant in the second half.

There's an underlying situation in which Zack made some arguments elsewhere about gender stuff, and prominent people in the Rationalist community responded with an argument along the lines of "since categories are in the map, not in the territory, there's no point in saying one categorization is more natural than another, we might as well just pick ones that don't hurt people's feelings."

These people are claiming a position on epistemology that Zack thinks is substantially mistaken. Zack is faced with a choice. Either they're giving a politically motivated anti-epistemology in order to shut down the conversation and not because they believe it - or they're making a mistake.

If we take the argument literally, it's worth correcting regardless of one's specific opinions on gender identity.

If we all know that such arguments aren't meant to be taken literally, but are instead meant to push one side of a particular political debate in that context, then arguing against it is actually just the political act of pushing back.

But part of how bad faith arguments work is that they fool some people into thinking they're good-faith arguments. Even if YOU know that people don't mean what they say in this case, they wouldn't say it unless SOMEONE was likely to be honestly mistaken.

"You're doing too much politics here" is not a helpful critique. It doesn't give Zack enough information to get clued in if he's not already, and leaves the key controversial premise unstated. If your actual position is, "come on, Zack, everyone on this site knows that people aren't making this mistake honestly, posts like this one by Scott are mindkilled politics and engaging with them lowers the quality of discourse here," then you need to actually say that.

Personally, I DON'T see people behaving as though it were common knowledge that people claiming to be making this mistake are actually just lying. And if we write off people like Scott we might as well just close down the whole project of having a big Rationalist community on the internet.

It's offensive to me that there's even a question about this.

Gonna try a point-by-point version in case that's clearer.

It isn’t clear to me what this error is meant to be. If it’s something like “thinking that there must be a definite objectively-correct division of all things into bleggs and rubes” then I agree that it’s an error but it’s an error already thoroughly covered by EY’s and SA’s posts linked to in the article itself, and in any case it doesn’t seem to me that the article is mostly concerned with making that point; rather, it presupposes it.

I know from conversations elsewhere that Zack is responding to the opposite error - the claim that because the usual rule for separating Bleggs from Rubes is pragmatically motivated, it has no implications for edge cases. If you're making wrong guesses about what political position Zack is taking, you should really reconsider your claim that it's obvious what his political position is. This needs to be generalized, because it's obnoxious to have to bring in completely extraneous info about motives in order to figure out whether a post like this is political. Bothering to explain this at all feels a bit like giving in to extortion, and the fact that I expect this explanation to be necessary is a further update against continued substantive engagement on Lesswrong.

In any case, it seems to me that the main point of the linked article is not to correct some epistemic error, but to propose a particular position on the political issue it’s alluding to, and that most of the details of its allegory are chosen specifically to support that aim. [...] Constructing a hypothetical situation designed to match your view of a politically contentious question and drawing readers’ attention to that matchup is not “depoliticized” in any useful sense.

This seems like a proposal to cede an untenable amount of conversation territory. If a controversial political position becomes associated with a particular epistemic error, then discussing that specific error becomes off-limits here, or at least needs to be deprecated as political. I don't know what that results in, but it's not a Rationalist community.

I do in fact think that Zack’s purpose in posting the article here is probably at least in part to promote the political position for which the article is arguing, and that if that isn’t so—if Zack’s intention was simply to draw our attention to a well-executed bit of epistemology—then it is likely that Zack finds it well-executed partly because of finding it politically congenial. In that sense, I do think it’s probably a “political act”.

A clear implication of Something to Protect [LW · GW] is that people can't be Rationalists unless getting the right answer has some practical importance to them.

The rest of your comment seems to be making a substantially wrong guess about Zack's position on gender in a way that - to me, since I know something about Zack's position - is pretty strong evidence that Zack succeeded in stripping out the accidental specifics and focusing on the core epistemic question. The standard you're actually holding Zack to is one where if you - perhaps knowing already that he has some thoughts on gender - can project a vaguely related politically motivated argument onto his post, then it's disingenuous to say it's nonpolitical.

It seems relevant here that Zack pretty much agreed with my description: see his comments using terms like "deniable allegory", "get away with it", etc.

So, from my perspective, I'm facing a pretty difficult writing problem here! (See my reply to Dagon [LW(p) · GW(p)].) I agree that we don't want Less Wrong to be a politicized space. On the other hand, I also think that a lot of self-identified rationalists are making a politically-motivated epistemology error in asserting category boundaries to be somewhat arbitrary, and it's kind of difficult to address what I claim is the error without even so much as alluding to the object-level situation that I think is motivating the error! For the long, object-level discussion, see my reply to Scott Alexander, "The Categories Were Made for Man To Make Predictions". (Sorry if the byline mismatch causes confusion; I'm using a pen name for that blog.) I didn't want to share "... To Make Predictions" on Less Wrong (er, at least not as a top-level post), because that clearly would be too political. But I thought the "Blegg Mode" parable was sufficiently sanitized such that it would be OK to share as a link post here?

I confess that I didn't put a lot of thought into the description text which you thought was disingenuous. I don't think I was being consciously disingenuous (bad intent is a disposition, not a feeling!), but after you pointed it out, I do see your point that, since there is some unavoidable political context here, it's probably better to explicitly label that, because readers who had a prior expectation that no such context would exist would feel misled upon discovering it. So I added the "Content notice" to the description. Hopefully that addresses the concern?

our categories are [...] somewhat arbitrary

No! Categories are not "somewhat arbitrary"! There is structure in the world, and intelligent agents need categories that carve the structure at the joints [LW · GW] so that they can make efficient probabilistic inferences about the variables they're trying to optimize! "Even if you cannot do the math, knowing that the math exists tells you that the dance step is precise and has no room in it for your whims." We had a whole Sequence [LW · GW] about this! Doesn't anyone else remember?!

Trans-ness is not always "cheap to detect". I guess it's cheaper to detect than, say, sex chromosomes. OK -- and how often are another person's sex chromosomes "decision-relevant with respect to the agent's goals"?

You seem to be making some assumptions about which parts of the parable are getting mapped to which parts of the real-world issue that obviously inspired the parable. I don't think this is the correct venue for me to discuss the real-world issue. On this website, under this byline, I'd rather only talk about bleggs and rubes—even if you were correct to point out that it would be disingenuous for someone to expect readers to pretend not to notice the real-world reason that we're talking about bleggs and rubes. With this in mind, I'll respond below to a modified version of part of your comment (with edits bracketed).

I guess it's cheaper to detect than, say, [palladium or vanadium content]. OK -- and how often [is a sortable object's metal content] "decision-relevant with respect to the agent's goals"? Pretty much only if [you work in the sorting factory.] [That's] fairly uncommon -- for most of us, very few of the [sortable objects] we interact with [need to be sorted into bins according to metal content].

Sure! But reality is very high-dimensional [LW · GW]—bleggs and rubes have other properties besides color, shape, and metal content—for example, the properties of being flexible-vs.-hard or luminescent-vs.-non-luminescent, as well as many others that didn't make it into the parable. If you care about making accurate predictions about the many properties of sortable objects that you can't immediately observe, then how you draw your category boundaries matters, because your brain is going to be using the category membership you assigned in order to derive your prior expectations about the variables that you haven't yet observed [LW · GW].

sex chromosomes, which is exactly the "expensive" feature the author identifies in the case of trans people.

The author did no such thing! It's epistemology fiction about bleggs and rubes! It's true that I came up with the parable while I was trying to think carefully about transgender stuff that was of direct and intense personal relevance to me. It's true that it would be disingenuous for someone to expect readers to not-notice that I was trying to think about trans issues. (I mean, it's in the URL.) But I didn't say anything about chromosomes! "If confusion threatens when you interpret a metaphor as a metaphor, try taking everything completely literally."

Trying to think of some examples, it seems to me that what matters is simply the presence of features that are "decision-relevant with respect to the agent's goals". [...]

Thanks for this substantive, on-topic criticism! I would want to think some more before deciding how to reply to this.

ADDENDUM: I thought some more and wrote a sister comment [LW(p) · GW(p)].

Trying to think of some examples, it seems to me that what matters is simply the presence of features that are "decision-relevant with respect to the agent's goals". [...]

So, I think my motivation (which didn't make it into the parable) for the "cheap to detect features that correlate with decision-relevant expensive to detect features" heuristic is that I'm thinking in terms of naïve Bayes models [LW · GW]. You imagine a "star-shaped" causal graph [LW · GW] with a central node (whose various values represent the possible categories you might want to assign an entity to), with arrows pointing to various other nodes (which represent various features of the entity). (That is, we're assuming that the features of the entity are conditionally independent given category membership: P(X|C) = Π_i P(X_i|C).) Then when we observe some subset of features, we can use that to update our probabilities of category-membership, and use that to update our probabilities of the features we haven't observed yet. The "category" node doesn't actually "exist" out there in the world—its something we construct to help factorize our probability distribution over the features (which do "exist").

So, as AI designers, we're faced with the question of how we want the "category" node to work. I'm pretty sure there's going to be a mathematically correct answer to this that I just don't know (yet) because I don't study enough and haven't gotten to Chapter 17 of Daphne Koller and the Methods of Rationality. Since I'm not there yet, if I just take at intuitive amateur guess at how I might expect this to work, it seems pretty intuitively plausible that we're going to want the category node to be especially sensitive to cheap-to-observe features that correlate with goal-relevant features? Like, yes, we ultimately just want to know as much as possible about the decision-relevant variables, but if some observations are more expensive to make than others, that seems like the sort of thing the network should be able to take into account, right??

Remember those 2% of otherwise ordinary bleggs that contain palladium? Personally, I'd want a category for those

I agree that "things that look like 'bleggs' that contain palladium" is a concept that you want to be able to think about. (I just described it in words, therefore it's representable!) But while working on the sorting line, your visual system's pattern-matching faculties aren't going to spontaneously invent "palladium-containing bleggs" as a thing to look out for if you don't know any way to detect them, whereas if adapted bleggs tend to look different in ways you can see, then that category is something your brain might just "learn from experience." In terms of the naïve Bayes model, I'm sort of assuming that the 2% of palladium containing non-adapted bleggs are "flukes": that variable takes that value with that probability independently of the other blegg features. I agree that if that assumption were wrong, then that would be really valuable information, and if you suspect that assumption is wrong, then you should definitely be on the lookout for ways to spot palladium-containing bleggs.

But like, see this thing I'm at least trying to do here, where I think there's learnable statistical structure in the world that I want to describe using language? That's pretty important! I can totally see how, from your perspective, on certain object-level applications, you might suspect that the one who says, "Hey! Categories aren't even 'somewhat' arbitrary! There's learnable statistical structure in the world; that's what categories are for!" is secretly being driven by nefarious political motivations. But I hope you can also see how, from my perspective, I might suspect that the one who says, "Categories are somewhat arbitrary; the one who says otherwise is secretly being driven by nefarious political motivations" is secretly being driven by political motivations that have pretty nefarious consequences for people like me trying to use language to reason about the most important thing in my life, even if the psychological foundation of the political motivation is entirely kindhearted.

When I first read the post, I did not notice any reference to any mindkilling topics and was actually quite confused and surprised when I saw the comments about all of this being about something super political, and still found the post moderately useful. So I do think that I am a counterexample to your "I don't think many of those likely to be mindkilled will miss those implications; " argument.

better to be up front about them

... you're right. (I like the aesthetics of the "deniable allegory" writing style, but delusionally expecting to get away with it is trying to have one's cake and eat it, too.) I added a "Content notice" to the description here.

Thanks. In retrospect, possibly a better approach for this venue would have been to carefully rewrite the piece for Less Wrong in a way that strips more subtext/conceals more of the elephant (e.g., cut the "disrespecting that effort" paragraph).

WTF. I didn't downvote (until now), but didn't see any point to so many words basically saying "labels are lossy compression, get over it".

Now that I actually notice the website name and understand that it's an allegory for a debate that doesn't belong here (unless gender categorization somehow is important to LW posts), I believe it also doesn't belong here. I believe that it doesn't belong here regardless of which side I support (and I don't have any clue what the debate is, so I don't know what the lines are or which side, if any, I support).