Comment by zack_m_davis on Comment section from 05/19/2019 · 2019-05-23T06:24:14.920Z · score: 2 (1 votes) · LW · GW

one voice with an agenda which, if implemented, would put me in physical danger

Okay, I think I have a right to respond to this.

People being in physical danger is a bad thing. I don't think of myself as having a lot of strong political beliefs, but I'm going to take a definite stand here: I am against people being in physical danger.

If someone were to present me with a persuasive argument that my writing elsewhere is increasing the number of physical-danger observer-moments in the multiverse on net, then I would seriously consider revising or retracting some of it! But I'm not aware of any such argument.

Comment by zack_m_davis on Comment section from 05/19/2019 · 2019-05-23T06:03:40.979Z · score: 3 (3 votes) · LW · GW

(I had originally strong-downvoted the parent because I don't think it's relevant, but alas, it looks like the voting population disagreed.)

even when he is writing directly about the matter on his own blog, I am unclear what he is actually saying

Wait, really? Am I that bad of a writer??

There is still a certain abstractness and distance from the object level.

Well, yes. I'm a rationalist. What do you expect?

Comment by zack_m_davis on Why books don't work · 2019-05-23T05:24:55.742Z · score: 4 (2 votes) · LW · GW

we usually don't give babies problems, projects, etc to teach them to walk and talk, but they learn just fine

We have adaptations for learning to walk and talk, though: the process for learning evolutionarily-novel content could be different.

Comment by zack_m_davis on Comment section from 05/19/2019 · 2019-05-21T23:14:47.066Z · score: 9 (5 votes) · LW · GW

the meta level is too vague. That is, the error is in the way the abstract reasoning is applied to case X (it's just not the right model), rather than in the abstract reasoning itself

Why not write a meta-level post about the general class of problem for which the abstract reasoning doesn't apply? That could be an interesting post!

I'm guessing you might be thinking something along the lines of, "The 'draw category boundaries around clusters of high density in configuration space' moral doesn't apply straightfowardly to things that are socially constructed by collective agreement"? (Examples: money, or Christmas. These things exist, but only because everyone agrees that they exist.)

I personally want to do more thinking about how social construction works (I have some preliminary thoughts on the matter that I haven't finished fleshing out yet), and might write such a post myself eventually!

Comment by zack_m_davis on Comment section from 05/19/2019 · 2019-05-21T02:41:14.036Z · score: 6 (5 votes) · LW · GW

Thanks for clarifying!

people with negative intentions [...] deliberately

So, it's actually not clear to me that deliberate negative intentions are particularly important, here or elsewhere? Almost no one thinks of themselves as deliberately causing avoidable harm, and yet avoidable harm gets done, probably by people following incentive gradients that predictably lead towards harm, against truth, &c. all while maintaining a perfectly sincere subjective conscious narrative about how they're doing God's work, on the right side of history, toiling for the greater good, doing what needs to be done, maximizing global utility, acting in accordance with the moral law, practicing a virtue which is nameless, &c.

it was important for me to speak up, even if you're not a werewolf.

Agreed. If I'm causing harm, and you acquire evidence that I'm causing harm, then you should present that evidence in an appropriate venue in order to either persuade me to stop causing harm, or persuade other people to coördinate to stop me from causing harm.

I was claiming that the long post you wrote at the top of the thread where you made several analogies about your response, were exactly the sort of gray area situations where, depending on context, the community might decide to sacrifice it's sacred value.

So, my current guess (which is only a guess and which I would have strongly disagreed with ten years ago) is that this is a suicidally terrible idea that will literally destroy the world. Sound like an unreflective appeal to sacred values? Well, maybe!—you shouldn't take my word for this (or anything else) except to the exact extent that you think my word is Bayesian evidence. Unfortunately I'm going to need to defer supporting argumentation to future Less Wrong posts, because mental and financial health requirements force me to focus on my dayjob for at least the next few weeks. (Oh, and group theory.)

(End of thread for me.)

Minimax Search and the Structure of Cognition!

2019-05-20T05:25:35.699Z · score: 15 (4 votes)
Comment by zack_m_davis on Comment section from 05/19/2019 · 2019-05-20T03:19:59.145Z · score: 14 (6 votes) · LW · GW

And when you see someone explicitly pushing the gray area by trying to get you to accept harmful situations by appealing to that sacred value

Um, in context, this sounds to me like you're arguing that by writing "Where to Draw the Boundaries?" and my secret ("secret") blog, I'm trying to get people to accept harmful situations? Am I interpreting you correctly? If so, can you explain in detail what specific harm you think is being done?

Comment by zack_m_davis on Comment section from 05/19/2019 · 2019-05-20T02:22:20.535Z · score: 16 (5 votes) · LW · GW

Right, I agree that it doesn't sound difficult from a web-development perspective, but I also think that only praising difficult-to-implement features would create the wrong incentives.

Comment by zack_m_davis on Comment section from 05/19/2019 · 2019-05-20T01:55:06.455Z · score: 5 (7 votes) · LW · GW

or establish the case that agreement on those abstractions doesn't force agreement on the object-level issues

Why would anyone think that agreement on meta-level abstractions (like the thing I was trying to say in "Where to Draw the Boundaries?") would force agreement on object-level issues? That would be crazy! Object-level issues in the real world are really complicated: you'd need to spend a lot more wordcount just to make a case for any particular object-level claim—let alone reach agreement!

Comment by zack_m_davis on Comment section from 05/19/2019 · 2019-05-20T01:13:07.833Z · score: 8 (5 votes) · LW · GW

Wow, it's neat that the LW 2 codebase gives you tools to move a derailed thread to its own post! Good job, whoever wrote that feature!

Comment by zack_m_davis on Comment section from 05/19/2019 · 2019-05-19T21:27:45.552Z · score: 4 (2 votes) · LW · GW

I've found /r/TheMotte (recently forked from /r/slatestarcodex) to be a good place to discuss politically-charged topics? (Again, also happy to talk privately sometime.)

Comment by zack_m_davis on Comment section from 05/19/2019 · 2019-05-19T20:09:18.558Z · score: 27 (8 votes) · LW · GW

Thanks, these are some great points on some of the costs of decoupling norms! (As you've observed, I'm generally pretty strongly in favor of decoupling norms, but policy debates should not appear one-sided.)

someone brings it up all the time

I would want to distinguish "brings it up all the time" in the sense of "this user posts about this topic when it's not relevant" (which I agree is bad and warrants moderator action) versus the sense of "this user posts about this topic a lot, and not on other topics" (which I think is generally OK).

If someone is obsessively focused on their narrow special interest—let's say, algebraic topology—and occasionally comments specifically when they happen to think of an application of algebraic topology to the forum topic, I think that's fine, because people reading that particular thread get the benefit of a relevant algebraic topology application—even if looking at that user's posting history leaves one with an unsettling sense of, "Wow, this person is creepily obsessed with their hobbyhorse."

tries to twist other people's posts towards a discussion of their thing

I agree that this would be bad, but I think it's usually possible to distinguish "twist[ing] other people's posts towards a discussion of their thing" from a genuinely relevant mention of the thing that couldn't (or shouldn't) be reasonably expected to derail the discussion?

In the present case, my great-great-grandparent comment notes that the list-of-koans format lends itself to readers contributing their own examples in the comments, and I tried to give two such examples (trying to mimic the æsthetic of the OP by continuing the numbered list and Alice/Bob/Charlie/&c. character name sequence), one of which related the theme of the OP to the main point of one of my recent posts.

In retrospect, maybe I should've thought more carefully about how to phrase the proposed example in a way that makes the connection to the OP more explicit/obvious? (Probably-better version: "A meaningful 'Yes' answer to the question 'Is G an H?' requires a definition of H such that the answer could be 'No'.")

It's true that, while composing the great-great-grandparent, I was kind of hoping that some readers would click through the link and read my earlier post, which I worked really hard on and which I think is filling in a gap in "A Human's Guide to Words" that I've seen people be confused about. But I don't see how this can reasonably be construed as an attempt to derail the discussion? Like, I ordinarily wouldn't expect a brief comment of the form "Great post! Here's a couple more examples that occurred to me, personally" to receive any replies in the median case.

(Although unfortunately, it empirically looks like the discussion did, in fact, get derailed. I feel bad for Scott G. that we're cluttering up his comment section like this, but I can't think of anything I wish I had done differently other than wording the great-great grandparent more clearly, as mentioned in the paragraph-before-last. Given Vanessa's reply, I felt justified in writing my counterreply ... and here we are.)

It would be perfectly alright for moderators who didn't want to drive away their visitors to ask this person to stop.

Agreed, the moderators are God and their will must be obeyed.

kick out someone who has a bad reputation that makes important posters unable to post on your website because they don't want to associate with that person, even IF that person has good behavior

So, the dynamic you describe here definitely exists, but I actually think it's a pretty serious problem for our collective sanity: if some truths happen to lie outside of Society's Overton window, then systematic truthseekers (who want to collect all the truths, not just the majority of them that are safely within the Overton window) will find themselves on the wrong side of Respectability, and if people who care about being Respectable (and thereby having power in Society) can't even talk to people outside the Overton window (not even agree with—just talk to, using, for example, a website), then that could have negative instrumental consequences in the form of people with power in Society making bad policy decisions on account of having inaccurate beliefs.

I want to write more about this in the future (albeit not on Less Wrong), but in the meantime, maybe see the immortal Scott Alexander's "Kolmogorov Complicity And The Parable Of Lightning" for an expression of similar concerns:

Some other beliefs will be found to correlate heavily with lightning-heresy. Maybe atheists are more often lightning-heretics; maybe believers in global warming are too. The enemies of these groups will have a new cudgel to beat them with, "If you believers in global warming are so smart and scientific, how come so many of you believe in lightning, huh?" Even the savvy Kolmogorovs within the global warming community will be forced to admit that their theory just seems to attract uniquely crappy people. It won't be very convincing. Any position correlated with being truth-seeking and intelligent will be always on the retreat, having to forever apologize that so many members of their movement screw up the lightning question so badly.

Comment by zack_m_davis on Comment section from 05/19/2019 · 2019-05-19T01:05:21.541Z · score: 57 (26 votes) · LW · GW

That's understandable, but I hope it's also understandable that I find it unpleasant that our standard Bayesian philosophy-of-language somehow got politicized (!?), such that my attempts to do correct epistemology are perceived as attacking people?!

Like, imagine an alternate universe where posts about the minimum description length principle were perceived as an attack on Christians (because atheists often argue that Occam's razor implies that theories about God are unnecessarily complex), and therefore somewhat unseemly (because politics is the mind-killer, and criticizing a popular religion has inextricable political consequences).

I can see how it would be really annoying if someone on your favorite rationality forum wrote a post about minimum description length, if you knew that their work about MDL was partially derived from other work (on a separate website, under a pseudonym) about atheism, and you happened to think that Occam's razor actually doesn't favor atheism.

Or maybe that analogy is going to be perceived as unfair because we live in a subculture that pattern-matches religion as "the bad guys" and atheism as the "good guys"? (I could try to protest, "But, but, you could imagine as part of the thought experiment that maybe Occam's razor really doesn't favor atheism", but maybe that wouldn't be perceived as credible.)

Fine. We can do better. Imagine instead some crank racist psuedoscientist who, in the process of pursuing their blatantly ideologically-motiviated fake "science", happens to get really interested in the statistics of the normal distribution, and writes a post on your favorite rationality forum about the ratio of areas in the right tails of normal distributions with different means.

I can see how that would be really annoying—maybe even threatening! Which might make it all the more gratifying if you can find a mistake in the racist bastard's math: then you could call out the mistake in the comments and bask in moral victory as the OP gets downvoted to oblivion for the sin of bad math.

But if you can't find a mistake—if, in fact, the post is on-topic for the forum and correct in the literal things that it literally says, then complaining about the author's motive for being interested in the normal distribution doesn't seem like an obviously positive contribution to the discourse?—even if you're correct about the author's motive. (Although, you might not be correct.)

Like, maybe statistics is part of the common interest of many causes, such that, as a matter of local validity, you should assess arguments about statistics on their own merits in the context that those arguments are presented, without worrying about how those arguments might or might not be applied in other contexts?

What, realistically, do you expect the atheist—or the racist, or me—to do? Am I supposed to just passively accept that all of my thoughts about epistemology are tainted and unfit for this forum, because I happen to be interested in applying epistemology to other topics (on a separate website, under a pseudonym)?

I think the grandparent is an on-topic response to the OP, relating the theme of the OP (about how if you don't have negative feedback or "No"s, then that makes positive feedback or "Yes"es less significant) to both a hypothetical example about social network voting mechanisms, and, separately, to another philosophy topic (about the cognitive function of categories) that I've been thinking a lot about lately! That's generally what happens when people comment on posts: they think about the post in the context of their own knowledge and their own priorities, and then write a comment explaining their actual thoughts!

Like, if you think the actual text of anything I write on this website is off-topic, or poorly-reasoned, or misleading on account of omitting relevant considerations, then please:

  • Downvote it, and
  • Leave a critical comment explaining what I got wrong (if you have time).

Those actions are unambiguously prosocial, because downvotes help other users decide what's worth their time to read, and criticism of bad reasoning helps everyone reading get better at reasoning! But criticizing me because of what you know about my personal psychological motives for making otherwise-not-known-to-be-negative contributions seems ... maybe less obviously prosocial?

Like, what happens if you apply this standard consistently? Did you know that Eliezer Yudkowsky's writings that are ostensibly about human rationality, were actually mostly conceived in the context of his plans to build a superintelligence to literally take over the world?! (Although he denies it, of course.) That's politics! Should we find it unpleasant that Yudkowsky always brings his hobbyhorse in, but in an "abstract" way that doesn't allow discussing the actual object-level political question about whether he should rule the world?

Am I wrong here? Like, I see your concern! I really do! I'm sorry if we happen to be trapped in a zero-sum game whereby my attempts to think seriously in public about things I'm interested in ends up imposing negative externalities on you! But what, realistically, do you expect me to do? Happy to talk privately sometime if you'd like. (In a few weeks; I mostly want to focus on group theory and my dayjob for the rest of May.)

Comment by zack_m_davis on Yes Requires the Possibility of No · 2019-05-18T04:52:29.037Z · score: 10 (19 votes) · LW · GW

This is great! One subtle advantage of the list-of-koans format is that it provides a natural prompt for the reader to think up their own as an exercise.

  1. Irene wants to believe the claim "G is an H." H is a fuzzy category and the word "H" can be used in many ways depending on context. Irene finds she can make the sentence "G is an H" more probable by deliberately choosing a sufficiently broad definition of "H", but only at the cost of making the word less useful.

  2. Jessica posts something to a social-media platform that has "Like"s, but not downvotes. She doesn't know whether no one saw her post, or if lots of people saw it but they all hated it.

Comment by zack_m_davis on Where to Draw the Boundaries? · 2019-04-29T01:16:29.293Z · score: 3 (2 votes) · LW · GW

(I continue to regret my slow reply turnaround time.)

But there are times when it's not a dishonest rhetorical move to do this, right?

Right. In Scott's example, the problem was using the "eargrayish" concept to imply (bad) inferences about size, but your example isn't guilty of this.

However, it's also worth emphasizing that the inferential work done by words and categories is often spread across many variables, including things that aren't as easy to observe as the features that were used to perform the categorization. You can infer that "mice" have very similar genomes, even if you never actually sequence their DNA. Or if you lived before DNA had been discovered, you might guess that there exists some sort of molecular mechanism of heredity determining the similarities between members of a "species", and you'd be right (whereas similar such guesses based on concepts like "eargrayishness" would probably be wrong).

(As it is written: "Having a word for a thing, rather than just listing its properties, is a more compact code precisely in those cases where we can infer some of those properties from the other properties.")

Since it doesn't seem to make sense to never use a word to point to a cluster in a "thin" subspace, what is your advice for when it's ok to do this or accept others doing this?

Um, watch out for cases where the data clusters in the "thin" subspace, but doesn't cluster in other dimensions that are actually relevant in the context that you're using the word? (I wish I had a rigorous reduction of what "relevant in the context" means, but I don't.)

As long as we're talking about animal taxonomy (dolphins, mice, elephants, &c.), a concrete example of a mechanism that systematically produces this kind of distribution might be Batesian or Müllerian mimicry (or convergent evolution more generally, as with dolphins' likeness to fish). If you're working as a wildlife photographer and just want some cool snake photos, then a concept of "red-'n'-yellow stripey snake" that you formed from observation (abstractly: you noticed a cluster in the subspace spanned by "snake colors" and "snake stripedness") might be completely adequate for your purposes: as a photographer, you just don't care whether or not there's more structure to the distribution of snakes than what looks good in your pictures. On the other hand, if you actually have to handle the snakes, suddenly the difference between the harmless scarlet kingsnake and the poisonous coral snake ("red on yellow, kill a fellow; red on black, venom lack") is very relevant and you want to be modeling them as separate species!

Comment by zack_m_davis on Where to Draw the Boundaries? · 2019-04-21T03:39:32.996Z · score: 16 (6 votes) · LW · GW

(continued from sister comment)

My reply to Wei is now up. (I finally looked at his four links and didn't end up engaging with them, but I endorse Benquo's comment on #4.)

I also left a brief reply to your comment about chronic fatigue syndrome, and a reply to your comment critiquing the paragraph about "poison." I hope this helps clarify what I'm trying to communicate.

Unfortunately, I don't think your participation here has been a net-positive for the value of the comments section, and (with some sadness) I have decided to add you to the "Banned Users" list in the moderation section of my account settings.

Comment by zack_m_davis on Where to Draw the Boundaries? · 2019-04-21T03:32:43.252Z · score: 17 (5 votes) · LW · GW

It's not clear whether or not that important difference is supposed to imply to the reader that one is better then the other. Given that there seems to be a clear value judgement in the others, maybe it does here?

All three paragraphs starting with "There's an important difference [...]" are trying to illustrate the distinction between choosing a model because it reflects value-relevant parts of reality (which I think is good), and choosing a model because of some non-reality-mapping consequences of the choice of model (which I think is generally bad).

words that are unnecessarily obscure (most people in society won't understand what wasting cycles is about)

The primary audience of this post is longtime Less Wrong readers; as an author, I'm not concerned with trying to reach "most people in society" with this post. I expect Less Wrong readers to have trained up generalization instincts motivating the leap to thinking about AIs or minds-in-general even though this would seem weird or incomprehensible to the general public.

To those people who proofread and appeartly didn't find an issue in that sentence, is it really necessary to mix all those different issues into a 6-line sentence?

It's true that I tend to have a "dense" writing style (with lots of nested parentheticals and subordinate clauses), and that I should probably work on writing more simply in order to be easier to read. Sorry.

Comment by zack_m_davis on Where to Draw the Boundaries? · 2019-04-21T03:30:38.190Z · score: 11 (3 votes) · LW · GW

CFS and SEID are both cases where certain states correlate with each other Zacks post doesn't help us at all to reason about whether we should prefer CFS or SEID as a term.

I'm definitely not claiming to have the "correct" answer to all terminological disputes. (As the post says, "Of course, there isn't going to be a unique way to encode the knowledge into natural language.")

Suppose, hypothetically, that it were discovered that there are actually two or more distinct etiologies causing cases that had historically been classified as "chronic fatigue syndrome", and cases with different etiologies responded better to different treatments. In this hypothetical scenario, medical professionals would want to split what they had previously called "chronic fatigue syndrome" into two or more categories to reflect their new knowledge. I think someone who insisted that "chronic fatigue syndrome" was still a good category given the new discovery of separate etiologies would be making a mistake (with respect to the goals doctors have when they talk about diseases), even if the separate etiologies had similar symptoms (which is what motivated the CFS label in the first place).

In terms of the configuration space visual metaphor, we would say that while "chronic fatigue syndrome" is a single cluster in the "symptoms" subspace of Diseasespace, more variables than just symptoms are decision-relevant to doctors, and the CFS cluster doesn't help them reason about those other variables.

Comment by zack_m_davis on Where to Draw the Boundaries? · 2019-04-21T03:28:19.955Z · score: 23 (6 votes) · LW · GW

Hi, Wei—thanks for commenting! (And sorry for the arguably somewhat delayed reply; it's been a really tough week for me.)

can you give a concrete example that might come up on LW or otherwise have some relevance to us?

Is Slate Star Codex close enough? In his "Anti-Reactionary FAQ", Scott Alexander writes—

Why use this made-up word ["demotism"] so often?

Suppose I wanted to argue that mice were larger than grizzly bears. I note that both mice and elephants are "eargreyish", meaning grey animals with large ears. We note that eargreyish animals such as elephants are known to be extremely large. Therefore, eargreyish animals are larger than noneargreyish animals and mice are larger than grizzly bears.

As long as we can group two unlike things together using a made-up word that traps non-essential characteristics of each, we can prove any old thing.

This post is mostly just a longer, more detailed version (with some trivial math) of the point Scott is making in these three paragraphs: mice and elephants form a cluster if you project into the subspace spanned by "color" and "relative ear size", but using a word to point to a cluster in such a "thin", impoverished subspace is a dishonest rhetorical move when your interlocutors are trying to use language to mostly talk about the many other features of animals which don't covary much with color and relative-ear-size. This is obvious in the case of mice and elephants, but Scott is arguing that a similar mistake is being made by reactionaries who classify Nazi Germany and the Soviet Union as "demotist", and then argue that liberal democracies suffer from the same flaws on account of being "demotist." Scott had previously dubbed this kind of argument the "noncentral fallacy" and analyzed how it motivates people to argue over category boundaries like "murder" or "theft."

Downthread, you wrote

My interest in terminological debates is usually not to discover new ideas but to try to prevent confusion (when readers are likely to infer something wrong from a name, e.g., because of different previous usage or because a compound term is defined to mean something that's different from what one would reasonably infer from the combination of individual terms).

I agree that preventing confusion is the main reason to care about terminology; it only takes a moderate amount of good faith and philosophical sophistication for interlocutors to negotiate their way past terminology clashes ("I wouldn't use that word because I think it conflates these-and-such things, but for the purposes of this conversation ..." &c.) and make progress discussing actual ideas. But I wanted to have this post explaining in detail a particular thing that can go wrong when philosophical sophistication is lacking or applied selectively, which was mostly covered by Eliezer's "A Human's Guide to Words", but of which I hadn't seen the "which subspace to pay attention to / do clustering on" problem treated anywhere in such terms.

Comment by zack_m_davis on Where to Draw the Boundaries? · 2019-04-18T07:30:16.760Z · score: 7 (3 votes) · LW · GW

Sure, casual use of categories is convenient and pretty good for a lot of purposes. [...] Where precision matters, though, you're better off using more words. Don't try to cram so much inferential power into a categorization that's not a good fit for the domain of predictions you're making.

So, I actually don't think "casual" vs. "precise" is a good characterization of the distinction I was trying to make in the grandparent! I'm saying that for "sparse", tightly-clustered distributions in high-dimensional spaces, something like "essentialism" is actually doing really useful cognitive work, and using more words to describe more basic, lower-level ("precise"?) features doesn't actually get you better performance—it's not just about minimizing cognitive load.

A good example might be the recognition of accents. Which description is more useful, both for your own thinking, and for communicating your observations to others—

At the level of consciousness, it's much easier to correctly recognize accents than to characterize and articulate all the individual phoneme-level features that your brain is picking up on to make the categorization. Categories let you make inferences about hidden variables that you haven't yet observed in a particular case, but which are known to correlate with features that you have observed. Once you hear the non-rhoticity in someone's speech, your brain also knows how to anticipate how they'll pronounce vowels that they haven't yet said—and where the person grew up! I think this is a pretty impressive AI capability that shouldn't be dismissed as "casual"!

Comment by zack_m_davis on Where to Draw the Boundaries? · 2019-04-18T02:16:31.177Z · score: 22 (8 votes) · LW · GW

You're being bizarrely demanding, and I don't understand why. Have I done something to offend you somehow? (If I have accidentally offended and there's some way I could make amends, feel free to PM me.)

I agree that authors advocating an idea should provide examples. That's why the OP does, in fact, provide some examples (about dolphins, abstract points in ℝ³, and job titles). I also have a couple other cached "in the wild" examples in mind that I intend to include in my reply to Wei (e.g., search for the word eargreyish in Scott Alexander's "Anti-Reactionary FAQ"). But, as the grandparent mentions, Wei specifically asked if I had any thoughts on four of his comments (which I still haven't read, incidentally). I can't possibly have cached such thoughts in advance!

Writing good comments takes nontrivial time and mental energy and given that at least some Less Wrong readers probably have things like jobs (!) or possibly even families (?!), I really don't think it's reasonable to infer that someone is incapable of offering a satisfactory reply just because they haven't replied within a couple days.

I had a really stressful day yesterday. I just got home today. After posting this comment, I want to make dinner and relax and read the new Greg Egan novel for a while. After that, I intend to spend some time writing blog comment replies—to Wei, to Dagon again, to someone on Reddit—and then maybe to some of your comments, if I still have time. (I also need to look up what I need to bring to my DMV appointment tomorrow.) Please be patient with me—although if you're so dissatisfied by both the post, and my comments so far, then I fear my future comments are unlikely to be that much more to your liking, so it's not clear why you should be so eager to see them be posted faster.

In conclusion, I'm sorry you didn't like my blog post about the information theory of dolphins. Please feel free to downvote it if you haven't already.

Comment by zack_m_davis on Where to Draw the Boundaries? · 2019-04-16T15:55:49.534Z · score: 7 (4 votes) · LW · GW

That's also likely why WeiDei's request to get practical examples went unanswered.

Alternative explanation: that comment was made on a Sunday afternoon in my timezone, I have a Monday-through-Friday dayjob that occupies a lot of my attention, and I wanted to set aside a larger block of time to read through the four comments (and surrounding context) Wei linked (1 2 3 4) and think carefully about them before composing a careful reply. (I spent my Sunday afternoon writing budget on my reply to dadadarren, which took a while because I had to study the "Ugly duckling theorem" Wikipedia page he linked.) In contrast, a reply like this one, or my reply to Dagon don't require additional studying time to compose, which is why I can manage to type something like this now without being too late to my dayjob.

your post is detached from any empiricsm but about the search of essenses of words.

I don't think this is a fair characterization of the post.

I need to go get dressed and catch a train now. I'll ping you when my reply to Wei is up.

Comment by zack_m_davis on Where to Draw the Boundaries? · 2019-04-15T16:18:48.445Z · score: 13 (4 votes) · LW · GW

We agree that models are only better or worse for a purpose, but ...

Ask whether this creature needs air. Ask how fast it swims. etc.

If there are systematic correlations between many particular creature-features like whether it needs air, how fast it swims, what it's shaped like, what its genome is, &c., then it's adaptive to have a short code for the conjunction of those many features that such creatures have in common.

Category isn't identity, but the cognitive algorithm that makes people think category is identity actually performs pretty well when things are tightly-clustered in configuration space rather than evenly distributed, which actually seems to be the case for a lot of things! (E.g., while there are (or were) transitional forms between species related by evolutionary descent, it makes sense that we have separate words for cats and dogs rather than talking about individual creature properties of ear-shape, &c., because there aren't any half-cats in our real world.)

Comment by zack_m_davis on Where to Draw the Boundaries? · 2019-04-15T15:16:44.182Z · score: 6 (3 votes) · LW · GW

Saying that the primary intention which which language is used isn't to create some effect in the recipient of the language act

It's notable to me that both of the passages from this post that you quoted in the great-grandparent comment were from the final section. Would your assessment of the post change if you pretend it had ended just before the Musashi quote, with the words "resulting in fallacies of compression"?

I was trying to create an effect in the recipients of the language act by riffing off Yudkowsky's riff off Musashi in "Twelve Virtues of Rationality", which I expected many readers to be familiar with (and which is the target of the hyperlink with the text "is quoted"). My prereaders seemed to get it, but it might have been the wrong choice if too many readers' reactions were like yours.

Comment by zack_m_davis on Where to Draw the Boundaries? · 2019-04-15T03:30:25.872Z · score: 3 (2 votes) · LW · GW

Thanks for the pointer! I've played with word2vec and similar packages before, but had never thought to explore how those algorithms connect with the content of "A Human's Guide to Words".

Comment by zack_m_davis on Where to Draw the Boundaries? · 2019-04-15T03:22:01.149Z · score: 13 (3 votes) · LW · GW

Well, I usually try not to argue against theorems (as contrasted to arguing that a theorem's premises don't apply in a particular situation)—but in spirit, I guess so! Let me try to work out what's going on here—

The boxed example on the Wikipedia page you link, following Watanabe, posits a universe of three ducks—a White duck that comes First, a White duck that is not First, and a nonWhite duck that is not First—and observes that every pair of ducks agrees on half of the possible logical predicates that you can define in terms of Whiteness and Firstness. Generally, there are sixteen possible truth functions on two binary variables (like Whiteness or Firstness), but here only eight of them are distinct. (Although really, only eight of them could be distinct, because that's the number of possible subsets of three ducks (2³ = 8).) In general, we can't measure the "similarity" between objects by counting the number of sets that group them together, because that's the same for any pair of objects. We also get a theorem on binary vectors: if you have some k-dimensional vectors of bits, you can use Hamming distance to find the "most dissimilar" one, but if you extend the vectors into 2^k-dimensional vectors of all k-ary boolean functions on the original k bits, then you can't.

Watanabe concludes, "any objects, in so far as they are distinguishable, are equally similar" (!!).

So, I think the reply to this is going to have to do with inductive bias and the "coincidence" that we in fact live in a low-entropy universe where some cognitive algorithms actually do have an advantage, even if they wouldn't have an advantage averaged over all possible universes? Unfortunately, I don't think I understand this in enough detail to explain it well (mumble mumble, new riddle of induction, blah blah, no canonical universal Turing machine for Solomonoff induction), but the main point I'm trying to make in my post is actually much narrower and doesn't require us to somehow find non-arbitrary canonical categories or reason about all possible categories.

I'm saying that which "subspace" of properties a rational agent is interested in will depend on the agent's values, but given such a choice, the categories the agent ends up with is going to be the result of running some clustering algorithm on the actual distribution of things in the world, which depends on the world, not the agent's values. In terms of Watanabe's ducks: you might not care about a duck's color or its order, but redefining Whiteness to include the black duck is cheating; it's wireheading yourself; it can't help you optimize the ducks.

Where to Draw the Boundaries?

2019-04-13T21:34:30.129Z · score: 56 (28 votes)
Comment by zack_m_davis on Blegg Mode · 2019-03-16T21:01:41.466Z · score: 8 (4 votes) · LW · GW

I wasn't claiming to summarize "Disguised Queries".

I may have misinterpreted what you meant by the phrase "makes essentially the point that."

the thing that you say no one says other than to push a particular position on trans issues

I see. I think I made a mistake in the great-great-grandparent comment. That comments' penultimate paragraph ended: "[...] and who somehow never seem to find it useful to bring up the idea that categories are somewhat arbitrary in seemingly any other context." I should not have written that, because as you pointed out in the great-grandparent, it's not true. This turned out to be a pretty costly mistake on my part, because we've now just spent the better part of four comments litigating the consequences of this error in a way that we could have avoided if only I had taken more care to phrase the point I was trying to make less hyperbolically.

The point I was trying to make in the offending paragraph is that if someone honestly believes that the choice between multiple category systems is arbitrary or somewhat-arbitrary, then they should accept the choice being made arbitrarily or somewhat-arbitrarily. I agree that "It depends on what you mean by X" is often a useful motion, but I think it's possible to distinguish when it's being used to facilitate communication from when it's being used to impose frame control. Specifically: it's incoherent to say, "It's arbitrary, so you should do it my way," because if it were really arbitrary, the one would not be motivated to say "you should do it my way." In discussions about my idiosyncratic special interest, I very frequently encounter incredibly mendacious frame-control attempts from people who call themselves "rationalists" and who don't seem to do this on most other topics. (This is, of course, with respect to how I draw the "incredibly mendacious" category boundary.)

Speaking of ending conversations, I'm feeling pretty emotionally exhausted, and we seem to be spending a lot of wordcount on mutual misunderstandings, so unless you have more things you want to explain to me, maybe this should be the end of the thread? Thanks for the invigorating discussion! This was way more productive than most of the conversations I've had lately! (Which maybe tells you something about the quality of those other discussions.)

Comment by zack_m_davis on Blegg Mode · 2019-03-16T13:31:33.014Z · score: 9 (3 votes) · LW · GW

I mean, yes, there's the allusion in the title! (The post wasn't originally written for being shared on Less Wrong, it just seemed sufficiently sanitized to be shareable-here-without-running-too-afoul-of-anti-politics-norms after the fact.)

Comment by zack_m_davis on Blegg Mode · 2019-03-16T01:01:03.586Z · score: 7 (4 votes) · LW · GW

The "Disguised Queries" post that first introduced bleggs and rubes makes essentially the point that categories are somewhat arbitrary, that there's no One True Right Answer to "is it a blegg or a rube?", and that which answer is best depends on what particular things you care about on a particular occasion.

That's not how I would summarize that post at all! I mean, I agree that the post did literally say that ("The question 'Is this object a blegg?' may stand in for different queries on different occasions"). But it also went on to say more things that I think substantially change the moral—

If [the question] weren't standing in for some query, you'd have no reason to care.

[...] People who argue that atheism is a religion "because it states beliefs about God" are really trying to argue (I think) that the reasoning methods used in atheism are on a par with the reasoning methods used in religion, or that atheism is no safer than religion in terms of the probability of causally engendering violence, etc... [...]

[...] The a priori irrational part is where, in the course of the argument, someone pulls out a dictionary and looks up the definition of "atheism" or "religion". [...] How could a dictionary possibly decide whether an empirical cluster of atheists is really substantially different from an empirical cluster of theologians? How can reality vary with the meaning of a word? The points in thingspace don't move around when we redraw a boundary. [bolding mine—ZMD]

But people often don't realize that their argument about where to draw a definitional boundary, is really a dispute over whether to infer a characteristic shared by most things inside an empirical cluster...

I claim that what Yudkowsky said about the irrationality about appealing to the dictionary, goes the same for appeal to personal values or priorities. It's not false exactly, but it doesn't accomplish anything.

Suppose Bob says, "Abortion is murder, because it's the killing of a human being!"

Alice says, "No, abortion isn't murder, because murder is the killing of a sentient being, and fetuses aren't sentient."

As Alice and Bob's hired rationalist mediator, you could say, "You two just have different preferences about somewhat-arbitary category boundaries, that's all! Abortion is murder-with-respect-to-Bob's-definition, but it isn't murder-with-respect-to-Alice's-definition. Done! End of conversation!"

And maybe sometimes there really is nothing more to it than that. But oftentimes, I think we can do more work to break the symmetry: to work out what different predictions Alice and Bob are making about reality, or what different preferences they have about reality, and refocus the discussion on that. As I wrote in "The Categories Were Made for Man to Make Predictions":

If different political factions are engaged in conflict over how to define the extension of some common word—common words being a scarce and valuable resource both culturally and information-theoretically—rationalists may not be able to say that one side is simply right and the other is simply wrong, but we can at least strive for objectivity in describing the conflict. Before shrugging and saying, "Well, this is a difference in values; nothing more to be said about it," we can talk about the detailed consequences of what is gained or lost by paying attention to some differences and ignoring others.

We had an entire Sequence specifically about this! You were there! I was there! Why doesn't anyone remember?!

Comment by zack_m_davis on Blegg Mode · 2019-03-15T23:09:20.281Z · score: 3 (2 votes) · LW · GW

I don't see bullets on Firefox 65.0.1, but I do on Chromium 72.0.3626.121 (both Xubuntu 16.04.5).

Comment by zack_m_davis on Blegg Mode · 2019-03-15T23:06:08.839Z · score: 4 (2 votes) · LW · GW

I think that this allegory misses crucial aspects of the original situation

That makes sense! As gjm noted, sometimes unscrupulous authors sneakily construct an allegory with the intent of leading the reader to a particular conclusion within the context of the allegory with the hope that the reader will map that conclusion back onto the real-world situation in a particular way, without doing the work of actually showing that the allegory and the real-world situation are actually analogous in the relevant aspects.

I don't want to be guilty of that! This is a story about bleggs and rubes that I happened to come up with in the context of trying to think about something else (and I don't want to be deceptive about that historical fact), but I definitely agree that people shouldn't map the story onto some other situation unless they actually have a good argument for why that mapping makes sense. If we wanted to discuss the something else rather than the bleggs and rubes, we should do that on someone else's website. Not here.

Comment by zack_m_davis on Blegg Mode · 2019-03-15T07:42:47.336Z · score: 2 (1 votes) · LW · GW

your argument seems to rely purely on the intuition of your fictional character

Yes, the dependence on intuition is definitely a weakness of this particular post. (I wish I knew as much math as Jessica Taylor! If I want to become stronger, I'll have to figure out how fit more studying into my schedule!)

you seem to assume that categories are non-overlapping.
you seem to assume that categories are crisp rather than fuzzy

I don't believe either of those things. If you have any specific wording suggestions on how I can write more clearly so as to better communicate to my readers that I don't believe either of those things, I'm listening.

If you take a table made out of a block of wood, and start to gradually deform its shape until it becomes perfectly spherical, is there an exact point when it is no longer called a "table"?

No, there is no such exact point; like many longtime Less Wrong readers, I, too, am familiar with the Sorities paradox.

But in any case, the categorization would depend on the particular trade-offs that the designers of the production line made (depending on things like, how expensive is it to run the palladium scanner)

Right. Another example of one of the things the particular algorithm-design trade-offs will depend on is the distribution of objects.

We could imagine a slightly altered parable in which the frequency distribution of objects is much more evenly spread out in color–shape–metal-content space: while cubeness has a reasonably strong correlation with redness and palladium yield, and eggness with blueness and vanadium yield, you still have a substantial fraction of non-modal objects: bluish-purple rounded cubes, reddish-purple squarish eggs, &c.

In that scenario, a natural-language summary of the optimal decision algorithm wouldn't talk about discrete categories: you'd probably want some kind of scoring algorithm with thresholds for various tests and decisions as you describe, and no matter where you set the threshold for each decision, you'd still see a lot of objects just on either side of the boundary, with no good "joint" to anchor the placement of a category boundary.

In contrast, my reading of Yudkowsky's original parable posits a much sparser, more tightly-clustered distribution of objects in configuration space. The objects do vary somewhat (some bleggs are purple, some rubes contain vanadium), but there's a very clear cluster-structure: virtually all objects are close to the center of—and could be said to "belong to"—either the "rube" cluster or the "blegg" cluster, with a lot of empty space in between.

In this scenario, I think it does make sense for a natural-language summary of the optimal decision algorithm to talk about two distinct "categories" where the density in the configuration space is concentrated. Platonic essences are just the limiting case as the overlap between clusters goes to zero.

In my fanfiction, I imagine that some unknown entity has taken objects that were originally in the "rube" cluster, and modified them so that they appear, at first glance but not on closer inspection, to be members of the "blegg" cluster. At first, the protagonist wishes to respect the apparent intent of the unknown entity by considering the modified objects to be bleggs. But in the process of her sorting work, the protagonist finds herself wanting to mentally distinguish adapted bleggs from regular bleggs, because she can't make the same job-relevant probabilistic inferences with the new "bleggs (either regular or adapted)" concept as she could with the old "bleggs (only standard bleggs)" concept.

To see why, forget about the category labels for a moment and just consider the clusters in the six-dimensional color–shape–texture–firmness–luminesence–metal-content configuration space.

Before the unknown entity's intervention, we had two distinct clusters: one centered at {blue, egg, furry, flexible, luminescent, vanadium}, and another centered at {red, cube, smooth, hard, non-luminescent, palladium}.

After the unknown entity's intervention, we have three distinct clusters: the two previously-existing clusters, and a new cluster centered at {blue, egg, furry, hard, non-luminescent, palladium}. This is a different situation! Workers on the sorting line might want different language in order to describe this new reality!

Now, if we were to project into the three-dimensional color–shape–texture subspace, then we would have two clusters again: with just these attributes, we can't distinguish between bleggs and adapted bleggs. But since workers on the sorting line can observe hardness, and care about metal content, they probably want to use the three-cluster representation, even if they suspect the unknown entity might thereby feel disrespected.

Comment by zack_m_davis on Blegg Mode · 2019-03-14T06:00:46.856Z · score: 3 (2 votes) · LW · GW

I share jessicata's feeling that the best set of concepts to work with may not be very sensitive to what's easy to detect. [...] there doesn't seem to be a general pattern of basing that refinement on the existence of convenient detectable features

Yeah, I might have been on the wrong track there. (Jessica's comment is great! I need to study more!)

I am concerned that we are teetering on the brink of -- if we have not already fallen into -- exactly the sort of object-level political/ideological/personal argument that I was worried about

I think we're a safe distance from the brink.

Words like "nefarious" and "terrorist" seem like a warning sign

"Nefarious" admittedly probably was a high-emotional-temperature warning sign (oops), but in this case, "I don't negotiate with terrorists" is mostly functioning as the standard stock phrase to evoke the timeless-decision-theoretic "don't be extortable" game-theory intuition, which I don't think should count as a warning sign, because it would be harder to communicate if people had to avoid genuinely useful metaphors because they happened to use high-emotional-valence words.

Comment by zack_m_davis on Blegg Mode · 2019-03-14T05:32:24.759Z · score: 6 (3 votes) · LW · GW

But it seems to me that that the relative merits of these depend on the agent's goals, and the best categorization to adopt may be quite different depending on whether you're [...] and also on your own values and priorities.

Yes, I agree! (And furthermore, the same person might use different categorizations at different times depending on what particular aspects of reality are most relevant to the task at hand.)

But given an agent's goals in a particular situation, I think it would be a shocking coincidence for it to be the case that "there are [...] multiple roughly-equally-good categorizations." Why would that happen often?

If I want to use sortable objects as modern art sculptures to decorate my living room, then the relevant features are shape and color, and I want to think about rubes and bleggs (and count adapted bleggs as bleggs). If I also care about how the room looks in the dark and adapted bleggs don't glow in the dark like ordinary bleggs do, then I want to think about adapted bleggs as being different from ordinary bleggs.

If I'm running a factory that harvests sortable objects for their metal content and my sorting scanner is expensive to run, then I want to think about rubes and ordinary bleggs (because I can infer metal content with acceptably high probability by observing the shape and color of these objects), but I want to look out for adapted bleggs (because their metal content is, with high probability, not what I would expect based on the color/shape/metal-content generalizations I learned from my observations of rubes and ordinary bleggs). If the factory invests in a new state-of-the-art sorting scanner that can be cheaply run on every object, then I don't have any reason to care about shape or color anymore—I just care about palladium-cored objects and vanadium-cored objects.

and picking one of those rather than another is not an epistemological error.

If you're really somehow in a situation where there are multiple roughly-equally-good categorizations with respect to your goals and the information you have, then I agree that picking one of those rather than another isn't an epistemological error. Google Maps and MapQuest are not exactly the same map, but if you just want to drive somewhere, they both reflect the territory pretty well: it probably doesn't matter which one you use. Faced with an arbitrary choice, you should make an arbitrary choice: flip a coin, or call random.random().

And yet somehow, I never run into people who say, "Categories are somewhat arbitrary, therefore you might as well roll a d3 to decide whether to say 'trans women are women' or 'so-called "trans women" are men' or 'transwomen are transwomen', because each of these maps is doing a roughly-equally-good job of reflecting the relevant aspects of the territory." But I run into lots of people who say, "Categories are somewhat arbitrary, therefore I'm not wrong to insist that trans women are women," and who somehow never seem to find it useful to bring up the idea that categories are somewhat arbitrary in seemingly any other context.

You see the problem? If the one has some sort of specific argument for why I should use a particular categorization system in a particular situation, then that's great, and I want to hear it! But it has to be an argument and not a selectively-invoked appeal-to-arbitrariness conversation-halter.

Comment by zack_m_davis on Blegg Mode · 2019-03-14T01:50:43.077Z · score: 5 (3 votes) · LW · GW

Any "categories" you introduce here are at best helpful heuristics, with no deep philosophical significance.

I mean, yes, but I was imagining that there would be some deep philosophy about how computationally bounded agents should construct optimally helpful heuristics.

Comment by zack_m_davis on Blegg Mode · 2019-03-13T06:20:40.132Z · score: 4 (3 votes) · LW · GW

Trying to think of some examples, it seems to me that what matters is simply the presence of features that are "decision-relevant with respect to the agent's goals". [...]

So, I think my motivation (which didn't make it into the parable) for the "cheap to detect features that correlate with decision-relevant expensive to detect features" heuristic is that I'm thinking in terms of naïve Bayes models. You imagine a "star-shaped" causal graph with a central node (whose various values represent the possible categories you might want to assign an entity to), with arrows pointing to various other nodes (which represent various features of the entity). (That is, we're assuming that the features of the entity are conditionally independent given category membership: P(X|C) = Π_i P(X_i|C).) Then when we observe some subset of features, we can use that to update our probabilities of category-membership, and use that to update our probabilities of the features we haven't observed yet. The "category" node doesn't actually "exist" out there in the world—its something we construct to help factorize our probability distribution over the features (which do "exist").

So, as AI designers, we're faced with the question of how we want the "category" node to work. I'm pretty sure there's going to be a mathematically correct answer to this that I just don't know (yet) because I don't study enough and haven't gotten to Chapter 17 of Daphne Koller and the Methods of Rationality. Since I'm not there yet, if I just take at intuitive amateur guess at how I might expect this to work, it seems pretty intuitively plausible that we're going to want the category node to be especially sensitive to cheap-to-observe features that correlate with goal-relevant features? Like, yes, we ultimately just want to know as much as possible about the decision-relevant variables, but if some observations are more expensive to make than others, that seems like the sort of thing the network should be able to take into account, right??

Remember those 2% of otherwise ordinary bleggs that contain palladium? Personally, I'd want a category for those

I agree that "things that look like 'bleggs' that contain palladium" is a concept that you want to be able to think about. (I just described it in words, therefore it's representable!) But while working on the sorting line, your visual system's pattern-matching faculties aren't going to spontaneously invent "palladium-containing bleggs" as a thing to look out for if you don't know any way to detect them, whereas if adapted bleggs tend to look different in ways you can see, then that category is something your brain might just "learn from experience." In terms of the naïve Bayes model, I'm sort of assuming that the 2% of palladium containing non-adapted bleggs are "flukes": that variable takes that value with that probability independently of the other blegg features. I agree that if that assumption were wrong, then that would be really valuable information, and if you suspect that assumption is wrong, then you should definitely be on the lookout for ways to spot palladium-containing bleggs.

But like, see this thing I'm at least trying to do here, where I think there's learnable statistical structure in the world that I want to describe using language? That's pretty important! I can totally see how, from your perspective, on certain object-level applications, you might suspect that the one who says, "Hey! Categories aren't even 'somewhat' arbitrary! There's learnable statistical structure in the world; that's what categories are for!" is secretly being driven by nefarious political motivations. But I hope you can also see how, from my perspective, I might suspect that the one who says, "Categories are somewhat arbitrary; the one who says otherwise is secretly being driven by nefarious political motivations" is secretly being driven by political motivations that have pretty nefarious consequences for people like me trying to use language to reason about the most important thing in my life, even if the psychological foundation of the political motivation is entirely kindhearted.

Comment by zack_m_davis on Editor Mini-Guide · 2019-03-13T04:48:45.825Z · score: 2 (1 votes) · LW · GW

This comment is a test! I prefer the plain-Markdown editor, but I want to try using LaTeX for one comment, so I temporarily swapped my user settings to the "rich" editor so that I could try out the LaTeX editor here. Then after submitting this comment, I'll switch my settings _back_ to Markdown, and edit this comment to see what the syntax is for using LaTeX from that editor (wrap it in $s, maybe?), which didn't seem to be explained in the post above? I would be pretty disappointed if it were to turn out that there's no way to do LaTeX from the Markdown editor, but I would also be somewhat surprised.

Edit: even after switching my settings back, editing this comment gives me the rich editor? So ... I guess individual comments are saved on a per-editor basis, with no translation? I'll concede that that makes sense from a technical standpoint, but it's somewhat disappointing.

Comment by zack_m_davis on Blegg Mode · 2019-03-13T03:05:45.142Z · score: 9 (5 votes) · LW · GW

It seems relevant here that Zack pretty much agreed with my description: see his comments using terms like "deniable allegory", "get away with it", etc.

So, from my perspective, I'm facing a pretty difficult writing problem here! (See my reply to Dagon.) I agree that we don't want Less Wrong to be a politicized space. On the other hand, I also think that a lot of self-identified rationalists are making a politically-motivated epistemology error in asserting category boundaries to be somewhat arbitrary, and it's kind of difficult to address what I claim is the error without even so much as alluding to the object-level situation that I think is motivating the error! For the long, object-level discussion, see my reply to Scott Alexander, "The Categories Were Made for Man To Make Predictions". (Sorry if the byline mismatch causes confusion; I'm using a pen name for that blog.) I didn't want to share "... To Make Predictions" on Less Wrong (er, at least not as a top-level post), because that clearly would be too political. But I thought the "Blegg Mode" parable was sufficiently sanitized such that it would be OK to share as a link post here?

I confess that I didn't put a lot of thought into the description text which you thought was disingenuous. I don't think I was being consciously disingenuous (bad intent is a disposition, not a feeling!), but after you pointed it out, I do see your point that, since there is some unavoidable political context here, it's probably better to explicitly label that, because readers who had a prior expectation that no such context would exist would feel misled upon discovering it. So I added the "Content notice" to the description. Hopefully that addresses the concern?

our categories are [...] somewhat arbitrary

No! Categories are not "somewhat arbitrary"! There is structure in the world, and intelligent agents need categories that carve the structure at the joints so that they can make efficient probabilistic inferences about the variables they're trying to optimize! "Even if you cannot do the math, knowing that the math exists tells you that the dance step is precise and has no room in it for your whims." We had a whole Sequence about this! Doesn't anyone else remember?!

Trans-ness is not always "cheap to detect". I guess it's cheaper to detect than, say, sex chromosomes. OK -- and how often are another person's sex chromosomes "decision-relevant with respect to the agent's goals"?

You seem to be making some assumptions about which parts of the parable are getting mapped to which parts of the real-world issue that obviously inspired the parable. I don't think this is the correct venue for me to discuss the real-world issue. On this website, under this byline, I'd rather only talk about bleggs and rubes—even if you were correct to point out that it would be disingenuous for someone to expect readers to pretend not to notice the real-world reason that we're talking about bleggs and rubes. With this in mind, I'll respond below to a modified version of part of your comment (with edits bracketed).

I guess it's cheaper to detect than, say, [palladium or vanadium content]. OK -- and how often [is a sortable object's metal content] "decision-relevant with respect to the agent's goals"? Pretty much only if [you work in the sorting factory.] [That's] fairly uncommon -- for most of us, very few of the [sortable objects] we interact with [need to be sorted into bins according to metal content].

Sure! But reality is very high-dimensional—bleggs and rubes have other properties besides color, shape, and metal content—for example, the properties of being flexible-vs.-hard or luminescent-vs.-non-luminescent, as well as many others that didn't make it into the parable. If you care about making accurate predictions about the many properties of sortable objects that you can't immediately observe, then how you draw your category boundaries matters, because your brain is going to be using the category membership you assigned in order to derive your prior expectations about the variables that you haven't yet observed.

sex chromosomes, which is exactly the "expensive" feature the author identifies in the case of trans people.

The author did no such thing! It's epistemology fiction about bleggs and rubes! It's true that I came up with the parable while I was trying to think carefully about transgender stuff that was of direct and intense personal relevance to me. It's true that it would be disingenuous for someone to expect readers to not-notice that I was trying to think about trans issues. (I mean, it's in the URL.) But I didn't say anything about chromosomes! "If confusion threatens when you interpret a metaphor as a metaphor, try taking everything completely literally."

Trying to think of some examples, it seems to me that what matters is simply the presence of features that are "decision-relevant with respect to the agent's goals". [...]

Thanks for this substantive, on-topic criticism! I would want to think some more before deciding how to reply to this.

ADDENDUM: I thought some more and wrote a sister comment.

Comment by zack_m_davis on Blegg Mode · 2019-03-13T01:17:11.256Z · score: 5 (3 votes) · LW · GW

unless [...] categorization somehow is important to LW posts

Categorization is hugely relevant to Less Wrong! We had a whole Sequence about this!

Of course, it would be preferable to talk about the epistemology of categories with non-distracting examples if at all possible. One traditional strategy for avoiding such distractions is to abstract the meta-level point one is trying to make into a fictional parable about non-distracting things. See, for example, Scott Alexander's "A Parable on Obsolete Ideologies", which isn't actually about Nazism—or rather, I would say, is about something more general than Nazism.

Unfortunately, this is extremely challenging to do well—most writers who attempt this strategy fail to be subtle enough, and the parable falls flat. For this they deserve to be downvoted.

Comment by zack_m_davis on Blegg Mode · 2019-03-12T15:50:59.043Z · score: 11 (5 votes) · LW · GW

better to be up front about them

... you're right. (I like the aesthetics of the "deniable allegory" writing style, but delusionally expecting to get away with it is trying to have one's cake and eat it, too.) I added a "Content notice" to the description here.

Comment by zack_m_davis on Blegg Mode · 2019-03-12T14:46:52.302Z · score: 6 (3 votes) · LW · GW

Thanks. In retrospect, possibly a better approach for this venue would have been to carefully rewrite the piece for Less Wrong in a way that strips more subtext/conceals more of the elephant (e.g., cut the "disrespecting that effort" paragraph).

Comment by zack_m_davis on Blegg Mode · 2019-03-12T03:55:35.391Z · score: 3 (2 votes) · LW · GW

Can you say more? What should the description say instead? (I'm guessing you're referring to the fact that the post has some subtext that probably isn't a good topic fit for Less Wrong? But I would argue that the text (using the blegg/rube parable setting to make another point about the cognitive function of categorization) totally is relevant and potentially interesting!)

Blegg Mode

2019-03-11T15:04:20.136Z · score: 18 (13 votes)
Comment by zack_m_davis on Verbal Zendo · 2018-10-22T04:48:33.927Z · score: 1 (1 votes) · LW · GW

Another fun programming exercise is to do the other direction: have the user come up with a rule, and make the program come up with examples to try to test its hypotheses. (You want the program to generate examples that falsify half the remaining probability-mass.)

Comment by zack_m_davis on Open thread, June 5 - June 11, 2017 · 2017-06-11T19:58:20.627Z · score: 2 (2 votes) · LW · GW

What specific bad things would you expect to happen if the post was left up, with what probabilities? (I'm aware of the standard practice of not discussing ongoing legal cases, but have my doubts about whether allowing the legal system to operate under conditions of secrecy actually makes things better on net.)

Comment by zack_m_davis on Open thread, June 5 - June 11, 2017 · 2017-06-08T17:54:05.301Z · score: 1 (1 votes) · LW · GW

The mature way to handle suicidal people is to call professional help, as soon as possible.

It's worth noting that this creates an incentive to never talk about your problems.

My advice for people who value not being kidnapped and forcibly drugged by unaccountable authority figures who won't listen to reason is to never voluntarily talk to psychiatrists, for the same reason you should never talk to cops.

Comment by zack_m_davis on Open thread, June 5 - June 11, 2017 · 2017-06-08T03:54:39.643Z · score: 3 (3 votes) · LW · GW

I corresponded with sad_dolphin. It added a little bit of gloom to my day, but I don't regret doing it: having suffered from similar psychological problems in the past, I want to be there with my hard-won expertise for people working through the same questions. I agree that most people who talk about suicide in such a manner are unlikely to go through with it, but that doesn't mean they're not being subjectively sincere. I'd rather such cries for help not be disincentivized here (as you seem to be trying to do); I'd rather people be able to seek and receive support from people who actually understand their ideas, rather than callously foisted off onto alleged "experts" who don't understand.

Comment by zack_m_davis on Bet or update: fixing the will-to-wager assumption · 2017-06-08T03:19:18.207Z · score: 0 (0 votes) · LW · GW

But I'm being told that this is "meta-uncertainty" which right-thinking Bayesians are not supposed to have.

Hm. Maybe those people are wrong??

Clearly not since the normal distribution goes from negative infinity to positive infinity

That's right; I should have either said "approximately", or chosen a different distribution.

That 0.5 is conditional on the distribution of r, isn't it? That makes it not a different question at all.

Yes, it is averaging over your distribution for _r_. Does it help if you think of probability as relative to subjective states of knowledge?

Can you elaborate?

(Attempted humorous allusion to how Cox's theorem derives probability theory from simple axioms about how reasoning under uncertainty should work, less relevant if no one is talking about inventing new fields of math.)

Comment by zack_m_davis on Bet or update: fixing the will-to-wager assumption · 2017-06-07T23:45:53.184Z · score: 4 (4 votes) · LW · GW

As I said, I want a richer way to talk about probabilities, more complex than taking them as simple scalars. Do you think it's a bad idea?

That's right, I think it's a bad idea: it sounds like what you actually want is a richer way to talk about your beliefs about Coin 2, but you can do that using standard probability theory, without needing to invent a new field of math from scratch.

Suppose you think Coin 2 is biased and lands heads some unknown fraction _r_ of the time. Your uncertainty about the parameter _r_ will be represented by a probability distribution: say it's normally distributed with a mean of 0.5 and a standard deviation of 0.1. The point is, the probability of _r_ having a particular value is a different question from the the probability of getting heads on your first toss of Coin 2, which is still 0.5. You'd have to ask a different question than "What is the probability of heads on the first flip?" if you want the answer to distinguish the two coins. For example, the probability of getting exactly _k_ heads in _n_ flips is C(_n_, _k_)(0.5)^_k_(0.5)^(_n_−_k_) for Coin 1, but (I think?) ∫₀¹ (1/√(0.02π))_e_^−((_p_−0.5)^2/0.02) C(_n_, _k_)(_p_)^_k_(_p_)^(_n_−_k_) _dp_ for Coin 2.

Does St.Bayes frown upon it?

St. Cox probably does.

Comment by zack_m_davis on Bet or update: fixing the will-to-wager assumption · 2017-06-07T19:49:30.595Z · score: 4 (4 votes) · LW · GW

Can my utility function include risk aversion?

That would be missing the point. The vNM theorem says that if you have preferences over "lotteries" (probability distributions over outcomes; like, 20% chance of winning $5 and 80% chance of winning $10) that satisfy the axioms, then your decisionmaking can be represented as maximizing expected utility for some utility function over outcomes. The concept of "risk aversion" is about how you react to uncertainty (how you decide between lotteries) and is embodied in the utility function; it doesn't apply to outcomes known with certainty. (How risk-averse are you about winning $5?)

See "The Allais Paradox" for how this was covered in the vaunted Sequences.

In my hypothetical the two 50% probabilites are different. I want to express the difference between them. There are no sequences involved.

Obviously you're allowed to have different beliefs about Coin 1 and Coin 2, which could be expressed in many ways. But your different beliefs about the coins don't need to show up in your probability for a single coinflip. The reason for mentioning sequences of flips, is because that's when your beliefs about Coin 1 vs. Coin 2 would start making different predictions.

Comment by zack_m_davis on Birth of a Stereotype · 2017-06-05T22:52:02.195Z · score: 0 (0 votes) · LW · GW

Correlations are symmetric, but is evidence for may not be (depending on how you interpret the phrase): P(A|B) ≠ P(B|A) (unless P(A) == P(B)).

Comment by zack_m_davis on A Comment on Expected Utility Theory · 2017-06-05T04:59:32.685Z · score: 3 (3 votes) · LW · GW

Expected utility is not the same thing as expected dollars. As AgentStonecutter explained to you on Reddit last month, the standard assumption of diminishing marginal utility of money is entirely sufficient to account for preferring the guaranteed $250,000; no need to patch standard decision theory. (The von Neumann–Morgenstern theorem doesn't depend on decisions being repeated; if you want to escape your decisions being describable as the maximization of some utility function, you have to reject one of the axioms, even if your decision is the only decision in the universe.)

Change

2017-05-06T21:17:45.731Z · score: 1 (1 votes)

An Intuition on the Bayes-Structural Justification for Free Speech Norms

2017-03-09T03:15:30.674Z · score: 5 (6 votes)

Dreaming of Political Bayescraft

2017-03-06T20:41:16.658Z · score: 1 (1 votes)

Rationality Quotes January 2010

2010-01-07T09:36:05.162Z · score: 3 (6 votes)

News: Improbable Coincidence Slows LHC Repairs

2009-11-06T07:24:31.000Z · score: 7 (8 votes)