Posts

Less Wrong Poetry Corner: Walter Raleigh's "The Lie" 2020-01-04T22:22:56.820Z · score: 20 (12 votes)
Don't Double-Crux With Suicide Rock 2020-01-01T19:02:55.707Z · score: 68 (19 votes)
Speaking Truth to Power Is a Schelling Point 2019-12-30T06:12:38.637Z · score: 52 (14 votes)
Stupidity and Dishonesty Explain Each Other Away 2019-12-28T19:21:52.198Z · score: 35 (15 votes)
Firming Up Not-Lying Around Its Edge-Cases Is Less Broadly Useful Than One Might Initially Think 2019-12-27T05:09:22.546Z · score: 94 (33 votes)
Funk-tunul's Legacy; Or, The Legend of the Extortion War 2019-12-24T09:29:51.536Z · score: 13 (17 votes)
Free Speech and Triskaidekaphobic Calculators: A Reply to Hubinger on the Relevance of Public Online Discussion to Existential Risk 2019-12-21T00:49:02.862Z · score: 63 (23 votes)
A Theory of Pervasive Error 2019-11-26T07:27:12.328Z · score: 21 (7 votes)
Relevance Norms; Or, Gricean Implicature Queers the Decoupling/Contextualizing Binary 2019-11-22T06:18:59.497Z · score: 72 (22 votes)
Algorithms of Deception! 2019-10-19T18:04:17.975Z · score: 17 (6 votes)
Maybe Lying Doesn't Exist 2019-10-14T07:04:10.032Z · score: 56 (27 votes)
Heads I Win, Tails?—Never Heard of Her; Or, Selective Reporting and the Tragedy of the Green Rationalists 2019-09-24T04:12:07.560Z · score: 202 (68 votes)
Schelling Categories, and Simple Membership Tests 2019-08-26T02:43:53.347Z · score: 52 (19 votes)
Diagnosis: Russell Aphasia 2019-08-06T04:43:30.359Z · score: 47 (13 votes)
Being Wrong Doesn't Mean You're Stupid and Bad (Probably) 2019-06-29T23:58:09.105Z · score: 16 (11 votes)
What does the word "collaborative" mean in the phrase "collaborative truthseeking"? 2019-06-26T05:26:42.295Z · score: 27 (7 votes)
The Univariate Fallacy 2019-06-15T21:43:14.315Z · score: 27 (11 votes)
No, it's not The Incentives—it's you 2019-06-11T07:09:16.405Z · score: 89 (30 votes)
"But It Doesn't Matter" 2019-06-01T02:06:30.624Z · score: 47 (31 votes)
Minimax Search and the Structure of Cognition! 2019-05-20T05:25:35.699Z · score: 15 (6 votes)
Where to Draw the Boundaries? 2019-04-13T21:34:30.129Z · score: 84 (36 votes)
Blegg Mode 2019-03-11T15:04:20.136Z · score: 18 (13 votes)
Change 2017-05-06T21:17:45.731Z · score: 1 (1 votes)
An Intuition on the Bayes-Structural Justification for Free Speech Norms 2017-03-09T03:15:30.674Z · score: 4 (8 votes)
Dreaming of Political Bayescraft 2017-03-06T20:41:16.658Z · score: 9 (3 votes)
Rationality Quotes January 2010 2010-01-07T09:36:05.162Z · score: 3 (6 votes)
News: Improbable Coincidence Slows LHC Repairs 2009-11-06T07:24:31.000Z · score: 7 (8 votes)

Comments

Comment by zack_m_davis on Preliminary thoughts on moral weight · 2020-01-13T03:12:24.596Z · score: 14 (8 votes) · LW · GW

This kind of thinking actively drives me and many others I know away from LW/EA/Rationality

And that kind of thinking (appeal to the consequence of repelling this-and-such kind of person away from some alleged "community") has been actively driving me away. I wonder if there's some way to get people to stop ontologizing "the community" and thereby reduce the perceived need to fight for control of the "LW"/"EA"/"rationalist" brand names? (I need to figure out how to stop ontologizing, because I'm exhausted from fighting.) Insofar as "rationality" is a thing, it's something that Luke-like optimization processes and Zvi-like optimization processes are trying to approximate, not something they're trying to fight over.

Comment by zack_m_davis on Realism about rationality · 2020-01-12T19:31:23.741Z · score: 4 (2 votes) · LW · GW

I struggle to name a way that evolution affects an everyday person (ignoring irrelevant things like atheism-religion debates).

Evolutionary psychology?

Comment by zack_m_davis on Free Speech and Triskaidekaphobic Calculators: A Reply to Hubinger on the Relevance of Public Online Discussion to Existential Risk · 2020-01-06T08:19:38.850Z · score: 29 (5 votes) · LW · GW

when you talk to them in private they get everything 100% right.

I'm happy for them, but I thought the point of having taxpayer-funded academic departments was so that people who aren't insider experts can have accurate information with which to inform decisions? Getting the right answer in private can only help those you talk to in private.

I also don't think we live in the world where everyone has infinite amounts of slack to burn endorsing taboo ideas and nothing can possibly go wrong.

Can you think of any ways something could possibly go wrong if our collective map of how humans work fails to reflect the territory?

(I drafted a vicious and hilarious comment about one thing that could go wrong, but I fear that site culture demands that I withhold it.)

Comment by zack_m_davis on Less Wrong Poetry Corner: Walter Raleigh's "The Lie" · 2020-01-06T06:41:25.356Z · score: 2 (1 votes) · LW · GW

Oh, sorry, I wasn't trying to offer a legal opinion; I was just trying to convey worldview-material while riffing off your characterization of "defrauding everyone about the El Dorado thing."

Comment by zack_m_davis on Less Wrong Poetry Corner: Walter Raleigh's "The Lie" · 2020-01-06T02:32:44.950Z · score: 21 (3 votes) · LW · GW

Would your views on speaking truth to power change if the truth were 2x as offensive as you currently think it is? 10x? 100x?

For some multiplier, yes. (I don't know what the multiplier is.) If potentates would murder me on the spot unless I deny that that they live acting by others' action, and affirm that they are loved even if they don't give and are strong independently of a faction, then I will say those things in order to not be murdered on the spot.

I guess I need to clarify something: I tend to talk about this stuff in the language of virtues and principles rather than the language of consequentialism, not because I think the language of virtues and principles is literally true as AI theory, but because humans can't use consequentialism for this kind of thing. Some part of your brain is performing some computation that, if it works, to the extent that it works, is mirroring Bayesian decision theory. But that doesn't help the part of you can that talk, that can be reached by the part of me that can talk.

"Speak the truth, even if your voice trembles" isn't a literal executable decision procedure—if you programmed your AI that way, it might get stabbed. But a culture that has "Speak the truth, even if your voice trembles" as a slogan might—just might be able to do science or better—to get the goddamned right answer even when the local analogue of the Pope doesn't like it. I falsifiably predict that a culture that has "Use Bayesian decision theory to decide whether or not to speak the truth" as its slogan won't be able to do science—Platonically, the math has to exist, but letting humans appeal to Platonic math whenever they want is just too convenient of an excuse.

Would your views on speaking truth to power change if the truth were 2x less expensive as you currently think it is? 10x? 100x? I falsifiably predict that your answer is "Yes." Followup question: have you considered performing an experiment to test whether the consequences of speech are as dire as you currently think? I think I have more data than you! (We probably mostly read the same blogs, but I've done field work.)

(If so, are you sure that's not why you don't think the truth is more offensive than you currently think it is?)

Great question! No, I'm not sure. But if my current view is less wrong than the mainstream, I expect to do good by talking about it, even if there exists an even better theory that I wouldn't be brave enough to talk about.

Immaterial souls are stabbed all the time in the sense that their opinions are discredited.

Can you be a little more specific? "Discredited" is a two-place function (discredited to whom).

Comment by zack_m_davis on Less Wrong Poetry Corner: Walter Raleigh's "The Lie" · 2020-01-05T03:15:36.675Z · score: 37 (8 votes) · LW · GW

I think the Vassarian–Taylorist conflict–mistake synthesis moral is that in order to perform its function, the English court system needs to be able to punish Raleigh for "fraud" on the basis of his actions relative what he knew or could have reasonably been expected to know, even while Raleigh is subjectively the hero of his own story and a sympathetic psychologist could eloquently and truthfully explain how easy it was for him to talk himself into a biased narrative.

Where mistake theorists treat politics as "science, engineering, or medicine" and conflict theorists treat politics as war, this view treats politics as evolutionary game theory: the unfolding over time of a population of many dumb, small agents executing strategies, forming coalitions, occasionally switching strategies to imitate those that are more successful in the local environment, &c. The synthesis view is mistake-theoretic insofar as the little agents are understood to be playing far from optimally and could do much better if they were smarter, but conflict-theoretic insofar as the games being played have large zero-sum components and you mostly can't take the things the little agents say literally. The "mistakes" aren't random and not easily fixable with more information (in contrast to how if I said 57 was prime and you said "But 3 × 19", I would immediately say "Oops"), but rather arise from the strategies being executed: it's not a coincidence that Raleigh talked himself into a narrative where he was lone angel who would discover limitless gold.

Agents select beliefs on the basis of either their being true (and therefore useful for navigating the world) or because they successfully deceive other agents into mis-navigating the world in a way that benefits the belief-holder. "Be more charitable to other people" isn't necessarily great advice in general, because while sometimes other agents have useful true information to offer (Raleigh's The Discovery of Guiana "includes some material of a factual nature"), it's hard to distinguish from misinformation that was optimized to benefit the agents who propogate it (Discovery of Guiana also says you should invest in Raleigh's second expedition).

Mistake theorists think conflict theorists are making a mistake; conflict theorists think mistake theorists are the enemy. Evolutionary game theorists think that conflict theorists are executing strategies adapted to an environment predominated by zero-sum games, and that mistake theorists are executing strategies adapted to an environment containing cooperative games (where the existence of a mechanism for externally enforcing agreements, like a court system, aligns incentives and thereby makes it easier to propogate true infromation).

Comment by zack_m_davis on Meta-discussion from "Circling as Cousin to Rationality" · 2020-01-04T21:05:17.167Z · score: 4 (2 votes) · LW · GW

When designing norms, we should take into account an asymmetry between reading and writing: each comment is only written once, but read many times. Each norm imposed on writers to not be unduly annoying constrains the information flow of the forum much more than each norm imposed on readers to not be unduly annoyed.

Comment by zack_m_davis on Meta-discussion from "Circling as Cousin to Rationality" · 2020-01-04T17:05:40.492Z · score: 16 (7 votes) · LW · GW

Why should Said be the one to change, though? Maybe relatively subtle tweaks to your reading style could make a big difference.

A "surprised bafflement" tone is often seen as a social attack because it's perceived as implying, "You should know this already, therefore I'm surprised that you don't, therefore I should have higher status than you." But that's not the only possible narrative. What happens if you reframe your reaction as, "He's surprised, but surprise is the measure of a poor hypothesis—the fact that he's so cluelessly self-centered as to not be able to predict what other people know means that I should have higher status"?

Comment by zack_m_davis on Meta-discussion from "Circling as Cousin to Rationality" · 2020-01-04T02:48:08.538Z · score: 33 (11 votes) · LW · GW

What I meant by the word "our" was "the broader context culture-at-large," not Less Wrong or my own personal home culture or anything like that. Apologies, that could've been clearer.

No, I got that, I was just using the opportunity to riff off your "In My Culture" piece[1] while defending Said, who is a super valuable commenter who I think is being treated pretty unfairly in this 133-comment-and-counting meta trainwreck!

Sure, sometimes he's insistent on pressing for rigor in a way could seem "nitpicky" or "dense" to readers who, like me, are more likely to just shrug and say, "Meh, I think I mostly get the gist of what the author is trying to say" rather than honing in on a particular word or phrase and writing a comment asking for clarification.

But that's valuable. I am glad that a website nominally devoted to mastering the hidden Bayesian structure of cognition to the degree of precision required to write a recursively self-improving superintelligence to rule over our entire future lightcone has people whose innate need for rigor is more demanding than my sense of "Meh, I think I mostly get the gist"!


  1. This is actually the second time in four months. Sorry, it writes itself! ↩︎

Comment by zack_m_davis on Meta-discussion from "Circling as Cousin to Rationality" · 2020-01-04T00:57:34.022Z · score: 2 (1 votes) · LW · GW

I also apologize for breaking a lot of the comment-permalinks in that thread

It looks like the post comment-counter is also broken. (The header for this post says "4 comments".)

Comment by zack_m_davis on Meta-discussion from "Circling as Cousin to Rationality" · 2020-01-03T21:54:29.524Z · score: 39 (11 votes) · LW · GW

I note that we, as a culture, have reified a term for this, which is "sealioning."

Perhaps in your culture. In my culture, use of the term "sealioning" is primarily understood as an expression of anti-intellectualism (framing requests for dialogue as aggression).

In my culture, while the need to say "I don't expect engaging with you to be productive, therefore I must decline this and all future requests for dialogue from you" is not unheard of, it is seen as a sad and unusual occasion—definitely not something meriting a short codeword with connotations of contempt.

Comment by zack_m_davis on Don't Double-Crux With Suicide Rock · 2020-01-02T00:04:38.795Z · score: 14 (5 votes) · LW · GW

Everything I'm saying is definitely symmetric across persons, even if, as an author, I prefer to phrase it in the second person. (A previous post included a clarifying parenthetical to this effect at the end, but this one did not.)

That is, if someone who trusted your rationality noticed that you seemed visibly unmoved by their strongest arguments, they might think that the lack of agreement implies that they should update towards your position, but another possibility is that their trust has been misplaced! If they find themselves living a world of painted rocks where you are one of the rocks, then it may come to pass that protecting the sanctity of their map would require them to master the technique of lonely dissent.

You could argue that my author's artistic preference to phrase things in the second person is misleading, but I'm not sure what to do about that while still accomplishing everything else I'm trying to do with my writing: my reply to Wei Dai and a Reddit user's commentary on another previous post seem relevant.

Comment by zack_m_davis on Meta-discussion from "Circling as Cousin to Rationality" · 2020-01-01T22:27:13.936Z · score: 5 (3 votes) · LW · GW

I agree with your main point that authors are not obligated to respond to comments, but—

I don't know whose judgement you would trust on this, but if I drag Eliezer into this thread, and have him say decisively that the norms of LessWrong should not put an obligation on authors to respond to every question, and to be presumed wrong or ignorant in the absence of a response, would that change your mind on this?

Why would this kind of appeal-to-authority change his mind? (That was a rhetorical question; I wouldn't expect you to reply to this comment unless you really wanted to.) Said thinks his position is justified by normatively correct general principles. If he's wrong, he's wrong because of some counterargument that normative general principles don't actually work like that, not because of Eliezer Yudkowsky's say-so.

Comment by zack_m_davis on Meta-discussion from "Circling as Cousin to Rationality" · 2020-01-01T22:08:34.228Z · score: 7 (3 votes) · LW · GW

There is always an obligation by any author to respond to anyone's comment along these lines. [...] What is the point of posting here, if you're not going to engage with commenters?

Can you clarify what you mean by "along these lines"? Not all comments or commenters are equally worth engaging with (in terms of some idealized "insight per unit effort" metric).

I think I agree that simple questions like "What do you mean by this-and-such word?" are usually not that expensive to answer, but there are times when I write off a comment or commenter as not worth my time, and it can be annoying when someone is being unduly demanding even after a "reasonable" attempt to clarify has been made.

Comment by zack_m_davis on Don't Double-Crux With Suicide Rock · 2020-01-01T20:46:38.669Z · score: 4 (2 votes) · LW · GW

An earlier draft actually specified "... on questions of fact", but I deleted that phrase because I didn't think it was making the exposition stronger. (Omit needless words!) People who understand the fact/value distinction, instrumental goals, &c. usually don't have trouble "relativizing" policy beliefs. (Even if I don't want to maximize paperclips, I can still have a lawful discussion about what the paperclip-maximizing thing to do would be.)

Comment by zack_m_davis on Meta-discussion from "Circling as Cousin to Rationality" · 2020-01-01T17:28:34.111Z · score: 20 (7 votes) · LW · GW

Let us say that these inquiries were usually not answered—either not at all, or not satisfactorily. And now imagine that someone started downvoting these inquiries of yours.

Maybe we should write a post about this kind of conversational dynamic![1]

Alice asks Bob a question. Bob can't answer, either for legitimate or illegitimate[2] reasons, but doesn't want to straightforwardly say, "Sorry, I can't answer that because ..." for fear of losing face in front of the audience, so instead resorts to more opaque stonewalling tactics. Usually, Alice will eventually take a hint and give up. But if she doesn't, we have a high-stakes battle of wills adjudicated by the audience—will Bob be exposed as being phony, or will Alice be derided as a pedant?!


  1. Where by "dynamic", I mean "thingy". ↩︎

  2. A legitimate reason for not being able to answer might be: the question is an isolated demand for rigor, where Bob doesn't have a rigorous formulation of his point, but thinks the non-rigorous version is good enough and should be conversationally "admissible." ↩︎

Comment by zack_m_davis on Speaking Truth to Power Is a Schelling Point · 2020-01-01T05:52:01.398Z · score: 7 (4 votes) · LW · GW

I actually feel okay about letting readers fill in this kind of generalization for themselves? Similarly, in the real world, punishable truths aren't about literal naked Emperors, but I tend to assume most readers are familiar with (or can figure out) the trope of the famous Hans Christian Andersen story being used as an allegory for politically-unfavorable truths in general.

I guess you could argue that my choice of illustrative fictitious examples is algorithmically-dishonestly "rigged": that, as a result of my ongoing "People should be braver about saying stuff!" meta-political campaign, the elephant in my brain knew to generate an example (applying for grants) that would make forthrightness seem like the right choice (piggybacking off of traditional "money is corrupting" moral intuitions), rather than an example that would make conformity seem like the right choice (like protecting one's family)?

I'm not sure what my response to this charge is. The most reliable way of removing such biases would plausibly be to pose everything as an abstract math problem without any illustrative examples, but that seems like it would decrease reading comprehension a lot (and maybe still suffer from the "encouraging hidden agendas" problem, only in the form of assumption choices rather than illustrative-example choices).

I guess in future posts, I could try harder to actively look for illustrative examples that don't narratively support my agenda? (I think I'm already unusually good at this when pursuing what I think of as an "object-level" agenda, but it feels less necessary when pursuing something that I construe as an obvious common interest of many causes, like free speech.) But people often do this social move where they say "I should have tried harder" as a way of accepting blame in exchange for not doing work, so you should only give me credit if you actually see me include counter-narrative examples in future posts; I don't get credit (or as much credit) for merely this comment noticing the problem.

Comment by zack_m_davis on Speaking Truth to Power Is a Schelling Point · 2019-12-31T23:37:20.186Z · score: 4 (2 votes) · LW · GW

Comment of the week goes to hillsump at /r/TheMotte (emphasis mine):

[T]he text is somewhat incoherent. It claims that in-between positions are not sustainable and also that both extremes are Schelling points, yet the title suggests that the truth-telling extreme is the "right" focus point. I happen to share the author's belief that the extremes may be points of attraction, but the claim at the end that they form dual Schelling points needs further evidence. A system with two points of attraction is inherently unstable, negating the feedback cycle that seems necessary for a Schelling point in the first place, and it is not clear why out of band signalling about the current consensus cannot lead to an in-between position as the "obvious" future consensus point. Keeping in mind the paradigmatic Schelling point that people prefer "heads" in a game involving choice between heads or tails, I think the fable is trying to create a future consensus around the truth telling extreme via out of band signalling to children, making this extreme a priori more salient to future generations than a socially signalled non-truth position. In contrast, my takeaway from this piece is that the author is either arguing badly, or the text is meant as a kind of rationality koan, promoting enlightenment via engagement with its flawed argument.

The Straussian reading is definitely not intended on my part—I wouldn't play that kind of mind game with you guys! Or at least, it definitely wasn't consciously intended, but I have to concede that it's probably not a coincidence that the title ended up being "Speaking Truth to Power Is ..." rather than "Preferred Narratives of the Powerful Are ...". Every author should hope to attend her own funeral.

Comment by zack_m_davis on Speaking Truth to Power Is a Schelling Point · 2019-12-31T23:03:10.584Z · score: 4 (2 votes) · LW · GW

OP is trying to convey a philosophical idea (which could be wrong, and whose wrongness would reflect poorly on me, although I think not very poorly, quantitatively speaking) about "true maps as a Schelling point." (You can see a prelude to this in the last paragraph of a comment of mine from two months ago.)

I would have thought you'd prefer that I avoid trying to apply the philosophy idea to a detailed object-level special case (specifically, that of this website) in the comment section of a Frontpaged post (as a opposed to a lower-visibility meta post or private conversation)?? (Maybe this is another illustration of Wei's point that our traditional norms just end up encouraging hidden agendas.)

Comment by zack_m_davis on Stupidity and Dishonesty Explain Each Other Away · 2019-12-30T05:41:59.393Z · score: 2 (1 votes) · LW · GW

Not obvious to me that defensiveness on their part interferes with learning from them? Providing information to the audience would be the main other reason, but the attitude I'm trying to convey more broadly is that I think I'm just ... not a consequentialist about speech? (Speech is thought! Using thinking in order to select actions becomes a lot more complicated if thinking is itself construed as an action! This can't literally be the complete answer, but I don't know how to solve embedded agency!)

Comment by zack_m_davis on Stupidity and Dishonesty Explain Each Other Away · 2019-12-29T22:35:34.767Z · score: 14 (5 votes) · LW · GW

you are establishing an antogonistic [sic] tone to the interaction

Yes, that's right, but I don't care about not establishing an antagonistic tone to the interaction. I care about achieving the map that reflects the territory. To be sure, different maps can reflect different aspects of the same territory, and words can be used in many ways depending on context! So it would certainly be possible to write a slightly different blog post making more-or-less the same point and proposing the same causal graph, but labeling the parent nodes something like "Systematic Error" and "Unsystematic Error", or maybe even "Conflict" and "Mistake". But that is not the blog post that I, personally, felt like writing yesterday.

I don't think it's useful to include only two arrows here. For instance

Right, I agree that more detailed models are possible, that might achieve better predictions at the cost of more complexity.

make the other person defensive and less likely to listen to reason...

I guess that's possible, but why is that my problem?

Comment by zack_m_davis on Free Speech and Triskaidekaphobic Calculators: A Reply to Hubinger on the Relevance of Public Online Discussion to Existential Risk · 2019-12-29T00:47:30.885Z · score: 2 (1 votes) · LW · GW

I was thinking that reputation-hit contributes to neglectedness. Maybe what we really need is a way to reduce reputational "splash damage", so that people with different levels of reputation risk-tolerance can work together or at least talk to each other (using, for example, a website).

Comment by zack_m_davis on Free Speech and Triskaidekaphobic Calculators: A Reply to Hubinger on the Relevance of Public Online Discussion to Existential Risk · 2019-12-29T00:13:18.709Z · score: 9 (3 votes) · LW · GW

Hm, touché. Although ... if "the community" were actually following a policy of strategically arguing for things based on importance-times-neglectedness, I would expect to see a lot more people working on eugenics, which looks really obviously potentially important to me, either on a Christiano-esque Outside View (smarter humans means relatively more human optimization power steering the future rather than unalignable machine-learning algorithms), or a hard-takeoff view (smarter humans sooner means more time to raise alignment-researcher tykebombs). Does that seem right or wrong to you? (Feel free to email or PM me.)

Comment by zack_m_davis on Meta-Honesty: Firming Up Honesty Around Its Edge-Cases · 2019-12-27T05:10:10.970Z · score: 16 (4 votes) · LW · GW

Reply: "Firming Up Not-Lying Around Its Edge-Cases Is Less Broadly Useful Than One Might Initially Think"

Comment by zack_m_davis on Maybe Lying Doesn't Exist · 2019-12-26T05:50:01.140Z · score: 16 (4 votes) · LW · GW

(Scott and I had a good conversation today. I think I need to write a followup post (working title: "Instrumental Categories, Wireheading, and War") explaining in more detail exactly what distinction I'm making when I say I want to consider some kinds of appeals-to-consequences invalid while still allowing, e.g. "Requiring semicolons in your programming language will have the consequence of being less convenient for users who forget them." The paragraphs in "Where to Draw the Boundaries?" starting with "There is an important difference [...]" are gesturing at the distinction, but perhaps not elaborating enough for readers who don't already consider it "obvious.")

Comment by zack_m_davis on Maybe Lying Doesn't Exist · 2019-12-25T06:28:42.350Z · score: 11 (5 votes) · LW · GW

Like, sibling comments are very not-nice, but I argue that they meet the Slate Star commenting policy guidelines on account of being both true and necessary.

Comment by zack_m_davis on Maybe Lying Doesn't Exist · 2019-12-25T06:14:43.198Z · score: 15 (5 votes) · LW · GW

Like, I know my sibling comment is hugely inappropriately socially aggressive of me, and I don't want to hurt your feelings any more than is necessary to incentivize you to process information, but we've been at this for a year! "This definition will make people angry" is not one of the 37 Ways Words Can Be Wrong.

Comment by zack_m_davis on Maybe Lying Doesn't Exist · 2019-12-25T05:41:39.273Z · score: 10 (12 votes) · LW · GW

but has many free variables, so that the structure of reality doesn't constrain it completely. This forces us to make decisions, and since these are not about factual states of the world (eg what the definition of "lie" REALLY is, in God's dictionary) we have nothing to make those decisions on except consequences

Scott, I appreciate the appearance of effort, but I'm afraid I just can't muster the willpower to engage if you're going to motivatedly play dumb like this. (I have a memoir that I need to be writing instead.) You know goddamned well I'm not appealing to God's dictionary. I addressed this shit in "Where to Draw the Boundaries?". I worked really really hard on that post. My prereaders got it. Said got it. 82 karma points says the audience got it. If the elephant in your brain thinks it can get away with stringing me along like this when I have the math and you don't, it should think again.

In the incredibly unlikely event that you're actually this dumb, I'll try to include some more explanations in my forthcoming memoir (working title: "'I Tell Myself to Let the Story End'; Or, A Hill of Validity in Defense of Meaning; Or, The Story About That Time Everyone I Used to Trust Insisted on Playing Dumb About the Philosophy of Language in a Way That Was Optimized for Confusing Me Into Cutting My Dick Off (Independently of the Empirical Facts Determining Whether or Not Cutting My Dick Off Is a Good Idea) and Wouldn't Even Cut It Out Even After I Spent Five Months and Thousands of Words Trying to Explain the Mistake in Exhaustive Detail Including Dozens of Links to Their Own Writing; Or, We Had an Entire Sequence About This, You Lying Motherfuckers").

Comment by zack_m_davis on Free Speech and Triskaidekaphobic Calculators: A Reply to Hubinger on the Relevance of Public Online Discussion to Existential Risk · 2019-12-24T09:41:47.749Z · score: 30 (9 votes) · LW · GW

not talking about taboo political issues on LW

Sure, I'm happy to have separate discussion forums for different topics. For example, I wouldn't want people talking about football on /r/mylittlepony—that would be crazy![1]

"Take it to /r/TheMotte, you guys" is not that onerous of a demand, and it's a demand I'm happy to support: I really like the Less Wrong æsthetic of doing everything at the meta level.[2]

But Hubinger seems to argue that the demand should be, "Take it offline," and that seems extremely onerous to me.

The operative principle here is "Permalink or It Didn't Happen": if it's not online, does it really exist? I mean, okay, there's a boring literal sense in which it "exists", but does it exist in a way that matters?

If they used method (2), it's hard for me to imagine exactly how this would work but probably they would have a lot of problems.

The problem is that between the massive evidential entanglement between facts, the temptation to invent fake epistemology lessons to justify conclusions that you couldn't otherwise get on the merits, and the set of topics that someone has an interest in distorting being sufficiently large, I think we do end up with the analogue of nonsense-math in large areas of psychology, sociology, political science, history, &c. Which is to say, life.

In terms of the calculator metaphor, imagine having to use a triskaidekaphobic calculator multiple times as part of solving a complicated problem with many intermediate results. Triskaidekaphobia doesn't just break your ability to compute 6 + 7. It breaks your ability to compute the infinite family of expressions that include 13 as an intermediate result, like (6 + 7) + 1. It breaks the associativity of addition, because now you can't count on (6 + 7) + 1 being the same as 6 + (7 + 1).[3] And so on.

Also, what Interstice said.

with lots of consultation first to make sure we're not stuck in the Unilateralist's Curse.

So, I agree that regression to the mean exists, which implies that the one who unilaterally does a thing is likely to be the one who most overestimated the value of the thing. I'm very suspicious of this "Unilateralist's Curse" meme being motivatedly and selectively wielded as an excuse for groupthink and conformity, to police and intimidate anyone in your cult[4] who tries to do anything interesting that might damage the "rationalist" or "effective altruism" brand names.

In the context of existential risk, regression to the mean recommends a principle of conformity because you don't want everyone deciding independently that their own AI is probably safe. But if we're talking about censorship about boring ordinary things like political expression or psychology research (not nukes or bioweapons), I suspect considering regression to the mean makes policy recommendations in the other direction.[5] I've been meaning to write a post to be titled "First Offender Models and the Unilateralist's Blessing": if shattering a preference-falsification equilibrium would be good for Society, but requires some brave critical group to eat the upfront cost, then that group is likely to be the one that most underestimated the costs to themselves. Maybe we should let them!

if people burned down calculator factories whenever any of their calculators displayed "13"

If.

On the other hand, if people threatened to burn down factories that produced correct calculators but were obviously bluffing, or if they TPed the factories, then calculator manufacurers who care about correct arithmetic might find it better policy to say, "I don't negotiate with terrorists![6] Do your worst, you superstitious motherfuckers!"

It would be nice if calculator manufacturers with different risk-tolerances or decision theories could manage to cooperate with each other. For a cute story about a structually similar scenario in which that kind of cooperation doesn't emerge, see my latest Less Wrong post, "Funk-tunul's Legacy; Or, The Legend of the Extortion War."[7]


  1. At this point the hypothetical adversary in my head is saying, "Zack, you're being motivatedly dense—you know damned well why that example of separate forums isn't analogous!" I reply, "Yeah, sorry, sometimes the text-generating process in my head is motivatedly dense to make a rhetorical point when I understand the consideration my interlocutor is trying to bring up, but I consider it non-normative, the sort of thing an innocent being wouldn't understand. Call it angelic irony, after the angels in Unsong who can't understand deception. It's not intellectually dishonest if I admit I'm doing it in a footnote." ↩︎

  2. Although as Wei Dai points out, preceded by an earlier complaint by Vanessa Kosoy, this does carry the cost of encouraging hidden agendas. ↩︎

  3. This is the part where a pedant points out that real-world floating-point numbers (which your standard desk calculator uses) aren't associative anyway. I hope there aren't any pedants on this website! ↩︎

  4. In an earlier draft of this comment, this phrase was written in the first person: "our cult." (Yes, this is a noncentral usage of the word cult, but I think the hyperlink to "Every Cause Wants To Be", and this footnote, is adequate to clarify what I mean.) On consideration, the second person seems more appropriate, because by now I think I've actually reached the point of pseudo-ragequitting the so-called "rationalist" community. "Pseudo" because once you've spent your entire adult life in a cult, you can't realistically leave, because your vocabulary has been trained so hard on the cult's foundational texts that you can't really talk to anyone else. Instead, what happens is you actually become more active in intra-cult discourse, except being visibly contemptuous about it (putting the cult's name in scare quotes, using gratuitous cuss words, being inappropriately socially-aggressive to the cult leaders, &c.). ↩︎

  5. But I have pretty intense psychological reasons to want to believe this, so maybe you shouldn't believe me until I actually come up with the math. ↩︎

  6. I tend to use this slogan and appeals to timeless decision theory a lot in the context of defying censorship (example), but I very recently realized that this was kind of stupid and/or intellectually dishonest of me. The application of decision theory to the real world can get very complicated very quickly: if the math doesn't turn out the way I hope, am I actually going to change my behavior? Probably not. Therefore I shouldn't pretend that my behavior is the result of sophisticated decision-theoretic computations on my part, when the real explanation is a raw emotional disposition that might be usefully summarized in English as, "Do your worst, you motherfuckers!" That disposition probably is the result of a sophisticated decision-theoretic computation—it's just that it was a distributed computation that took place over thousands of years in humanity's environment of evolutionary adaptedness. ↩︎

  7. But you should be suspicious of the real-world relevance of my choice of modeling assumptions in accordance with the psychological considerations in the previous two footnotes, especially since I kind of forced it because it's half past one in the morning and I really really wanted to shove this post out the door. ↩︎

Comment by zack_m_davis on Funk-tunul's Legacy; Or, The Legend of the Extortion War · 2019-12-24T09:31:00.302Z · score: 2 (1 votes) · LW · GW

Another Author's Note to "Funk-tunul's Legacy"; Or, A Criminal Confession

Okay, sorry, the sentence beginning with "It's not obviously possible ..." is bullshit handwaving on my part because the modeling assumptions I chose aren't giving me the result I need to make the story come out the way I want. (Unless I made yet another algebra mistake.) But it's almost half past one in the morning, and I'm mostly pretty happy with this post—you see the thing I'm getting at—so I'm pretty eager to shove it out the door and only make it more rigorous later if someone actually cares, because I have a lot of other ideas to write up!

Comment by zack_m_davis on Funk-tunul's Legacy; Or, The Legend of the Extortion War · 2019-12-24T09:30:11.754Z · score: 2 (1 votes) · LW · GW

An Author's Note to "Funk-tunul's Legacy"; Or, A Crime That Is Its Own Confession

This story was inspired by the following three paragraphs Jessica Taylor wrote in an email on 15 April 2019. I didn't actually get permission from Jessica to quote these paragraphs in public, but I falsifiably predict that she won't mind. If she does mind, then she's welcome to sue me for damages in the Court of Benquo. But if the Court of Benquo awards more than $100 in damages (which I falsifiably predict it won't), then I'm going to strongly consider the hypothesis that the Court is corrupt and maybe become a Benquo-anarchist.

CDT ends up in a symbiotic relationship with extort-bot, mutually winning relative to agents that don't give in to extortion, until there are none of those left and extort-bot eats the remaining CDTs.

(The scarier variants of extort-bots are those that cooperate with other copies of themselves, a mixture of clique-bot and extort-bot)

The correct strategy for UDTs is to extort CDT, while being strategic about when to give in to or not give in to extortion from extortbots; giving in early on can be necessary to maintain population growth (getting strictly higher growth than CDT), and at the end it's necessary to stop giving in to extort-bot, to ultimately win the war.

Comment by zack_m_davis on Against Premature Abstraction of Political Issues · 2019-12-24T08:40:33.558Z · score: -4 (2 votes) · LW · GW

Given that that's the world we live in

It's not the world we live in—it's you!

Comment by zack_m_davis on ialdabaoth is banned · 2019-12-24T06:24:55.036Z · score: 2 (1 votes) · LW · GW

I see the argument, but I don't buy it empirically. Understanding social dynamics, power, manipulation, &c. is useful for acquiring the funds to buy the best statues.

Comment by zack_m_davis on Tetraspace Grouping's Shortform · 2019-12-24T00:55:26.394Z · score: 6 (3 votes) · LW · GW

Tulpas??

Comment by zack_m_davis on Free Speech and Triskaidekaphobic Calculators: A Reply to Hubinger on the Relevance of Public Online Discussion to Existential Risk · 2019-12-21T05:09:00.516Z · score: 16 (6 votes) · LW · GW

I'm not sure what your own position is on the "should we discuss object-level politics on LW" question. Are you suggesting that we should just go ahead and do it without giving any weight to taintedness?

No. (Sorry, I guess this isn't clear at all from the post, which was written hastily and arguably should have just been a mere comment on the "Against Premature Abstraction" thread; I wanted to make it a top-level post for psychological reasons that I probably shouldn't elaborate on because (a) you probably don't care, and (b) they reflect poorly on me. Feel free to downvote if I made the wrong call.)

You didn't fully spell out the analogy with the Triskaidekaphobic Calculator, but it seems like you're saying that worrying about taintedness while trying to solve AI safety is like trying to make a calculator while worrying about triskaidekaphobia, so we shouldn't do that?

More like—we should at least be aware that worrying about taintedness is making the task harder and that we should be on the lookout for ways to strategize around that (e.g., encouraging the use of a separate forum and pseudonyms for non-mathy topics, having "mutual defense pact" norms where curious thinkers support each other rather than defaulting to "Well, you should've known better than to say that" victim-blaming, &c.).

Comment by zack_m_davis on Against Premature Abstraction of Political Issues · 2019-12-21T00:49:50.319Z · score: 4 (2 votes) · LW · GW

"Free Speech and Triskaidekaphobic Calculators: A Reply to Hubinger on the Relevance of Public Online Discussion to Existential Risk"

Comment by zack_m_davis on Against Premature Abstraction of Political Issues · 2019-12-18T23:29:09.392Z · score: 23 (8 votes) · LW · GW

anything academic (like AI safety), but not at all for politics [...] avoiding of any hot-button issues

"Politics" isn't a separate magisterium, though; what counts as a "hot-button issue" is a function of the particular socio-psychological forces operative in the culture of a particular place and time. Groups of humans (including such groups as "corportations" or "governments") are real things in the real physical universe and it should be possible to build predictive models of their behavior using the same general laws of cognition that apply to everything else.

To this one might reply, "Oh, sure, I'm not objecting to the study of sociology, social psychology, economics, history, &c., just politics." This sort of works if you define "political" as "of or concerning any topic that seems likely to trigger motivated reasoning and coalition-formation among the given participants." But I don't see how you can make that kind of clean separation in a principled way, and that matters if you care about getting the right answer to questions that have been infused with "political" connotations in the local culture of the particular place and time in which you happen to live.

Put it this way: astronomy is not a "political" topic in Berkeley 2019. In Rome 1632, it was. The individual cognitive algorithms and collective "discourse algorithms" that can't just get the right answer to questions that seem "political" in Berkeley 2019, would have also failed to get the right answer on heliocentrism in Rome 1632—and I really doubt they're adequate to solve AGI alignment in Berkeley 2039.

Comment by zack_m_davis on ialdabaoth is banned · 2019-12-18T04:42:29.621Z · score: 13 (6 votes) · LW · GW

Vaniver mentioned that Michael Vassar was one of the partial inspirations for a supervillain in one of Eliezer Yudkowsky's works of fiction. I'm saying that, firstly, I don't think that's germane in a discussion of moderation policies that aspires to impartiality, even as a playful "Appropriately enough [...]" parenthetical. But secondly, if such things are somehow considered to be relevant, then I want to note that Michael was also the explicit namesake of a morally-good fictional character ("Vhazhar") in another one of Yudkowsky's stories.

The fact that the latter story is also about the importance of judging things on their true merits rather than being misled by shallow pattern-matching (e.g., figuing that a "Lord of Dark" must be evil, or using someone's association with a fictional character to support the idea that they might be worth banning) made it seem worth quoting at length.

Comment by zack_m_davis on ialdabaoth is banned · 2019-12-17T02:42:02.166Z · score: 5 (2 votes) · LW · GW

I don't think this conversation is going to make any progress at this level of abstraction and in public. I might send you an email.

Comment by zack_m_davis on ialdabaoth is banned · 2019-12-16T05:27:49.320Z · score: 2 (1 votes) · LW · GW

Close! I'll PM you.

Comment by zack_m_davis on ialdabaoth is banned · 2019-12-16T04:57:42.412Z · score: 21 (3 votes) · LW · GW

it's a 2-place word [...] It seems like we could easily have very different standards for trustworthiness that cause us to not disagree on the facts while disagreeing on the implications.

Right, I agree that we don't want to get into a pointless pseudo-argument where everyone agrees that x = 60, and yet we have a huge shouting match over whether this should be described using the English word "large" or "small."

Maybe a question that would lead to a more meaningful disagreement would be, "Should our culture become more or less centralized?"—where centralized is the word I'm choosing to refer to a concept I'm going to try to describe extensionally/ostensively in the following two paragraphs.[1]

A low-centralization culture has slogans like, "Nullis in verba" or "Constant vigilance!". If a fringe master sets up shop on the outskirts of town, the default presumption is that (time permitting) you should "consider it open-mindedly and then steal only the good parts [...] [as] an obvious guideline for how to do generic optimization", not because most fringe masters are particularly good (they aren't), but because thinking for yourself actually works and it's not like our leaders in the town center know everything already.

In a high-centralization culture, there's a stronger presumption that our leaders in the town center come closer to knowing everything already, and that the reasoning styles or models being hawked by fringe masters are likely to "contain traps that the people absorbing the model are unable to see": that is, thinking for yourself doesn't work. As a result, our leaders might talk up "the value of having a community-wide immune system" so that they can "act against people who are highly manipulative and deceitful before they have clear victims." If a particular fringe master starts becoming popular, our leaders might want to announce that they are "actively hostile to [the fringe master], and make it clear that [we] do not welcome support from those quarters."

You seem to be arguing that we should become more centralized. I think that would be moving our culture in the absolute wrong direction. As long as we're talking about patterns of adversarial optimization, I have to say that, to me, this kind of move looks optimized for "making it easier to ostracize and silence people who could cause trouble for MIRI and CfAR (e.g., Vassar or Ziz), either by being persistent critics or by embarrassing us in front of powerful third parties who are using guilt-by-association heuristics", rather than improving our collective epistemics.

This seems like a substantial disagreement, rather than a trivial Sorites problem about how to use the word "trustworthy".

do deeply appreciate you championing particular virtues even when I disagree on where the balance of virtues lies

Thanks. I like you, too.


  1. I just made this up, so I'm not at all confident this is the right concept, much like how I didn't think contextualing-vs.-decoupling was the right concept. ↩︎

Comment by zack_m_davis on ialdabaoth is banned · 2019-12-15T18:15:37.838Z · score: 21 (5 votes) · LW · GW

becoming a 'fringe master', where you create something new at the boundary of something pre-existing. You don't have to pay the costs of being part of the 'normal system' and dealing with its oversight, but do gain many of the benefits of its advertising / the draw of the excellence at the center of the pre-existing thing. This is basically the category that I am most worried about / want to be able to act against, where someone will take advantage of new people drawn in by LessWrong / the rationality community who don't know about the missing stairs to watch out for or who is held in low regard.

Thanks for explaining this part; this is really helpful. This model seems to assume that the "oversight" of the "normal system" at the center of the gravity well is trustworthy. I'm currently most worried[1] about the scenario where the "normal system" is corrupt: that new people are getting drawn in by the awesomeness of the Sequences and Harry Potter and the Methods,[2] only to get socialized into a community whose leadership and dominant social trend pays lip service to "rationality", but is not actually interested in using reasoning (at least, not in public) when using reasoning would be socially inconvenient (whether due to the local area's political environment, the asymmetric incentives faced by all sufficiently large organizations, the temptation to wirehead on our own "rationality" and "effectiveness" marketing promises, or many other possible reasons) and therefore require a small amount of bravery.

As Michael Vassar put it in 2013:

The worst thing for an epistemic standard is not the person who ignores or denies it, but the person who tries to mostly follow it when doing so feels right or is convenient while not acknowledging that they aren't following it when it feels weird or inconvenient, as that leads to a community of people with such standards engaging in double-think WRT whether their standards call for weird or inconvenient behavior.

Have you thought at all about how to prevent the center of the gravity well from becoming predatory? Obviously, I'm all in favor of having systems to catch missing-stair rapists. But if you're going to build an "immune system" to delegitimize anyone "held in low regard" without having to do the work of engaging with their arguments—without explaining why an amorphous mob that holds something in "low regard" can be trusted to reach that judgement for reliably good reasons—then you're just running a cult. And if enough people who remember the spirit of the Sequences notice their beloved rationality community getting transformed into a cult, then you might have a rationalist civil war on your hands.

(Um, sorry if that's too ominous or threatening of a phrasing. I think we mostly want the same thing, but have been following different strategies and exposed to different information, and I notice myself facing an incentive to turn up the rhetoric and point menacingly at my BATNA in case that helps with actually being listened to, because recent experiences have trained my brain to anticipate that even high-ranking "rationalists" are more interested in avoiding social threat than listening to arguments. As I'm sure you can also see, this is already a very bad sign of the mess we're in.)


  1. "Worried" is an understatement. It's more like panicking continuously all year with many hours of lost sleep, crying fits, pacing aimlessly instead of doing my dayjob, and eventually doing enough trauma processing to finish writing my forthcoming 20,000-word memoir explaining in detail (as gently and objectively as possible while still telling the truth about my own sentiments and the world I see) why you motherfuckers are being incredibly intellectually dishonest (with respect to a sense of "intellectual dishonesty" that's about behavior relative to knowledge, not conscious verbal "intent"). ↩︎

  2. Notably, written at a time Yudkowsky and "the community" had a lower public profile and therefore faced less external social pressure. This is not a coincidence because nothing is ever a coincidence. ↩︎

Comment by zack_m_davis on ialdabaoth is banned · 2019-12-15T17:59:10.269Z · score: 7 (3 votes) · LW · GW

You know, this is a really lame cheap shot—

(Appropriately enough, his last comment was about suggesting that someone who is losing mental ground to their model of HPMOR!Quirrell talk to him to get the decision procedures from the source.)

If we're going to play this frankly puerile game of bringing up who partially inspired what fictional characters, do I at least get to bring up "The Sword of Good"?

The Lord of Dark stared at Hirou as though he were the crazy one. "The Choice between Good and Bad," said the Lord of Dark in a slow, careful voice, as though explaining something to a child, "is not a matter of saying 'Good!' It is about deciding which is which."

Dolf uttered a single bark of laughter. "You're mad!" his voice boomed. "Can you truly not know that you are evil? You, the Lord of Dark?"

"Names," said the Lord of Dark quietly.

[...]

Hirou staggered, and was distantly aware of the Lord of Dark catching him as he fell, to lay him gently on the ground.

In a whisper, Hirou said "Thank you—" and paused.

"My name is Vhazhar."

"You didn't trust yourself," Hirou whispered. "That's why you had to touch the Sword of Good."

Hirou felt Vhazhar's nod, more than seeing it.

The air was darkening, or rather Hirou's vision was darkening, but there was something terribly important left to say. "The Sword only tests good intentions," Hirou whispered. "It doesn't guide your steps. That which empowers a hero does not make us wise—desperation strengthens your hand, but it strikes with equal force in any direction—"

"I'll be careful," said the Lord of Dark, the one who had mastered and turned back the darkness. "I won't trust myself."

"You are—" Hirou murmured. "Than me, you are—"

I should have known. I should have known from the beginning. I was raised in another world. A world where royal blood is not a license to rule, a world whose wizards do more than sneer from their high towers, a world where life is not so cheap, where justice does not come as a knife in the night, a world where we know that the texture of a race's skin shouldn't matter—

And yet for you, born in this world, to question what others took for granted; for you, without ever touching the Sword, to hear the scream that had to be stopped at all costs—

"I don't trust you either," Hirou whispered, "but I don't expect there's anyone better," and he closed his eyes until the end of the world.

Comment by zack_m_davis on ialdabaoth is banned · 2019-12-14T00:00:30.410Z · score: 24 (10 votes) · LW · GW

the value of having a community-wide immune system, rather than being content with not getting sick myself

I'd be very interested if you could elaborate on what observations make you think "the community" is doing the kind of information-processing that would result in "immune system" norms actually building accurate maps, rather than accelerating our decline into a cult.

It seems to me that what actually helped build common knowledge in the Ialdabaoth case was the victims posting their specific stories online, serving a role analogous to transcripts of witness testimony in court.[1]

In contrast, the conjunction of the "immune system" metaphor and your mention of Anna's comment about Michael makes me imagine social norms that make it easier for high-ranking community members to silence potential rivals or whistleblowers by declaring them to be bad thinkers and therefore not worth listening to.

That is, I perceive a huge difference between, "Witnesses A, B, and C testified that X commited a serious crime and no exculpatory evidence has emerged, therefore I'm joining the coalition for ostracizing X" (analogous to a court) vs. "The mods declared that X uses manipulative epistemic tactics, therefore I'm going to copy that 'antibody' and not listen to anything X says" (analogous to an immune system).

But, maybe I'm completely misunderstanding what you meant by "immune system"? It would be great if you could clarify what you're thinking here.

It would certainly be nice to have a distributed intellectual authority I could trust. I can imagine that such a thing could exist. But painful personal experience has me quite convinced that, under present conditions, there really is just no substitute for thinking for yourself ("not getting sick [one]self").


  1. Thanks to Michael Vassar for teaching me about the historical importance of courts! ↩︎

Comment by zack_m_davis on ialdabaoth is banned · 2019-12-13T20:37:49.395Z · score: 21 (5 votes) · LW · GW

I think this is an example of what that sort of update looks like. Michael isn't banned from LessWrong

Interesting that you should mention this. I've hugely benefited from collaborating with Michael recently. I think the linked comment is terrible, and I've argued with Anna about it several times. I had started drafting a public reply several months ago, but I had set it aside because (a) it's incredibly emotionally painful to write because I simultaneously owe eternal life-debts of eternal loyalty to both Michael and Anna,[1] and (b) it isn't even the most important incredibly-emotionally-painful high-community-drama-content piece of writing I have to do. The fact that you seem to take it this seriously suggests that I should prioritize finishing and posting my reply, though I must ask for your patience due to (b).


  1. Like a robot in an Isaac Asimov story forced to choose between injuring a human being or, through inaction, allowing a human being to come to harm, I briefly worried that my behavior isn't even well-defined in the event of a Michael–Anna conflict. (For the same reason, I assume it's impossible to take more than one Unbreakable Vow in the world of Harry Potter and the Methods.) Then I remembered that disagreeing with someone's blog comment isn't an expression of disloyalty. If I were to write a terrible blog comment (and I've written many), then I should be grateful if Anna were to take the time to explain what she thinks I got wrong. ↩︎

Comment by zack_m_davis on ialdabaoth is banned · 2019-12-13T20:30:52.946Z · score: 14 (5 votes) · LW · GW

If you learned that someone in the rationality community had taken on ialdabaoth as a master (like in the context of zen, or a PhD advisor, or so on), would you expect them to grow in good directions or bad directions?

Bad directions. The problem is that I also think this of other users who we aren't banning, which suggests that our standard for "allowed to post on Less Wrong" is lower than "would be a good master."

[Ideally, from the epistemic state you were in ~2 years ago, rather than the epistemic state you're in now.]

Okay, right, I'm much less sure that I would have said "bad directions" as confidently 2 years ago.

am trying to help instantiate the version of LessWrong that is primarily focused on epistemic progress.

Thanks for this!! (I think I'm much more worried than you about the failure mode where something claiming to make intellectual progress is actually doing something else, which makes me more willing to tolerate pragamatic concessions of principle that are explicitly marked as such, when I'm worried that the alternative is the concession being made anyway with a fake rationale attached.)

Comment by zack_m_davis on ialdabaoth is banned · 2019-12-13T19:07:24.040Z · score: 2 (1 votes) · LW · GW

I agree that the attention sinkhole is a problem.

Comment by zack_m_davis on ialdabaoth is banned · 2019-12-13T19:00:54.709Z · score: 12 (7 votes) · LW · GW

I currently know of no person who would describe ialdabaoth as a good influence

"Good influence" in what context? I remember finding some of his Facebook posts/comments insightful, and I'm glad I read them. It shouldn't be surprising that someone could have some real insights (and thereby constitute a "good influence" in the capacity of social-studies blogging), while also doing a lot of bad things (and thereby constituting a very bad influence in the capacity of being a real-life community member), even if the content of the insights is obviously related to the bad things. (Doing bad things and getting away with them for a long time requires skills that might also lend themselves to good social-studies blogging.)

Less Wrong is in the awkward position of being a public website (anyone can submit blog posts about rationality under a made-up name), and also being closely associated with a real-life community with dense social ties, group houses, money, &c. If our actual moderation algorithm is, "Ban people who have have been justly ostracized from the real-life community as part of their punishment, even if their blog comments were otherwise OK", that's fine, but we shouldn't delude ourselves about what the algorithm is.

Comment by zack_m_davis on ialdabaoth is banned · 2019-12-13T18:02:31.965Z · score: 20 (7 votes) · LW · GW

it's not like there's anything I can do about it anyway.

Not having control of a website isn't the same thing as "nothing I can do." If an authority who you used to trust goes subtly crazy in a way that you can detect, but you see other people still trusting the authority, you could do those other people a favor by telling them, "Hey, I think the authority has gone crazy, which conclusion I came to on the basis of this-and-such evidence. Maybe you should stop trusting the authority!"

The good/sane/reasonable moderator subjects their decisions to scrutiny, and thus stands to be perpetually criticized.

Right, and then they update on the good criticism (that contains useful information about how to be a better moderator) and ignore the bad criticism (that does not contain useful information). That's how communication works. Would you prefer to not be perpetually criticized?!

Fundamentally you stand to gain little and lose much by making posts like this

In a system of asymmetric justice where people are mostly competing to avoid being personally blamed for anything, sure. Maybe a website devoted to discovering and mastering the art of systematically correct reasoning should aspire to a higher standard than not getting personally blamed for anything?!

Comment by zack_m_davis on ialdabaoth is banned · 2019-12-13T08:05:44.406Z · score: 53 (18 votes) · LW · GW

We think that ialdabaoth poses a substantial risk to our epistemic environment due to manipulative epistemic tactics, based on our knowledge and experience of him. This is sufficient reason for the ban

I'm having trouble convincing myself that this is the real reason. Imagine an alternate universe where their local analogue of Ialdabaoth was just as manipulative, wrote almost all the same things about status, power, social reality, &c., but was definitely not guilty of any sex crimes for reasons that had nothing to do with his moral character. (Perhaps imagine him having some kind of exotic sexual orientation that would be satisfied without human contact, like a statue fetish.) Would we really ban such a person on the grounds of manipulative epistemic tactics?

Your "fraudster designs a financial instrument" scenario explains why one should definitely be suspicious of the value of such a user's contributions—but frankly, I'm suspicious of a lot of people in that way: how do you decide who to ban?

It occurs to me that the "reputation vs. merit" framing completely fails to address the reasons many would assign such bad reputation. (The function of bad reputation is to track things that are actually bad!) Maybe we just have a "moral taste" for not working with people who (are believed to) have committed sufficiently bad crimes, and we're willing to pay the cost of forgoing positive contributions from them?

If that's actually the psychology driving the decision, it would be better to fess up to it rather making up a fake reason. Better for the physics journal editor to honestly say, "Look, I just don't want to accept a paper from a murderer, okay?", rather than claiming to be impartial and then motivatedly subjecting the paper to whatever isolated demands for rigor were necessary to achieve the appearance of having rejected it on the merits.