Posts

Scoring 2020 U.S. Presidential Election Predictions 2020-11-08T02:28:29.234Z
Message Length 2020-10-20T05:52:56.277Z
Msg Len 2020-10-12T03:35:05.353Z
Artificial Intelligence: A Modern Approach (4th edition) on the Alignment Problem 2020-09-17T02:23:58.869Z
Maybe Lying Can't Exist?! 2020-08-23T00:36:43.740Z
Algorithmic Intent: A Hansonian Generalized Anti-Zombie Principle 2020-07-14T06:03:17.761Z
Optimized Propaganda with Bayesian Networks: Comment on "Articulating Lay Theories Through Graphical Models" 2020-06-29T02:45:08.145Z
Philosophy in the Darkest Timeline: Basics of the Evolution of Meaning 2020-06-07T07:52:09.143Z
Comment on "Endogenous Epistemic Factionalization" 2020-05-20T18:04:53.857Z
"Starwink" by Alicorn 2020-05-18T08:17:53.193Z
Zoom Technologies, Inc. vs. the Efficient Markets Hypothesis 2020-05-11T06:00:24.836Z
A Book Review 2020-04-28T17:43:07.729Z
Brief Response to Suspended Reason on Parallels Between Skyrms on Signaling and Yudkowsky on Language and Evidence 2020-04-16T03:44:06.940Z
Why Telling People They Don't Need Masks Backfired 2020-03-18T04:34:09.644Z
The Heckler's Veto Is Also Subject to the Unilateralist's Curse 2020-03-09T08:11:58.886Z
Relationship Outcomes Are Not Particularly Sensitive to Small Variations in Verbal Ability 2020-02-09T00:34:39.680Z
Book Review—The Origins of Unfairness: Social Categories and Cultural Evolution 2020-01-21T06:28:33.854Z
Less Wrong Poetry Corner: Walter Raleigh's "The Lie" 2020-01-04T22:22:56.820Z
Don't Double-Crux With Suicide Rock 2020-01-01T19:02:55.707Z
Speaking Truth to Power Is a Schelling Point 2019-12-30T06:12:38.637Z
Stupidity and Dishonesty Explain Each Other Away 2019-12-28T19:21:52.198Z
Firming Up Not-Lying Around Its Edge-Cases Is Less Broadly Useful Than One Might Initially Think 2019-12-27T05:09:22.546Z
Funk-tunul's Legacy; Or, The Legend of the Extortion War 2019-12-24T09:29:51.536Z
Free Speech and Triskaidekaphobic Calculators: A Reply to Hubinger on the Relevance of Public Online Discussion to Existential Risk 2019-12-21T00:49:02.862Z
A Theory of Pervasive Error 2019-11-26T07:27:12.328Z
Relevance Norms; Or, Gricean Implicature Queers the Decoupling/Contextualizing Binary 2019-11-22T06:18:59.497Z
Algorithms of Deception! 2019-10-19T18:04:17.975Z
Maybe Lying Doesn't Exist 2019-10-14T07:04:10.032Z
Heads I Win, Tails?—Never Heard of Her; Or, Selective Reporting and the Tragedy of the Green Rationalists 2019-09-24T04:12:07.560Z
Schelling Categories, and Simple Membership Tests 2019-08-26T02:43:53.347Z
Diagnosis: Russell Aphasia 2019-08-06T04:43:30.359Z
Being Wrong Doesn't Mean You're Stupid and Bad (Probably) 2019-06-29T23:58:09.105Z
What does the word "collaborative" mean in the phrase "collaborative truthseeking"? 2019-06-26T05:26:42.295Z
The Univariate Fallacy 2019-06-15T21:43:14.315Z
No, it's not The Incentives—it's you 2019-06-11T07:09:16.405Z
"But It Doesn't Matter" 2019-06-01T02:06:30.624Z
Minimax Search and the Structure of Cognition! 2019-05-20T05:25:35.699Z
Where to Draw the Boundaries? 2019-04-13T21:34:30.129Z
Blegg Mode 2019-03-11T15:04:20.136Z
Change 2017-05-06T21:17:45.731Z
An Intuition on the Bayes-Structural Justification for Free Speech Norms 2017-03-09T03:15:30.674Z
Dreaming of Political Bayescraft 2017-03-06T20:41:16.658Z
Rationality Quotes January 2010 2010-01-07T09:36:05.162Z
News: Improbable Coincidence Slows LHC Repairs 2009-11-06T07:24:31.000Z

Comments

Comment by zack_m_davis on Straight-edge Warning Against Physical Intimacy · 2020-11-24T19:42:54.351Z · LW · GW

some people have a vagina (so someone you might call a female) and produce predominantly testosterone [...] because they're a cis woman and have PCOS, or because they're intersex, or for other reasons

Really? An article in Clinical Medicine and Research claims that most testosterone values in females with PCOS are ≤150 ng/dL, whereas (for comparison) the American Urological Association says that low testosterone in males should be diagnosed below 300 ng/dL.

What specific intersex condition or specific other reason would result in people with a vagina producing predominantly testosterone? (Also, I'm not really sure what "predominantly" means in this context—as compared to what?)

Comment by zack_m_davis on DanielFilan's Shortform Feed · 2020-11-23T08:19:07.402Z · LW · GW

(Or M-x insert-char GREEK SM<tab> L<tab>M<tab> in Emacs.)

Comment by zack_m_davis on Open & Welcome Thread – November 2020 · 2020-11-23T06:39:18.207Z · LW · GW

I was confused why I got a "Eigil Rischel has created a new post: Demystifying the Second Law of Thermodynamics" notification. Turns out there's a "Subscribe to posts" button on people's user pages!

I don't remember clicking that option on Eigil's page (nor having any particular memory of who Eigil Rischel is), but I presumably must have clicked it by accident sometime in the past, probably last year (because the document icon tab of my notifications pane also has an old notification for his previous post, and it looks like I had already upvoted his two next-previous posts from October and September 2019).

Comment by zack_m_davis on DanielFilan's Shortform Feed · 2020-11-23T06:31:02.602Z · LW · GW

It's μCOVID, with a μ!

Comment by zack_m_davis on The Darwin Game - Rounds 21-500 · 2020-11-21T23:35:11.984Z · LW · GW

(Aside: I continue to be quietly amused at how much AbstractSpyTreeBot has ended up affecting the game (by partially inspiring EarlyBirdMimicBot and getting exploited by MeasureBot and accidentally spawning a new timeline), given how little I've invested in the game compared to others—the backstory is not one of strategic thinking about how to compete, but rather, I was feeling bleh in the afternoon of 9 October, and writing a simulator-bot using the ast module just seemed like a fun exercise that should only take a couple hours.)

Comment by zack_m_davis on AGI Predictions · 2020-11-21T19:25:14.773Z · LW · GW

At time of writing, I'm assigning the highest probability to "Will AGI cause an existential catastrophe?" at 85%, with the next-highest predictions at 80% and 76%. Why ... why is everyone so optimistic?? Did we learn something new about the problem actually being easier, or our civilization more competent, than previously believed?

Should—should I be trying to do more x-risk-reduction-relevant stuff (somehow), or are you guys saying you've basically got it covered? (In 2013, I told myself it was OK for dumb little ol' me to personally not worry about the Singularity and focus on temporal concerns in order to not have nightmares, and it turned out that I have a lot of temporal concerns which could be indirectly very relevant to the main plot, but that's not my real reason for focusing on them.)

Comment by zack_m_davis on The Darwin Game - Rounds 3 to 9 · 2020-11-14T02:15:45.918Z · LW · GW

I wish your graphing software had a better label-layout algorithm. (I'm not seeing AbstractSpyTreeBot on the charts, even after downloading the images to look closer? Was it disqualified, or am I just blind?)

Comment by zack_m_davis on Open & Welcome Thread – November 2020 · 2020-11-09T17:52:27.686Z · LW · GW

(I thought the playful intent would be inferred from lying-accusations being incongruous with the genre of math help. Curious what article this was?)

Comment by zack_m_davis on Open & Welcome Thread – November 2020 · 2020-11-09T06:59:19.014Z · LW · GW

The problem is that you're lying!

You claim that P(B|A) = 0.21, P(A) = 0.07, and P(B) = 0.01. But that can't actually be true! Because P(B) = P(B|A)P(A) + P(B|¬A)P(¬A), and if what you say is true, then P(B|A)P(A) = (0.21)(0.07) = 0.0147, which is bigger than 0.01. So because P(B|¬A)P(¬A) can't be negative, P(B) also has to be bigger than 0.01. But you said it was 0.01! Stop lying!

Comment by zack_m_davis on Location Discussion Takeaways · 2020-11-04T19:22:28.689Z · LW · GW

The relevance is that your decisions as part of a group's coordination process should depend on what you think the group is actually doing in practice.

(If that sentence didn't make sense, then please forget it and write off this thread as a waste; I started typing a parable about the Northern Conservative Baptist Great Lakes Region Council of 1879, but on reflection, it's probably better if I withhold further commentary on this topic until I finish a future top-level non-Frontpageable post, which has been delayed a while because I also have to finish 20,000+ words of background material that I want to publish first or concurrently. Sorry if this has been obnoxious.)

Comment by zack_m_davis on Location Discussion Takeaways · 2020-11-04T01:01:56.621Z · LW · GW

In response to

Social safety is a much tougher nut to crack. [...] The prominence of cancel culture and its effects on people such as Robin Hanson and Steve Hsu have put some rationalists on edge, especially those who write (or act) publicly in a way that's traceable to their real identity. [...] I guess cities are maybe worse on the cancel culture dimension because if you're hidden in the middle of nowhere it's harder for people to credibly threaten to physically attack you.

Comment by zack_m_davis on Location Discussion Takeaways · 2020-11-03T22:35:02.282Z · LW · GW

You misunderstand the problem with cancel culture. It's not about personal safety. The political significance of a man with a gun is not that he might shoot you, but that the credible threat to shoot lets him boss everyone else around. (Because though he doesn't have enough bullets to shoot everybody, no one wants to step out of line first.)

If the occasional dissident is willing the bear the costs of speaking, but can't win the argument in public despite being correct (because winning the argument in public would require one's interlocutors to concede, which they can never do if they can't bear the costs of speaking), then your rationalist community fails to achieve the map that reflects the territory. The problem isn't "safety" for individuals; the problem is that your community is failing to do the thing it markets itself as doing. At that point, you should probably stop calling the thing a "rationalist community" in order to avoid confusing innocents who came to you looking for shared maps? (But you still might want to have a cool website where people talk about Bayesian statistics!)

Comment by zack_m_davis on The Darwin Game - Round 1 · 2020-10-27T00:35:34.588Z · LW · GW

I wrote an 8-part series with 11,277 words and 24 graphs.

I think a lot of us are still curious what happened! Maybe concatenate the whole thing into a non-Frontpageable post with an explanatory disclaimer at the top?

Comment by zack_m_davis on The Darwin Game - Rounds 0 to 10 · 2020-10-25T16:14:27.277Z · LW · GW

I see; I was naïvely thinking in terms of "only losing by 50 points doesn't sound so bad, right?!", not carefully thinking about how the update rule works. Now that you point it out, I agree that (200/(200+9*250))/0.1 ≈ 0.82.

Comment by zack_m_davis on The Darwin Game - Rounds 0 to 10 · 2020-10-24T06:40:45.117Z · LW · GW

I guess some got really lucky/unlucky with other bots' random rolls?

No, 10 rounds of 100 turns is a decently large sample size—I think some are actually doing badly against outsiders.

Comment by zack_m_davis on The Darwin Game - Rounds 0 to 10 · 2020-10-24T06:27:03.597Z · LW · GW

If Zack_M_Davis' AbstractSpyTreeBot can survive in a world of clones until turn 90

I'm feeling optimistic about this! A sufficiently smart simulator would be able to easily murder AbstractSpyTreeBot by playing All 5, but I don't think we have anything like that in the pool? Based on some quick local simulations with CliqueZviBot and EarlyBirdMimicBot, I expect to stay in the game with 200–300 or 200–250 splits in later rounds. (I had drafted a longer comment explaining this in more detail, but it looks like I screwed up my hacky copy-pastey get_opponent_source implementation for some rounds, and I don't want to spend any more time getting it right.)

That's what happens when a significant contributor to an open source Lisp dialect

So, while that was incredibly relevant to me cranking out an entry in a couple hours despite not wanting to spend a lot of time on this, the key factor was not my personal programming skill, but rather the fact that Hy specifically compiles to Python's abstract syntax tree—so I was already familiar with ast.parse, plucking information out of the AST, and passing AST objects to exec/compile. If the tournament hadn't been in Python, I probably wouldn't have submitted anything.

Comment by zack_m_davis on What is the current bottleneck on genetic engineering of human embryos for improved IQ · 2020-10-23T04:41:49.063Z · LW · GW

The question title says "bottleneck", but the body says "technological constraint." But I wonder—is the bottleneck a technological constraint, or is it a political constraint? (That is, maybe the expertise exists, but literally no one has the conjunction of expertise and bravery to actually do it.) The FAQ page of Genomic Prediction says:

Does Genomic Prediction Clinical Laboratory screen embryos for increased intelligence i.e. high IQ?

No. We only screen against negative (disease) risks.

But if you're already providing polygenic scores for the purpose of choosing an IVF embryo for implantation, there would seem to be no technical obstacle to using a polygenic score for intelligence. Right?

Comment by zack_m_davis on Message Length · 2020-10-23T03:31:55.920Z · LW · GW

Realistically?—Python and Rust are the only two languages I'm Really Good at, and I'm more Excited about Rust because I write enough Python at my dayjob? In my defense, the .windows iterator on slices was really handy here and is shamefully missing from Python's itertools.

Comment by zack_m_davis on The Darwin Game · 2020-10-23T01:53:00.519Z · LW · GW

Seeing people actually use Hy is making me nostalgic!

Comment by zack_m_davis on Message Length · 2020-10-22T02:57:01.372Z · LW · GW

You're right! This is something that the literature has details on, that I chose to simplify/gloss-over in pursuit of my goal of "write Actual Executable Code implementing minimum-description-length model selection and then tell a cute story about it."

Chapter 5 of Peter D. Grünwald's The Minimum Description Length Principle gives the complexity term (assuming that the parameter space is compact) as , where is the number of parameters, is the bits of percision per parameter, and is the codelength function for a universal prior on the integers (the theoretical ideal that Levenshtein coding and the Elias ω coding approach). Basically, in order to specify your parameters, you need to first say how many of them there are and to what precision, and for that, you need a prefix-free code for the integers, which is itself kind of intricate: you can't give every integer the same codelength, so you end up doing something like "first give the order of magnitude (in, um, unary), and then pick out a specific integer in that range", but recursively (so that you first give the order of magnitude of the order of magnitude if it gets too big, &c.).

(Oh, and Grünwald warns that this still isn't optimal and that question of the best encoding is left to §15.3, but I'm not there yet—it's a big textbook, okay?)

I was originally imagining implementing the integer coding, but I decided it was too complicated for the cute story I wanted to tell about model selection, particularly since I wanted to provide Actual Executable Code with minimal boilerplate and minimal dependencies. (Rust just gives me f32 and f64, not variable-length floats.) I think this kind of handwaving should be admissible in the cute-blog-story genre as long as the author addresses objections in the comment section.

Comment by zack_m_davis on Message Length · 2020-10-22T01:54:53.080Z · LW · GW

Thanks! But ... I'm not seeing the Curated "★" icon (e.g., in the Posts list on my profile)? Am I missing something, or is there an extra button you still have to click?

Comment by zack_m_davis on Mark Xu's Shortform · 2020-10-21T19:41:33.319Z · LW · GW

What are you using the word "rationalist" to mean? If you just mean "members of any subculture bearing some line of memetic descent from early Less Wrong" (which I don't think deserves the pretentious term rationalist, but putting that aside), why is "communication platform" a useful way to chop up that social grouping? A lot of the same people use Less Wrong and Facebook and Twitter and Discord and Google Docs, and a lot of the people who use the same platform wouldn't be in the same cluster if you were to analyze the graph of what actual conversations people are using these platforms to have.

Comment by zack_m_davis on Message Length · 2020-10-21T14:48:38.741Z · LW · GW

Your childhood dislike of asymmetry led you to invent the Thue–Morse sequence?!

Comment by zack_m_davis on Problems Involving Abstraction? · 2020-10-21T05:25:31.446Z · LW · GW

How do socially-constructed concepts work?!

Negative example: trees. Trees exist, and trees are not socially constructed. An alien AI observing Earth from behind a Cartesian veil would be able to compress its observations by formulating a concept that pretty closely matches what we would call tree, because the atom-configurations we call "trees" robustly have a lot of things in common: once the AI has identified something as a "tree" by observing its trunk and leaves, the AI can make a lot of correct predictions about the "tree" having roots, this-and-such cellular walls, &c. without observing them directly, but rather by inference from knowledge about "trees" in general.

Positive example: Christmas. Christmas exists. An alien AI observing Earth from behind a Cartesian veil would be able to make better predictions about human behavior in many places around Epoch time 62467200 ± 31557600·n (for integer n ) by formulating the concept of "Christmas". However, Christmas is socially constructed: if humans didn't haven't a concept of "Christmas", there would be no Christmas (the AI-trick for improving predictions using the idea of "Christmas" would stop working), but if humans didn't have a concept of trees, there would still be trees (the AI-trick for improving predictions using the idea of "trees" would still work).

Semi-positive example: adulthood. Adulthood exists. There's a Sorites situation on exactly how old a human has to be to be an "adult", and different human cultures make different choices about where to draw that line. But this isn't just a boring Sorites non-problem, where different agents might use different communication signals without disagreeing about about the underlying reality (like when I say it's "hot" and you say it's "not hot, just warm" and our friend Hannelore says it's "heiß", but we all agree that it's exactly 303.6 K): an alien AI observing Earth from behind a Cartesian veil can make better predictions about whether I'll be allowed to sign contracts by reasoning about whether my Society considers me an "adult", not by directy using the simple measurement test that Society usually uses to make that determination, with exceptions like minor emancipation.

My work-in-progress take: an agent outside Society observing from behind a Cartesian veil, who only needs to predict, but never to intervene, can treat socially-constructed concepts the same as any other: "Christmas" is just a pattern of behavior in some humans, just like "trees" are a pattern of organic matter. What makes social construction special is that it's a case where a "map" is exerting control over the "territory": whether I'm considered an "adult" isn't just putting a semi-arbitrary line on the spectrum of how humans differ by age (although it's also that); which Schelling point the line settles on is used as an input into decisions—therefore, predictions that depend on those decisions also need to consider the line, a self-fulfilling prophecy. Alarmingly, this can give agents an incentive to fight over shared maps!

Comment by zack_m_davis on Rafael Harth's Shortform · 2020-10-20T20:04:53.347Z · LW · GW

I found it super surprising that the gradient determines the value of every directional derivative. Like, really?

When reading this comment, I was surprised for a moment, too, but now that you mention it—it's because if the function is smooth at the point where you're taking the directional derivative, then it has to locally resemble a plane, just like a how a differentiable function of a single variable is said to be "locally linear". If the directional derivative varied in any other way, then the surface would have to have a "crinkle" at that point and it wouldn't be differentiable. Right?

Comment by zack_m_davis on Message Length · 2020-10-20T15:46:17.749Z · LW · GW

Yeah, but I couldn't get that program to run on my desktop.

Comment by zack_m_davis on Police violence: The veil of darkness · 2020-10-16T16:48:19.769Z · LW · GW

So where’s that discussion actually happening?

Try /r/TheMotte or /r/CultureWarRoundup?

Comment by zack_m_davis on Journal article: "The Mythical Taboo on Race and Intelligence" · 2020-10-16T06:23:54.371Z · LW · GW

its existence makes it seem worth linking to.

Worth linking to here—on this website, specifically? Are you sure?? Personally, I'm happy to take the bait as part of my desperate and vehement Free Speech for Shared Maps campaign, but I'm guessing a lot of other users would rather we silently Just Not? That is, having Less Wrong posts about censorship as a general phenomenon (perhaps illustrated with fictional or non-contemporary examples, as tradition requires) is clearly within-charter of advancing the art of human rationality, but having Less Wrong posts about alleged censorship of specific topics might be off-charter if it were the case that those topics are censored and conversations about whether or not they're censored are too likely to run afoul of the censorship, or if they're not censored but talking about whether they were would leak too much info about censored topics in nearby counterfactual worlds.

Okay, probably you can guess from the previous paragraph that I disagree with Jackson and Winston about the taboo being mythical (I tried to write the last sentence of the previous graf in a way that paid lip service towards Glomarization, but I doubt anyone bought it), but more importantly and fundamentally than that, I think the "taboo: mythical or real?" framing suffers from a pretty bad Sorites problem which I think we could do a pretty good job of dissolving (albeit with more effort than I have to spare right now) by going into more detail about specifically which coalitions are trying to suppress what information about which topics, by what means, in which venues, and how well they're succeeding. For example, Jackson and Winston are correct to point out that Eysenck/Jensen/Rushton/Lynn/Gottfredson kept their tenured jobs, but that's only setting an upper bound on the strength of incentives against hereditarianism (the tenure system still works!), not showing that the incentives are mythical.

I don't think this paper is being very fair to the scientific claims of hereditarian researchers. For example, Jackson and Winston write,

a high heritability index [...] had no bearing on the source of any Black–White differences in intelligence test scores, as the sources of within-group variation could not be used to explain between-group variation (Lewontin, 1970).

Now, it's true that within-group heritability doesn't directly imply between-group heritability—Lewontin's seed analogy is germane and competent hereditarians absolutely know this—but I think "no bearing" is false in a Bayesian sense: there's a relationship between within-group and between-group heritability such that the former is relevant (if not in itself dispositive) evidence about the latter. Jensen covers these issues in a lot of detail in Ch. 12 of his The g factor: basically, if two groups differ by one standard deviation in some phenotypic trait, and the within-group heritability is 0.7 but you think the between-group heritability is zero, that implies that the groups' environments differ by 1.83 standard deviations (in whatever aspects are relevant to the development of the trait). But that's a specific in-principle-falsifiable empirical prediction! If you want to argue that the environments do differ that much (perhaps due to factors like structural racism that we don't know how to adequately measure), fine!—but that's not the same thing as "no bearing".

Hereditarian researchers still call for establishing a two-tiered educational system for White and Black people (Cofnas, 2020, p. 134).

Notably, Cofnas denies this.

Comment by zack_m_davis on The Darwin Game · 2020-10-10T03:26:32.720Z · LW · GW

I don't know whether it's a competitive entry (it would be a FoldBot in Zvi's taxonomy), but as a quick piece of software, I'm pretty proud of AbstractSpyTreeBot!

Comment by zack_m_davis on The Treacherous Path to Rationality · 2020-10-10T03:00:41.566Z · LW · GW

The Rationality community [...] has been the main focus of Rationality [...] rationality's most famous infohazard [...] join the tribe [bolding mine]

I agree that explicit reasoning is powerful and that the lesswrong.com website has hosted a lot of useful information about COVID-19, but this self-congratulatory reification of "the community"—identifying "rationality" (!!) with this particular cluster of people who read each other's blogs—is super toxic. (Talk about "bucket errors"!) Our little robot cult does not have a monopoly on reason itself!

the young rationalist must navigate strange status hierarchies and bewildering memeplexes. I've seen many people bounce off the Rationalist community over those two things.

Great! If bright young people read and understand the Sequences and go on to apply the core ideas (Bayesian reasoning, belief as anticipated experience, the real reasons being the ones that compute your decision, &c.) somewhere else, far away from the idiosyncratic status hierarchy of our idiosyncratic robot cult, that's a good thing. Because it is about the ideas, not just roping in more warm bodies to join the tribe, right?!

Comment by zack_m_davis on The Darwin Game · 2020-10-10T00:54:37.935Z · LW · GW

This is the Nash bargaining game. Is there some specific reason to call it a "Prisoner's Dilemma variation", or is it just "everyone has heard of the Prisoner's Dilemma, and don't have a generic term for this kind of game-theory tournament"?

Comment by zack_m_davis on Covid 10/8: October Surprise · 2020-10-09T16:09:42.112Z · LW · GW

The Good Place is overrated; prefer to skip word entirely unless you really really need it. (Which is sometimes.)

Comment by zack_m_davis on Open Communication in the Days of Malicious Online Actors · 2020-10-09T05:49:20.858Z · LW · GW

Thanks (strong-upvoted), this is a really important objection! If I were to rewrite the grandparent more carefully, I would leave off the second invocation of the "I don't negotiate ..." slogan at the end. I think I do want to go as far as counting evolutionary (including cultural-evolutionary) forces under your "modified to become non-strategic by an agent with the aim [...]" clause—but, sure, okay, if I yell in a canyon and the noise causes a landslide, we don't want to say I was right to yell because keeping silent would be giving to to rock terrorism.

Importantly, however, in the case of the harrassed food critic, I stand by the "not my fault" response, whereas the landslide would be "my fault". This idea of "fault" doesn't apply to the God–Empress or other perfectly spherical generic consequentialists on a frictionless plane in a vacuum; it's a weird thing that we can only make sense of in scenarios where multiple agents are occupying something like the same "moral reference frame". (Real-world events have multiple causes; consequentialist agents do counterfactual queries on their models of the world in order to decide what action to output, but "who is 'to blame' for this event I assign negative utility" is never a question they need to answer.)

But I think blame-allocation is a really important feaure of what's actually going on when crazy monkeys like us have these discussions that purport to be about decision theory, but are really about monkey stuff. (It's not that I started out trying to minimize existential risk and happened to compute that going on a Free Speech for Shared Maps crusade was the optimal action; as you know, what actually happened was ... well, perhaps more on this in a forthcoming post, "Motivation and Political Context of my Philosophy of Language Agenda".) I have to admit it's plausible that a superintelligent singleton God-Empress programmed with the ideal humane utility function would advise me to self-censor for the greater good. And coming from Her, I wouldn't hesitate to take that advice (because She would know). But that's not the situation I'm actually in! In the "Provoking Malicious Supporters" section of the post, Gooen writes, "This closely mirrors legal discussions of negligence, gross neglect, and malice", but negligence and neglect are blame-allocation concepts, not single-agent decision theory concepts!

In accordance with the theory of universal algorithmic bad faith, we might speculate that some part of my monkey-brain is modeling "posts that imply speakers should be blamed for negative side-effects of their speech" as enemy propaganda from the Blight dressed up in the literary genre of consequentialism, for which my monkey-brain has cached counter-propaganda. The only reason this picture doesn't spell complete doom for the project of advancing the art of human rationality, is that the genre constraints are actually pretty hard to satisfy and have been set up in a way that extracts real philosophical work out of monkey-brains that have Something to Protect, much as a well-functioning court is set up in a way that extracts Justice, even if (say) the defendant is only trying to save her own neck.

Comment by zack_m_davis on Open Communication in the Days of Malicious Online Actors · 2020-10-08T04:09:02.016Z · LW · GW

If you are a consequentialist, you probably should avoid discussing porridge breakfasts. [...] Things could get worse if this person had an agenda. They realize they have power over you.

I don't negotiate with terrorists! Whether or not this person consciously thinks of themselves as having an agenda, if their behavior is conditioned on mine in a way that's optimized for controlling my behavior—the "I just wanted to let you know" message definitely counts—then I must regard them as an extortionist, who is only threatening harm because they expect the threat to succeed. It would be awfully short-sighted of me to let them get away with that—for the end of that game is oppression and shame, and the thinker that pays it is lost!

The function of speech is to convey information—to build shared maps that reflect the territory. The reason speech is such a powerful cognitive technology is because accurate beliefs are a convergent instrumental value—whatever you're trying to do, you'll probably do a better job if you can make accurate predictions. When I contribute accurate information to the commons, I don't know all the various downstream consequences of other agents incorporating that information into their maps—I don't see how I'm supposed to compute that. Even if "You should always choose the action with the best consequences" would be the correct axiology for some superintelligent singleton God–Empress who oversees the whole universe and all the consequences in it, I'm not a God–Empress, and "Just tell the goddamned truth" (with the expectation that this is good on net, because true maps are generically useful to other agents, almost none of whom are evil) seems like a much more tractable goal for me to aim at.

Things arguably get more complicated when the aggressor thinks of themself as being on your side.

What does that even mean? I read lots of authors, including a lot of people who I would personally dislike, because I benefit from reading the information that they have to convey. But they don't own me, and I don't own them. Obviously. What's this "side" thing about? Am I to be construed as being on the "side" of the so-called "rationalist" or "effective altruism" "communities" just because Eliezer Yudkowsky rewrote my personality over the internet twelve years ago? God, I hope not!

Option 4 [Don't Censor] [...] seem fairly common though deeply unfortunate. It’s generally not very pleasant to be in a world where those who are listened to routinely select options 4 [...]

It's not very pleasant to live in a world with terrorists trying to control what people think! And any sane blame-allocation algorithm puts the blame on the terrorists, not the people who are trying to think!

This person writes a tweet about food issues, and then a little while later some food critic gets a threat. We can consider this act a sort of provocation of malicious supporters, even if it were unintentional. [...] But we can speculate on what they might have been thinking when they did this.

I agree with Dagon that loudly condemning the malicious actors is the right play, but I'll accept that it's not enough to prevent harm in the least convenient possible world.

In that world, my actual response is that it's not my fault. It's bad for food critics to get threats! I unequivocally condemn people who do that! If there's some sort of causal relationship between me telling the truth about food, and food critics getting threats, that runs through other agents who are not me who won't stop their crimes even if I condemn them ... well, that's a really terrible situation, but I'm not going to stop telling the truth about food. I don't negotiate with terrorists!

Comment by zack_m_davis on Inaccessible finely tuned RNG in humans? · 2020-10-07T18:17:05.301Z · LW · GW

Getting a single random choice (e.g., of poll options) and getting a random sequence (e.g., of numbers, or coin faces, or star locations) are different tasks. Perhaps the problem isn't that people can't make an approximately random choice; it's that they can't make independent random choices. That's why people place the stars equidistantly or avoid repetitions. Having already placed the first star, or already said "Heads", we don't know how to forget. The second star, the second flip, is chosen in a different mental context: not in an unmarked uniform space, but in a space that already has a star on it.

Comment by zack_m_davis on What are examples of Rationalist fable-like stories? · 2020-09-28T17:59:30.824Z · LW · GW
Comment by zack_m_davis on Blog posts as epistemic trust builders · 2020-09-28T05:13:11.517Z · LW · GW

But fortunately, I have a very high level of epistemic trust for the rationalist community.

No! Not fortunately! Speaking from personal experience, succumbing to the delusion that there is any such thing as "the rationalist community" worthy of anyone's trust has caused me an enormous amount of psychological damage, such that I'm still (still?!) not done recovering from the betrayal trauma more than three years later—and I'm probably not the only one.

(Uh, can't say I wasn't warned.)

(I thought I was done recovering as of (specifically) 13 September, but the fact that I still felt motivated to write a "boo 'rationalists'" comment on Friday and then went into an anxiety spiral for the next 36 hours—and the fact that I'm drafting this comment in a paper notebook when I should be spending a relaxing network-free Sunday studying math—suggest that I'm still (somehow—still?!) not done grieving. I think I'm really close, though!)

There is no authority regulating who's allowed to use the "rationalist" brand name. Trusting "the rationalist community" leaves you open to getting fucked over[1] by any bad idea that can successfully market itself to high-Openness compsci nerds living in Berkeley, California in the current year. The craft is not the community. The ideology is not the movement. Don't revere the bearer of good info. Every cause—every cause—wants to be a cult. At this point, as a guard against my earlier mistakes, I've made a habit of using the pejorative "robot cult" to refer to the social cluster, reserving "rationalist" to describe the methodology set forth in the Sequences—and really, I should probably phase out "rationalist", too. Plain rationality is already a fine word for cognitive algorithms that create and exploit map–territory correspondences—maybe it doesn't have to be an -ism.

Real trust—trust that won't predictably blow up in your face and take three and a half years to recover from—needs to be to something finer-grained than some amorphous self-recommending "community." You need to model the specific competencies of specific people and institutions, and model their incentives to tell you the truth—or to fuck with you.

(Note: people don't have to be consciously fucking with you in order for modeling them as fucking with you to be useful for compressing the length of the message needed to describe your observations. I can't speak to what the algorithms of deception feel from the inside—just that the systematic production of maps that don't reflect the territory for any reason, even mere "bias", should be enough for you to mark them as hostile.)

COVID-19 is an unusually easy case, where people's interests are, for the most part, aligned. People may use the virus as a prop in their existing political games, but at least no one is actually pro-virus. Under those circumstances, sure, trusting an amorphous cluster of smart people who read each other's blogs can legitimately be a better bet than alternative aggregators of information. As soon as you step away from the unusually easy cases—watch your step!

If you learned a lot from the Sequences, I think that's a good reason to trust what Eliezer Yudkowsky in particular says about AI in particular, even if you can't immediately follow the argument. (There's a prior that any given nonprofit claiming you should give it money in order to prevent the destruction of all value in the universe is going to just be a scam—but you see, the Sequences are very good.) That trust does not bleed over (except at a very heavy quantitative discount) to an alleged "community" of people who also trust Yudkowsky—and I don't think it bleeds over to Yudkowsky's off-the-cuff opinions on (let's say, picking an arbitrary example) the relative merits of polyamory, unless you have some more specific reason to trust that he actually thought it through and actually made sure to get that specific question right, rather than happening to pick up that answer through random cultural diffusion from his robot cult. (Most people get most of their beliefs from random cultural diffusion; we can't think fast enough to do otherwise.) Constant vigilance!


  1. I (again) feel bad about cussing in a Less Wrong comment, but I want to be very emphatic here! ↩︎

Comment by zack_m_davis on Artificial Intelligence: A Modern Approach (4th edition) on the Alignment Problem · 2020-09-28T04:31:34.548Z · LW · GW

(This extended runaround on appeals to consequences is at least a neat microcosm of the reasons we expect unaligned AIs to be deceptive by default! Having the intent to inform other agents of what you know without trying to take responsibility for controlling their decisions is an unusually anti-natural shape for cognition; for generic consequentialists, influence-seeking behavior is the default.)

Comment by zack_m_davis on The rationalist community's location problem · 2020-09-25T23:40:04.727Z · LW · GW

But if the rationalist project is supposed to be about spreading our ideas and achieving things [emphasis mine]

Thanks for phrasing this as a conditional! To fill in another branch of the if/else-if/else-if ... conditional statement: if the rationalist project is supposed to be about systematically correct reasoning—having the right ideas because they're right, rather than spreading our ideas because they're ours—then things that are advantageous to the movement could be disadvantageous to the ideology, if the needs of growing the coalition's resources conflict with the needs of constructing shared maps that reflect the territory.

if we're trying to get the marginal person who isn't quite a community member yet but occasionally reads Less Wrong to integrate more

I don't know who "we" are, but my personal hope for the marginal person who isn't quite a community member but occasionally reads this website isn't that they necessarily integrate with the community, but that they benefit from understanding the ideas that we talk about on this website—the stuff about science and Bayesian reasoning, which, being universals, bear no distinguishing evidence of their origin. I wouldn't want to privilege the hypothesis that integrating with the community is the right thing to do if you understand the material, given the size of the space of competing alternatives. (The rest of the world is a much larger place than "the community"; you need more evidence to justify the plan of reorganizing your life around a community qua community than you do to justify the plan of reading an interesting blog.)

Comment by zack_m_davis on Has anyone written stories happening in Hanson's em world? · 2020-09-21T17:57:17.966Z · LW · GW

"Blame Me for Trying"

Comment by zack_m_davis on Open & Welcome Thread - September 2020 · 2020-09-19T22:15:21.678Z · LW · GW

I've reliably used the word "threat" to simply mean signaling some kind of intention of inflicting some kind punishment in response to some condition by the other person. Curi and other people from FI have done this repeatedly, and the "list of people who have evaded/lied/etc." is exactly one of such threats, whether explicitly labeled as such or not.

This game-theoretic concept of "threat" is fine, but underdetermined: what counts as a threat in this sense depends on where the the "zero point" is; what counts as aggression versus self-defense depends on what the relevant "property rights" are. (Scare quotes on "property rights" because I'm not talking about legal claims, but "property rights" is an apt choice of words, because I'm claiming that the way people negotiate disputes that don't rise to the level of dragging in the (slow, expensive) formal legal system, have a similar structure.)

If people have a "right" to not be publicly described as lying, evading, &c., then someone who puts up a "these people lied, evaded, &c." page on their own website is engaging in a kind of aggression. The page functions as a threat: "If you don't keep engaging in a way that satisfies my standards of discourse, I'll publicly call you a liar, evader, &c.."

If people don't have a "right" to not be publicly described as lying, evading, &c., then a website administrator who cites a user's "these people lied, evaded, &c." page on their own website as part of a rationale for banning that user, is engaging in a kind of aggression. The ban functions as a threat: "If you don't cede your claim on being able to describe other people as lying, evading, &c., I won't let you participate in this forum."

The size of the website administrator's threat depends on the website's "market power." Less Wrong is probably small enough and niche enough such that the threat doesn't end up controlling anyone's off-site behavior: anyone who perceives not being able to post on Less Wrong as a serious threat is probably already so deeply socially-embedded into our little robot cult, that they either have similar property-rights intuitions as the administrators, or are too loyal to the group to publicly accuse other group members as lying, evading, &c., even if they privately think they are lying, evading, &c.. (Nobody likes self-styled whistleblowers!) But getting kicked off a service with the market power of a Google, Facebook, Twitter, &c. is a sufficiently big deal to sufficiently many people such that those websites' terms-of-service do exert some controlling pressure on the rest of Society.

What are the consequences of each of these "property rights" regimes?

In a world where people have a right to not be publicly described as lying, evading, &c., then people don't have to be afraid of losing reputation on that account. But we also lose out on the possibility of having a public accounting of who has actually in fact lied, evaded, &c.. We give up on maintaining the coordination equilibrium such that words like "lie" have a literal meaning that can actually be true or false, rather than the word itself simply constituting an attack.

Which regime better fulfills our charter of advancing the art of human rationality? I don't think I've written this skillfully enough for you to not be able to guess what answer I lean towards, but you shouldn't trust my answer if it seems like something I might lie or evade about! You need to think it through for yourself.

Comment by zack_m_davis on Causal Reality vs Social Reality · 2020-09-18T16:50:30.131Z · LW · GW

No problem. Hope your research is going well!

(Um, as long as you're initiating an interaction, maybe I should mention that I have been planning to very belatedly address your concern about premature abstraction potentially functioning as a covert meta-attack by putting up a non-Frontpagable "Motivation and Political Context for My Philosophy of Language Agenda" post in conjunction with my next philosophy-of-language post? I'm hoping that will make things better rather than worse from your perspective? But if not, um, sorry.)

Comment by zack_m_davis on Artificial Intelligence: A Modern Approach (4th edition) on the Alignment Problem · 2020-09-18T03:51:44.214Z · LW · GW

Can I also point to this as (some amount of) evidence against concerns that "we" (members of this stupid robot cult that I continue to feel contempt for but don't know how to quit) shouldn't try to have systematically truthseeking discussions about potentially sensitive or low-status subjects because guilt-by-association splash damage from those conversations will hurt AI alignment efforts, which are the most important thing in the world? (Previously: 1 2 3.)

Like, I agree that some nonzero amount of splash damage exists. But look! The most popular AI textbook, used in almost fifteen hundred colleges and universities, clearly explains the paperclip-maximizer problem, in the authorial voice, in the first chapter. "These behaviors are not 'unintelligent' or 'insane'; they are a logical consequence of defining winning as the sole objective for the machine." Italics in original! I couldn't transcribe it, but there's even one of those pay-attention-to-this triangles (◀) in the margin, in teal ink.

Everyone who gets a CS degree from this year onwards is going to know from the teal ink that there's a problem. If there was a marketing war to legitimize AI risk, we won! Now can "we" please stop using the marketing war as an excuse for lying?!

Comment by zack_m_davis on Artificial Intelligence: A Modern Approach (4th edition) on the Alignment Problem · 2020-09-18T03:39:17.627Z · LW · GW

It is an "iff" in §16.7.2 "Deference to Humans", but the toy setting in which this is shown is pretty impoverished. It's a story problem about a robot Robbie deciding whether to book an expensive hotel room for busy human Harriet, or whether to ask Harriet first.

Formally, let be Robbie's prior probability density over Harriet's utility for the proposed action a. Then the value of going ahead with a is

(We will see shortly why the integral is split up this way.) On the other hand, the value of action d, deferring to Harriet, is composed of two parts: if u > 0 then Harriet lets Robbie go ahead, so the value is us, but if u < 0 then Harriet switches Robbie off, so the value is 0:

Comparing the expressions for EU(a) and EU(d), we see immediately that

because the expression for EU(d) has the negative-utility region zeroed out. The two choices have equal value only when the negative region has zero probability—that is, when Robbie is already certain that Harriet likes the proposed action.

(I think this is fine as a topic-introducing story problem, but agree that the sentence in Chapter 1 referencing it shouldn't have been phrased to make it sound like it applies to machines-in-general.)

Comment by zack_m_davis on Decoherence is Falsifiable and Testable · 2020-09-12T00:36:21.481Z · LW · GW

It's mentioned in passing in the "Technical Explanation" (but yes, not a full independently-linkable post):

Humans are very fond of making their predictions afterward, so the social process of science requires an advance prediction before we say that a result confirms a theory. But how humans may move in harmony with the way of Bayes, and so wield the power, is a separate issue from whether the math works. When we’re doing the math, we just take for granted that likelihood density functions are fixed properties of a hypothesis and the probability mass sums to 1 and you’d never dream of doing it any other way.

Comment by zack_m_davis on How easily can we separate a friendly AI in design space from one which would bring about a hyperexistential catastrophe? · 2020-09-11T15:22:05.633Z · LW · GW

Sleep is very important! Get regular sleep every night! Speaking from personal experience, you don't want to have a sleep-deprivation-induced mental breakdown while thinking about Singularity stuff!

Comment by zack_m_davis on Tofly's Shortform · 2020-09-06T01:42:36.720Z · LW · GW

Yudkowsky addresses some of these objections in more detail in "Intelligence Explosion Microeconomics".

Comment by zack_m_davis on Sherrinford's Shortform · 2020-08-02T20:20:33.091Z · LW · GW

The wikipedia article, as far as I can see, explains in that paragraph where the neoreactionary movement originated.

It's not true, though! The article claims: "The neoreactionary movement first grew on LessWrong, attracted by discussions on the site of eugenics and evolutionary psychology".

I mean, okay, it's true that we've had discussions on eugenics and evolutionary psychology, and it's true that a few of the contrarian nerds who enthusiastically read Overcoming Bias back in the late 'aughts were also a few of the contrarian nerds who enthusiastically read Unqualified Reservations. But "first grew" (Wikipedia) and "originated" (your comment) really doesn't seem like a fair summary of that kind of minor overlap in readership. No one was doing neoreactionary political theorizing on this website. Okay, I don't have a exact formalization of what I mean by "no one" in the previous sentence because I haven't personally read and remembered every post in our archives; maybe there are nonzero posts with nonnegative karma that could be construed to match this description. Still, in essence, you can only make the claim "true" by gerrymandering the construal of those words.

And yet the characterization will remain in Wikipedia's view of us—glancing at the talk page, I don't expect to win an edit war with David Gerard.

Comment by zack_m_davis on Generalized Efficient Markets in Political Power · 2020-08-01T19:07:10.765Z · LW · GW

It gets worse. We also face coordination problems on the concepts we use to think with. In order for language to work, we need shared word definitions, so that the probabilistic model in my head when I say a word matches up with the model in your head when you heard the word. A leader isn't just in a position to coordinate what the group does, but also which aspects of reality the group is able to think about.

Comment by zack_m_davis on Open & Welcome Thread - July 2020 · 2020-07-25T19:46:21.868Z · LW · GW

I'm disappointed that the LaTeX processor doesn't seem to accept \nicefrac ("TeX parse error: Undefined control sequence \nicefrac"), but I suppose \frac will suffice.