Estimating COVID-19 Mortality Rates 2020-06-07T15:45:30.271Z
Can crimes be discussed literally? 2020-03-22T20:17:05.545Z
When to Reverse Quarantine and Other COVID-19 Considerations 2020-03-10T16:37:41.717Z
Simulacra and Subjectivity 2020-03-05T16:25:10.430Z
REVISED: A drowning child is hard to find 2020-01-31T18:07:05.348Z
Approval Extraction Advertised as Production 2019-12-15T14:51:44.878Z
Judgment, Punishment, and the Information-Suppression Field 2019-11-06T17:44:38.908Z
Blatant lies are the best kind! 2019-07-03T20:45:56.948Z
Reason isn't magic 2019-06-18T04:04:58.390Z
Drowning children are rare 2019-05-28T19:27:12.548Z
A War of Ants and Grasshoppers 2019-05-22T05:57:37.236Z
Towards optimal play as Villager in a mixed game 2019-05-07T05:29:50.826Z
Hierarchy and wings 2019-05-06T18:39:43.607Z
Blame games 2019-05-06T02:38:12.868Z
Should Effective Altruism be at war with North Korea? 2019-05-05T01:50:15.218Z
Totalitarian ethical systems 2019-05-03T19:35:28.800Z
Authoritarian Empiricism 2019-05-03T19:34:18.549Z
Excerpts from a larger discussion about simulacra 2019-04-10T21:27:40.700Z
Blackmailers are privateers in the war on hypocrisy 2019-03-14T08:13:12.824Z
Moral differences in mediocristan 2018-09-26T20:39:25.017Z
Against the barbell strategy 2018-09-20T15:19:08.185Z
Interpretive Labor 2018-09-05T18:36:49.566Z
Zetetic explanation 2018-08-27T00:12:14.076Z
Model-building and scapegoating 2018-07-27T16:02:46.333Z
Culture, interpretive labor, and tidying one's room 2018-07-26T20:59:52.227Z
There is a war. 2018-05-24T06:44:36.197Z
Talents 2018-05-18T20:30:01.179Z
Oops Prize update 2018-04-20T09:10:00.873Z
Humans need places 2018-04-19T19:50:01.931Z
Kidneys, trade, sacredness, and space travel 2018-03-01T05:20:01.457Z
What strange and ancient things might we find beneath the ice? 2018-01-15T10:10:01.010Z
Explicit content 2017-12-02T00:00:00.946Z
Cash transfers are not necessarily wealth transfers 2017-12-01T10:10:01.038Z
Nightmare of the Perfectly Principled 2017-11-02T09:10:00.979Z
Poets are intelligence assets 2017-10-25T03:30:01.029Z
Seeding a productive culture: a working hypothesis 2017-10-18T09:10:00.882Z
Defense against discourse 2017-10-17T09:10:01.023Z
On the construction of beacons 2017-10-16T09:10:00.866Z
Sabbath hard and go home 2017-09-27T07:49:40.482Z
Why I am not a Quaker (even though it often seems as though I should be) 2017-09-26T07:00:28.116Z
Bad intent is a disposition, not a feeling 2017-05-01T01:28:58.345Z
Actors and scribes, words and deeds 2017-04-26T05:12:29.199Z
Effective altruism is self-recommending 2017-04-21T18:37:49.111Z
An OpenAI board seat is surprisingly expensive 2017-04-19T09:05:04.032Z
OpenAI makes humanity less safe 2017-04-03T19:07:51.773Z
Against responsibility 2017-03-31T21:12:12.718Z
Dominance, care, and social touch 2017-03-29T17:53:20.967Z
The D-Squared Digest One Minute MBA – Avoiding Projects Pursued By Morons 101 2017-03-19T18:48:55.856Z
Threat erosion 2017-03-15T23:32:30.000Z
Sufficiently sincere confirmation bias is indistinguishable from science 2017-03-15T13:19:05.357Z


Comment by Benquo on Occupational Infohazards · 2021-12-24T02:17:10.840Z · LW · GW

Listing "I don't know, some other reason we haven't identified yet" as an "obvious source" can make sense as a null option, but giving it a virtus dormitiva type name is silly.

I think that Jessica has argued with some plausibility that her psychotic break was in part the result of taking aspects of the AI safety discourse more seriously and unironically than the people around her, combined with adversarial pressures and silencing. This seems like a gears-level model that might be more likely in people with a cognitive disposition correlated with psychosis.

Comment by Benquo on Occupational Infohazards · 2021-12-23T01:02:29.990Z · LW · GW

When I got back into town and talked with Jessica, she was talking about how it might be wrong to take actions that might possibly harm others, i.e. pretty much any actions, since she might not learn fast enough for this to come out net positive. Seems likely to me that the content of Jessica's anxious perseveration was partly causally upstream of the anxious perseveration itself.

I agree that a decline in bodily organization was the main legitimate reason for concern. It seems obviously legitimate for Jessica (and me) to point out that Scott is proposing a standard that cannot feasibly be applied uniformly, since it's not already common knowledge that Scott isn't making sense here, and his prior comments on this subject have been heavily upvoted. The main alternative would be to mostly stop engaging on LessWrong, which I have done.

I don't fully understand what "latent tendency towards psychosis" means functionally or what predictions it makes, so it doesn't seem like an adequate explanation. I do know that there's correlation within families, but I have a family history of schizophrenia and Jessica doesn't, so if that's what you mean by latent tendency it doesn't seem to obviously have an odds ratio in the correct direction within our local cluster.

Comment by Benquo on Occupational Infohazards · 2021-12-22T21:40:33.821Z · LW · GW

My recollection is that at that time you were articulately expressing what seemed like a level of scrupulosity typical of many Bay Area Rationalists. You were missing enough sleep that I was worried, but you seemed oriented x3. I don't remember you talking about demons or auras at all, and have no recollection of you confusedly reifying agents who weren't there.

Comment by Benquo on Occupational Infohazards · 2021-12-21T01:40:28.170Z · LW · GW

The main thing I'd fight if I felt fighty right now is the claim that by not listening to talk about demons and auras MIRI (or by extension me, who endorsed MIRI's decision) is impinging on her free speech.

You wrote that talking about auras and demons the way Jessica did while at MIRI should be considered a psychiatric emergency. When done by a practicing psychiatrist this is an impingement on Jessica's free speech. You wrote this in response to a post that contained the following and only the following mentions of demons or auras:

  1. During this time, I was intensely scrupulous; I believed that I was intrinsically evil, had destroyed significant parts of the world with my demonic powers, and was in a hell of my own creation. [after Jessica had left MIRI]
  2. I heard that the paranoid person in question was concerned about a demon inside him, implanted by another person, trying to escape. [description of what someone else said]
  3. The weirdest part of the events recounted is the concern about possibly-demonic mental subprocesses being implanted by other people. [description of Zoe's post]
  4. As weird as the situation got, with people being afraid of demonic subprocesses being implanted by other people, there were also psychotic breaks involving demonic subprocess narratives around MIRI and CFAR. [description of what other people said, and possibly an allusion to the facts described in the first quote, after she had left MIRI]
  5. While in Leverage the possibility of subtle psychological influence between people was discussed relatively openly, around MIRI/CFAR it was discussed covertly, with people being told they were crazy for believing it might be possible.  (I noted at the time that there might be a sense in which different people have "auras" in a way that is not less inherently rigorous than the way in which different people have "charisma", and I feared this type of comment would cause people to say I was crazy.)

Only the last one is a description of a thing Jessica herself said while working at MIRI. Like Jessica when she worked at MIRI, I too believe that people experiencing psychotic breaks sometimes talk about demons. Like Jessica when she worked at MIRI, I too believe that auras are not obviously less real than charisma. Am I experiencing a psychiatric emergency?

Comment by Benquo on Quis cancellat ipsos cancellores? · 2021-12-20T05:42:30.551Z · LW · GW

Elsewhere, you wrote:

I was the first person to publish the accusations of harm against brent on behalf of my friend who was his ex

If the mittenscautious Tumblr or Medium account was not run by you, in what sense was this true?

Comment by Benquo on Occupational Infohazards · 2021-12-18T23:56:19.547Z · LW · GW

I'm just confused - if there's agreement that MIRI isn't particularly bad about this, then this seems to mostly preclude environmental attribution and suggest personal attribution?

I've read Moral Mazes and worked a few years in the corporate world at Fannie Mae. I've also talked a lot with Jessica and others in the MIRI cluster who had psychotic breaks. It seems to me like what happens to middle managers is in some important sense even worse than a psychotic break. Jessica, Zack, and Devi seem to be able to represent their perspectives now, to be able to engage with the hypothesis that some activity is in good faith, to consider symmetry considerations instead of reflexively siding with transgressors.

Ordinary statistical methods - and maybe empiricism more generally - cannot shed light on pervasive, systemic harms, when we lack the capacity to perform controlled experiments on many such systems. In such cases, we instead need rationalist methods, i.e. thinking carefully about mechanisms from first principles. We can also try to generalize efficiently from microcosms of the general phenomenon, e.g. generalizing from how people respond to unusually blatant abuse by individuals or institutions, to make inferences about the effects of pervasive abuse.

But corporate employers are not the only context people live in. My grandfather was an independent insurance broker for much of his career. I would expect someone working for a low-margin business in a competitive industry to sustain much less psychological damage, though I would also expect them to be paid less and maybe have a more strenuous job. I don't think the guys out on the street a few blocks from my apartment selling fruit for cash face anything like what Jessica faced, and I'd be somewhat surprised if they ended up damaged the way the people in Moral Mazes seem damaged.

Comment by Benquo on The Rationalists of the 1950s (and before) also called themselves “Rationalists” · 2021-12-12T09:10:14.697Z · LW · GW

If I try to steelman the Rationalist-Empiricist divide:

Empiricists think that arguments justifying organized violence are nonsense so we ought to ignore them, do what we like instead, and argue about math and science.

Rationalists think that arguments justifying organized violence are sketchy so we should investigate them carefully as hypotheses for how mind organizes itself in the world.

Comment by Benquo on The Rationalists of the 1950s (and before) also called themselves “Rationalists” · 2021-12-06T20:15:14.998Z · LW · GW

This distinction was always fake. The Wikipedia page on Rationalism begins with portraits of Descartes, Spinoza, and Leibniz. Spinoza was a lens grinder who worked closely with astronomer-physicist Christiaan Huygens and wrote in his magnum opus, Ethics, that we only know about things in the world through our bodies interacting with them. It is unclear to me how it is possible for someone to be more committed to looking at the world.  The Wikipedia page on Empiricism begins with portraits of Francis Bacon, John Locke, and David Hume. Hume's An Enquiry Concerning Human Understanding includes the following, which implies that abstract mathematical reasoning is one of the two valid sources of knowledge, and refers to experimental reasoning:

If we take in our hand any volume; of divinity or school metaphysics, for instance; let us ask, Does it contain any abstract reasoning concerning quantity or number? No. Does it contain any experimental reasoning concerning matter of fact and existence? No. Commit it then to the flames: for it can contain nothing but sophistry and illusion.

Comment by Benquo on Frame Control · 2021-12-06T18:18:23.800Z · LW · GW

A non-conscious intent? What on earth does this mean?

A brain can run computations optimizing for an outcome without running the additional computations needed to represent this optimization target to itself in explicit self-models available to reflective cognition.

Robin Hanson's developed extensive, detailed sociological models that include this component. I think that the entire Overcoming Bias archives, not just Eliezer's Sequences, ought to be canonical here, both because of their intellectual merit and because most of the Sequences were originally written on the Overcoming Bias blog in dialogue with Robin and his other co-bloggers there. A Theory of Identity seems particularly relevant here, and develops a directly relevant claim:

Our conscious minds are the public relations department of our larger minds, presenting and managing a story of ourselves to others.

Comment by Benquo on Communication Requires Common Interests or Differential Signal Costs · 2021-11-28T14:08:17.376Z · LW · GW

The stronger the common interest in accurate signaling, the higher the opportunity cost of doing something else, which is sufficient for an economic differential signaling cost, though not necessarily an evolutionary one; in some cases where the expected value of honesty far exceeds the expected value of dishonesty we may still want to prevent dishonest behavior from being selected for too fast to reach the payoff.

Comment by Benquo on Frame Control · 2021-11-28T13:30:57.955Z · LW · GW

This reply would be more interesting if it engaged with the last two paragraphs of my comment, in which I tried to develop a relevant causal hypothesis.

Comment by Benquo on Almost everyone should be less afraid of lawsuits · 2021-11-28T04:57:08.198Z · LW · GW

an organization that always listens to lawyers is being more cautious than the optimum.

Even an organization that disobeys the lawyers sometimes but never regrets not having obeyed them is being more than optimally cautious.

Comment by Benquo on Frame Control · 2021-11-28T04:25:33.472Z · LW · GW

This phenomenological account of frame control doesn't provide a causal model precise enough for me to understand what additional question someone would be trying to answer when asking "is this person doing frame control?" aside from noticing which of the features of "frame control" they satisfy.

Some of the "red flags" seem like they could equally well point to someone fanatically committed to totally dominating others, or someone whose perspective responds to evidence but not social pressure, and many of the signs that someone is not a "frame controller" seem like the opposite. From the description of the first red flag:

They don’t laugh nervously, don’t give tiny signals that they are malleable and interested in conforming to your opinion or worldview.

These are signs of submission, as is the description of the first sign that someone isn't "doing frame control":

They give you power over them, like indications that they want your approval or unconditional support in areas you are superior to them. They signal to you that they are vulnerable to you.

This resembles a pattern I see across many contexts where the idea of listening to someone because they have something to say is replaced with the submissive behavior of believing someone out of loyalty. I don't mean to say anything about your intent here, just to point out the general pattern, so we have a better chance of doing something else instead.

Second red flag:

A key component that makes frame control dangerous is when it’s linked to concrete consequences; maybe people really respect them, maybe they control resources, maybe they are the person throwing big events, maybe they gave you a new name, maybe they have the power to exclude you from your social group.

In general I would expect people whose beliefs are more responsive to valid evidence to have an easier time acting to produce the consequences they want. In environments that allocate credit straightforwardly I would also expect them to be more respected, entrusted with more resources, and have their concerns about others listened to more. In perverse environments I would expect the opposite.

The other red flags and opposite signs seem more ground-truth-oriented if I take them literally, as long as we don't skip the step of investigating whether what the other person is saying is false or true. For instance, reframing "harm" as "beneficial" is helpful if I start out by mistakenly think that my discomfort confronting a criticism is a sign that the criticism is hurting me, but harmful if I'm being trained to treat the grinding-my-gears feeling of endorsing an incoherent narrative as a sign of healthy spiritual challenge.

If you can evaluate directly whether someone has more of a tendency to introduce errors or to correct them, then you can additionally judge whether they are introducing errors in a way calculated to make error-correction harder, to make themselves more central, and to make you more responsive to their commands. If they're just giving commands and not making claims, then you can skip the first step and just see whether you think their commands are making you better off. In either case, it can help a lot to check whether you are behaving as though the other person is threatening you, whether they are behaving in a way optimized to produce that impression, and whether on reflection you believe that the threats are credible.

If you can't evaluate that, then you're in a more confusing situation. It can often be a good idea to switch contexts and see if things seem clear when outside that particular social field. If not, then you're stuck, and at best you can decide which interactions seem to give you the best chance of achieving autonomy.

Comment by Benquo on Which booster shot to get and when? · 2021-11-21T01:30:04.423Z · LW · GW

This prison study is weird but appears to show that Moderna is MUCH more effective than the other vaccines. Applying Robin Hanson's heuristic that the non-headline numbers in a study are less biased than the headline numbers, we should maybe treat this as more credible than official estimates of relative efficacy.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-11-21T01:09:09.415Z · LW · GW

What do you see as the main sorts of interventions CFAR was organized around? I feel like this is a "different worlds" thing where I ought to be pretty curious what the whole scene looked like to you, what it seemed like people were up to, what the important activities were, & where progress was being made (or attempted).

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-11-21T00:58:41.688Z · LW · GW

What do you see as the main sorts of interventions CFAR was organized around? I feel like this is a "different worlds" thing where I ought to be pretty curious what the whole scene looked like to you, what it seemed like people were up to, what the important activities were, & where progress was being made (or attempted).

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-11-21T00:54:37.438Z · LW · GW

I was relying on her self-reports, like

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-11-21T00:53:22.197Z · LW · GW

Specific claim: this is how to take over New York.

Didn’t work.

I think this needs to be broken up into 2 claims:

1 If we execute strategy X, we'll take over New York. 2 We can use straightforward persuasion (e.g. appeals to reason, profit motive) to get an adequate set of people to implement strategy X.

2 has been falsified decisively. The plan to recruit candidates via appealing to people's explicit incentives failed, there wasn't a good alternative, and as a result there wasn't a chance to test other parts of the plan (1).

That's important info and worth learning from in a principled way. Definitely I won't try that sort of thing again in the same way, and it seems like I should increase my credence both that plans requiring people to respond to economic incentives by taking initiative to play against type will fail, and that I personally might be able to profit a lot by taking initiative to play against type, or investing in people who seem like they're already doing this, as long as I don't have to count on other unknown people acting similarly in the future.

But I find the tendency to respond to novel multi-step plans that would require someone do take initiative by sitting back and waiting for the plan to fail, and then saying, "see? novel multi-step plans don't work!" extremely annoying. I've been on both sides of that kind of transaction, but if we want anything to work out well we have to distinguish cases of "we / someone else decided not to try" as a different kind of failure from "we tried and it didn't work out."

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-11-21T00:36:24.527Z · LW · GW

Specific claim: the only nontrivial obstacle in front of us is not being evil

This is false. Object-level stuff is actually very hard.

This seems to be conflating the question of "is it possible to construct a difficult problem?" with the question of "what's the rate-limiting problem?". If you have a specific model for how to make things much better for many people by solving a hard technical problem before making substantial progress on human alignment, I'd very much like to hear the details. If I'm persuaded I'll be interested in figuring out how to help.

So far this seems like evidence to the contrary, though, as it doesn't look like you thought you could get help making things better for many people by explaining the opportunity.

Comment by Benquo on Discussion with Eliezer Yudkowsky on AGI interventions · 2021-11-16T03:07:09.793Z · LW · GW

This is mostly a retread of confusions addressed in the Sequences. The main change in the discourse since 2011 seems to be new interest in deep learning. It seems to me like Eliezer is wasting his time exhibiting a "keep trying" behavior, and should expect to do better with a "reorient" behavior such as further investigating the causes of this sort of recalcitrance.

Comment by Benquo on Discussion with Eliezer Yudkowsky on AGI interventions · 2021-11-16T03:04:48.117Z · LW · GW

This reads like nothing new has happened in the discourse since 2011 except the development of interest in deep learning. Eliezer is still arguing with people exhibiting the same sorts of confusions he addressed in the Sequences. This seems like evidence that the "keep trying" behavior has long since hit diminishing returns relative to what I'd expect a "reorient" behavior to accomplish, e.g. more investigation into the causes of this kind of recalcitrance.

Comment by Benquo on Tell the Truth · 2021-10-31T02:10:38.453Z · LW · GW

Your failure starts with summarizing everything under one single label, here we are dealing with the word “liberalism.”

This reflects an empiricist view of language which is not compatible with analytic rigor. The kind of "liberalism" undermined by info about unequal capacities (a leftist commitment to economic leveling among citizens) is only associated with the word due to contingent facts about American coalitional politics. The natural cluster in political perspectives that "liberal" was invented to represent does not entail this position. Zack Davis's Where to Draw the Boundaries? is relevant. If you don't know the technical meaning of a term, this is a fact about you, not about the person using it.

Comment by Benquo on Selfishness, preference falsification, and AI alignment · 2021-10-30T20:31:24.929Z · LW · GW

Fred Kinnan is a comparatively sympathetic character among the looter coalition, for more or less the reason you just described. I think Rand's opinion is that people like Kinnan are being locally rational & self-interested, but within a worldview that is truncated in an unprincipled way to embed a conflict theory that is in tension with his ability to recognize & extract material concessions and, if taken to its logical conclusion, involves a death wish. It doesn't seem like he's enjoying his life or really has any specific concrete intentions.

Robert Stadler is another interesting mixed character. He starts out with specific intentions (learning how the physical world works on a deep level). This eventually puts him in conflict with the looters, and unlike the viewpoint character Danny Taggart he submits to their worldview, giving up his sanity & the agenda that made his life worth living in order to occupy a place in their regime.

Kinnan is better adapted to cynically hold onto his position for longer, but at the price of the kinds of hopes that created a conflict for Stadler.

Comment by Benquo on Everything Studies on Cynical Theories · 2021-10-28T17:18:14.424Z · LW · GW

This was the intent, I think; people engaged in modernistic state violence who won't listen to carefully reasoned heterodox perspectives are now scared enough to consciously notice there's some lawless violence going on, and getting worse at their jobs.

Too bad a bunch of good stuff is being interfered with too, but that's how total war works. In the short run we're less likely to have a Holocaust but maybe more likely to have famines, plagues, wildfires...

Comment by Benquo on Everything Studies on Cynical Theories · 2021-10-28T17:12:45.910Z · LW · GW

I found this account helpful in contextualizing postmodern fields. Insofar as they come from good intentions, postmodern grievance studies is a response to a society that won't pay for criticism; in the absence of epistemic trade relations or functioning civil courts, the main alternative way to make criticism ecologically sustainable is predation, in which critics expropriate from the agents being criticized, making use of the information advantage derived from possessing a valid critique.

Once you understand clearly that academic fields are not generically natural clusters in inquiry-space, but are social constructs exercising narrative power in the service of political agendas (including their own survival), the natural next step is to learn how to construct a performance that can claim resources to further your own agenda.

Postmodern academic grievance studies seems to be an attempt to construct a sort of church which uses its influence over a credentialing system to exercise moral authority over its members, i.e. to compel signs of submission, and sometimes material concessions.

It doesn't seem like an adequate social substrate for a free, technological society, but it does seem like it might be good at de-escalating some kinds of modern warfare in ways that might make me safer from state violence, and it seems like it might be easy for well-intentioned people who can talk with each other (a huge information advantage) to appease once we abandon the delusion that we live in a liberal society and stop trying to assist genocidal oppressors.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-28T02:05:53.832Z · LW · GW

Seems like you agree with what I actually said, and are claiming to find some implied posture objectionable, but aren't willing to criticize me explicitly enough for me or anyone else to learn from. ¯_(ツ)_/¯

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-25T04:15:13.698Z · LW · GW

I think we need more than two categories here. We can't allocate credit for input, only output. People can learn things by carefully observing stuff, but we shouldn't get to mint social capital as rationalists for hours meditating any more than Darwin's reputation should depend directly on time spent thinking about tortoises.

Discerning investors might profit by recognizing leading indicators of high productivity, but that only works if incentives are aligned, which means, eventually, objective tests. In hindsight it seems very unfortunate that MIRI was not mostly funded by relevant-to-its-expertise prediction markets.

Good art seems like it should make people sexy, not credible.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-25T03:07:44.780Z · LW · GW

I still count them among my closest friends, but I don’t want to be socially liable for the things they say. I don’t want the implicit assumption to be that I’d agree with them or back them up.

Same. I don't think I can exit a faction by declaration without joining another, but I want many of the consequences of this. I think I get to move towards this outcome by engaging nonfactional protocols more, not by creating political distance between me & some particular faction.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-24T00:35:07.326Z · LW · GW

What is psychological collapse?

For those who can afford it, taking it easy for a while is a rational response to noticing deep confusion, continuing to take actions based on a discredited model would be less appealing, and people often become depressed when they keep confusedly trying to do things that they don't want to do.

Are you trying to point to something else?

Personally, I interpreted Vassar’s words as factual claims then tried to implement a strategy on them. When I was surprised by reality a bunch, I updated away.

What specific claims turned out to be false? What counterevidence did you encounter?

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-23T13:14:37.209Z · LW · GW

Rationality ought to be totalizing.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-23T03:23:17.857Z · LW · GW

It seems like you're having difficulty imagining that I'm responding to my situation as I understand it, and I don't know what else you might think I'm doing.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-23T03:08:17.466Z · LW · GW

I think this line of discussion would be well served by marking a natural boundary in the cluster "crazy." Instead of saying "Vassar can drive people crazy" I'd rather taboo "crazy" and say:

Many people are using their verbal idea-tracking ability to implement a coalitional strategy instead of efficiently compressing external reality. Some such people will experience their strategy as invalidated by conversations with Vassar, since he'll point out ways their stories don't add up. A common response to invalidation is to submit to the invalidator by adopting the invalidator's story. Since Vassar's words aren't selected to be a valid coalitional strategy instruction set, attempting to submit to him will often result in attempting obviously maladaptive coalitional strategies.

People using their verbal idea-tracking ability to implement a coalitional strategy cannot give informed consent to conversations with Vassar, because in a deep sense they cannot be informed of things through verbal descriptions, and the risk is one that cannot be described without the recursive capacity of descriptive language.

Personally I care much more, maybe lexically more, about the upside of minds learning about their situation, than the downside of mimics going into maladaptive death spirals, though it would definitely be better all round if we can manage to cause fewer cases of the latter without compromising the former, much like it's desirable to avoid torturing animals, and it would be desirable for city lights not to interfere with sea turtles' reproductive cycle by resembling the moon too much.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-22T03:58:29.714Z · LW · GW

This seems mostly right; they're more likely to think "I don't understand a lot of these ideas, I'll have to think about this for a while" or "I don't understand a lot of these ideas, he must be pretty smart and that's kinda cool" than to feel invalidated by this and try to submit to him in lieu of understanding.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-22T02:31:16.135Z · LW · GW

The people I know who weren't brought up to go to college have more experience navigating concrete threats and dangers, which can't be avoided through conformity, since the system isn't set up to take care of people like them. They have to know what's going on to survive. This results in an orientation less sensitive to subtle threats of invalidation, and that sees more concrete value in being informed by someone.

In general this means that they're much more comfortable with the kind of confrontation Vassar engages in, than high-class people are.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-21T18:04:38.262Z · LW · GW

I did feel inhibited from having as much fun as I'd have liked to in this exchange because it seemed like while you were on the whole trying to make a good thing happen, you were somewhat scared in a triggered and triggerable way. This might have caused the distortion you're describing. Helpful and encouraging to hear that you picked up on that and it bothered you enough to mention.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-20T18:53:38.934Z · LW · GW

In most cases it seems intentional but not deliberate. People will resist pressure to change the pattern, or find new ways to execute it if the specific way they were engaged in this bias is effectively discouraged, but don't consciously represent to themselves their intent to do it or engage in explicit means-ends reasoning about it.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-20T18:48:00.847Z · LW · GW

Question about balance: how do you not end up reifying these in your mind, creating a paranoid sense of ‘there be ghosts lurking in shadows’ ?

Mostly just by trying to think about this stuff carefully, and check whether my responses to it add up & seem constructive. I seem to have been brought up somehow with a deep implicit faith that any internal problem I have, I can solve by thinking about - i.e. that I don't have any internal infohazards. So, once I consciously notice the opportunity, it feels safe to be curious about my own fear, aggression, etc. It seems like many other people don't have this faith, which would make it harder for them to solve this class of problem; they seem to think that knowing about conflicts they're engaged in would get them hurt by making them blameworthy; that looking the thing in the face would mark them for destruction.

My impression is that insofar as I'm paranoid, this is part of the adversarial process I described, which seems to believe in something like ontologically fundamental threats that can't be reduced to specific mechanisms by which I might be harmed, and have to be submitted to absolutely. This model doesn't stand up to a serious examination, so examining it honestly tends to dissolve it.

I've found psychedelics helpful here. Psilocybin seems to increase the conscious salience of fear responses, which allows me to analyze them. In one of my most productive shrooms trips, I noticed that I was spending most of my time pretending to be a reasonable person, under the impression that an abstract dominator wouldn't allow me to connect with other people unless I passed as a simulacrum of a rational agent. I noticed that it didn't feel available to just go to the other room and ask my friends for cuddles because I wanted to, and I considered maybe just huddling under the blankets scared in my bedroom until the trip ended and I became a simulacrum again. Then I decided I had no real incentive do to this, and plenty of incentive to go try to interact with my friends without pretending to be a person, so I did that and it worked.

THC seems to make paranoid thoughts more conscious, which allows me to consciously work through their implications and decide whether I believe them.

I agree that stories with a dramatic villain seem more memetically fit and less helpful, and I avoid them when I notice the option to.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-20T18:24:29.106Z · LW · GW

The effective direction the epistemic modesty / argument from authority bias pushes things, is away from shared narrative as something that dynamically adjusts to new information, and towards shared narrative as a way to identify and coordinate who's receiving instructions from whom.

People frequently make "mistakes" as a form of submission, and it shouldn't be surprising that other types of systematic error function as a means of domination, i.e. of submission enforcement.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-20T17:56:26.032Z · LW · GW

the phrase about ‘adversaries’ sounds like you’re talking about theoretical ghosts to me. But maybe you have real people in mind.

I'm talking about optimizing processes coordinating with copies of themselves, distributed over many people. My blog post Civil Law and Political Drama is a technically precise description of this, though Towards optimal play as Villager in a mixed game adds some color that might be helpful. I don't think my interests are opposed to the autonomous agency of almost anyone. I do think that some common trigger/trauma behavior patterns are coordinating against autonomous human agency.

The gaming detail helps me understand where you're coming from here. I don't think the right way to manage my resource constraints looks very much like playing a game of MtG. I am in a much higher-dimensional environment where most of my time should be spent playing/exploring, or resolving tension patterns that impede me from playing/exploring. My endorsed behavior pattern looks a little more like the process of becoming a good MtG player, or discovering that MtG is the sort of thing I want to get good at. (Though empirically that's not a game it made sense to me to invest in becoming good at - I chose Tai Chi instead for reasons!)

rather, I have an objection to the seeming endorsement of acting from a fear-aligned place.

I endorse using the capacities I already have, even when those capacities are imperfect.

When responding to social conflict, it would almost always be more efficient and effective for me to try to clarify things out of a sense of open opportunity, than from a fear-based motive. This can be true even when a proper decision-theoretic model the situation would describe it as an adversarial one with time pressure; I might still protect my interests better by thinking in a free and relaxed way about the problem, than tensing up like a monkey facing a physical threat.

But a relaxed attitude is not always immediately available to me, and I don't think I want to endorse always taking the time to detrigger before responding to something in the social domain.

Part of loving and accepting human beings as they are, without giving up on intention to make things better, is appreciating and working with the benefits people produce out of mixed motives. There's probably some irrational fear-based motivation in Elon Musk's and Jeff Bezos's work ethic, and maybe they'd have found more efficient and effective ways to help the world if their mental health were better, but I'm really, really glad I get to use Amazon, and that Tesla and SpaceX and Starlink exist, and it's not clear to me that I'd want to advise younger versions of them to spend a lot of time working on themselves first. That seems like making purity the enemy of the good.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-20T16:39:08.724Z · LW · GW

This can be unpacked into an alternative to the charisma theory.

Many people are looking for a reference person to tell them what to do. (This is generally consistent with the Jaynesian family of hypotheses.) High-agency people are unusually easy to refer to, because they reveal the kind of information that allows others to locate them. There's sufficient excess demand that even if someone doesn't issue any actual orders, if they seem to have agency, people will generalize from sparse data to try to construct a version of that person that tells them what to do.

A more culturally central example than Vassar is Dr Fauci, who seems to have mostly reasonable opinions about COVID, but is worshipped by a lot of fanatics with crazy beliefs about COVID.

The charisma hypothesis describes this as a fundamental attribute of the person being worshipped, rather than a behavior of their worshippers.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-20T15:50:21.721Z · LW · GW

This seems exactly backwards, if someone makes uncorrelated errors, they are probably unintentional mistakes. If someone makes correlated errors, they are better explained as part of a strategy.

Once is happenstance. Twice is coincidence. Three times is enemy action.

I can imagine, after reading the sequences, continuing to have the epistemic modesty bias in my own thoughts, but I don't see how I could have been so confused as to refer to it in conversation as a valid principle of epistemology.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-20T15:43:20.133Z · LW · GW

(a) seems implied by Thoreau's opinion, which a lot of people reported finding plausible well before psychedelics, so it's not an ad hoc hypothesis:

The mass of men lead lives of quiet desperation. What is called resignation is confirmed desperation. From the desperate city you go into the desperate country, and have to console yourself with the bravery of minks and muskrats. A stereotyped but unconscious despair is concealed even under what are called the games and amusements of mankind.

A lot of recent philosophers report that people are basically miserable, and psychiatry reports that a lot of people have diagnosable anxiety or depression disorders. This seems consistent with (a).

This is also consistent with my impression, and with the long run improvements in depression - it seems like for a lot of people psychedelics allow them to become conscious of ways they were hurting themselves and living in fear / conflict.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-20T15:36:49.232Z · LW · GW

This would seem to be a good argument for not paying taxes or helping the US government, or in particular an argument for excluding employees of the FBI, CIA, and DEA, since they are the institutions that have engaged in active violence to cause and perpetuate this situation. It doesn't seem like a plausible argument that it's wrong to take illegal drugs, except in the "there is no ethical consumption under capitalism" sense.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-20T08:08:30.654Z · LW · GW

To the best of my knowledge, the leadership of neither MIRI nor CFAR has ever slept with a subordinate, much less many of them.

This issue doesn't seem particularly important to me but the comparison you're making is a good example of a more general problem I want to talk about.

My impression is that the decision structure of CFAR was much less legible & transparent than that of Leverage, so that it would be harder to determine who might be treated as subordinate to whom in what context. In addition, my impression from the years I was around is that Leverage didn't preside over as much of an external scene, - Leverage followers had formalized roles as members of the organization, while CFAR had a "community," many of whom were workshop alumni.

And when we did mess up, we tried extremely hard to publicly and accurately describe our wrongdoing—e.g., Anna and I personally spent hundreds of hours investigating/thinking about the Brent affair, and tried so hard to avoid accidentally doing anti-epistemic reputational control that in my opinion, our writeup about it actually makes CFAR seem much more culpable than I think it was.

Am I missing something here? The communication I read from CFAR seemed like it was trying to reveal as little as it could get away with, gradually saying more (and taking a harsher stance towards Brent) in response to public pressure, not like it was trying to help me, a reader, understand what had happened.

Comment by Benquo on How to always have interesting conversations · 2021-10-20T07:32:42.496Z · LW · GW

In actual conversation I’d suppress this because I suspect the other person will quickly find it extremely annoying.

One of the nicest things anyone's done in conversation with me is say "hold on, I need a few minutes to think about that," actually go off and think for several minutes, and then come back to the conversation with an integrated perspective. I felt deeply respected as a mind.

People who don't appreciate this sort of thing aren't trying to make themselves understood about something surprising, so I expect that by your values you should care less about making them happy to talk with you, except as a way of getting something else from them.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-20T07:15:15.610Z · LW · GW

Thanks for verifying. In hindsight my comment reads as though it was condemning you in a way I didn't mean to; sorry about that.

The thing I meant to characterize as "crazy cult behavior" was people in the comments here attributing things like what you did in your mind to Michael Vassar's spooky mind powers. You seem to be trying to be helpful and informative here. Sorry if my comment read like a personal attack.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-20T07:11:36.473Z · LW · GW

If you want to clarify whether I mean to insinuate something in a particular comment, you could ask, like I asked Eliezer. I'm not going to make my comments longer without a specific idea of what's unclear, that seems pointless.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-20T07:00:23.790Z · LW · GW

I remember being a "guest instructor" at one workshop, and talking about curriculum design with Anna and Kenzi. I was also at a lot of official and unofficial CFAR retreats/workshops/etc. I don't think I participated in much of the normal/official CFAR process, though I did attend the "train the trainers workshop", and in this range of contexts saw some of how decisions were made, how workshops were run, how people related to each other at parties.

As I recall it, what I observed first-hand and was told second-hand at the time confirms bullets 2, 4, and 6 of the top-level comment. Many of the others are about how people felt, and are consistent with what people I knew reported at the time. Nothing in the top-level comment seems dissonant with what I observed.

It seems like there was a lot of fragmentation (which is why we mostly didn't interact). I felt bad about exercising (a small amount of) unaccountable influence at the time through these mechanisms, but I was confused about so much relative to the rate at which I was willing to ask questions that I didn't end up asking about the info-siloing. In hindsight it seems intended to keep the true nature of governance obscure and therefore unaccountable. I did see or at least hear reports of Anna pretending to give different people authority over things and then intervening if they weren't doing the thing she expected, which is consistent with that hypothesis.

I'm afraid I don't remember a lot of details beyond this, I had a lot going on that year aside from CFAR.

My comment initially said 2014-2016 but IIRC my involvement was much less after 2015 so I edited it.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-20T06:33:31.729Z · LW · GW

As a member of that cluster I endorse this description.

Comment by Benquo on My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage) · 2021-10-20T04:45:40.434Z · LW · GW

Aristotle seems (though he's vague on this) to be thinking in terms of fundamental attributes, while I'm thinking in terms of present capacity, which can be reduced by external interventions such as schooling.

Thinking about people I know who've met Vassar, the ones who weren't brought up to go to college* seem to have no problem with him and show no inclination to worship him as a god or freak out about how he's spooky or cultish; to them, he's obviously just a guy with an interesting perspective.

*As far as I know I didn't know any such people before 2020; it's very easy for members of the educated class to mistake our bubble for statistical normality.