My tentative best guess on how EAs and Rationalists sometimes turn crazy

The spiritual world is rife with bad communities and I've picked up a trick for navigating them. Many of the things named in this post could broadly be construed under the heading of "weird power dynamics." Isolation creates weird power dynamics, poor optionality creates weird power dynamics, and drugs, and skewed gender ratios, and etc etc.

When I spot a weird power dynamic I name it out loud to the group. A lot of bad groups will helpfully kick me out themselves. I naturally somewhat shy away from such actions of course, but an action that reliably loses me status points with exactly the people I don't want to be around is great.

It's the emperor's clothes principle: That which can be destroyed by being described by a beginner should be. And the parable illustrates something important about how it works. It needs to be sincere, not snark, criticism, etc.

My default model before reading this post was: some people are very predisposed to craziness spirals. They're behaviorally well-described as "looking for something to go crazy about", not necessarily in a reflectively-endorsed sense, but in the sense that whenever they stumble across something about which one could go crazy (like e.g. lots of woo-stuff), they'll tend to go into a spiral around it.

"AI is likely to kill us all" is definitely a thing in response to which one can fall into a spiral-of-craziness, so we naturally end up "attracting" a bunch of people who are behaviorally well-described as "looking for something to go crazy about". (In terms of pattern matching, the most extreme examples tend to be the sorts of people who also get into quantum suicide, various flavors of woo, poorly executed anthropic arguments, poorly executed acausal trade arguments, etc.)

Other people will respond to basically-the-same stimuli by just... choosing to not go crazy (to borrow a phrase from Nate). They'll see the same "AI is likely to kill us all" argument and respond by doing something useful, or just ignoring it, or doing something useless but symbolic and not thinking too hard about it. But they won't panic and then selectively engage with things which amplify their own panic (i.e. craziness-spiral behavior).

On that model, insofar as EAs and rationalists sometimes turn crazy, it's mostly a selection effect. "AI is likely to kill us all" is a kind of metaphorical flypaper for people predisposed to craziness spirals.

After reading the post... the main place where the OP's model and my previous default model conflict is in the extent to which craziness is determined by intrinsic characteristics vs environment. Not yet sure how to resolve that.

I also think this fits the FTX situation quite well. My current best model of what happened at an individual psychological level was many people being attracted to FTX/Alameda because of the potential resources, then many rounds of evaporative cooling as anyone who was not extremely hardcore according to the group standard was kicked out, with there being a constant sense of insecurity for everyone involved that came from the frequent purges of people who seemed to not be on board with the group standard.

While a lot of this post fits with my model of the world (the threat of exile is something I can viscerally feel change what my beliefs are), the FTX part as-written is sufficiently non-concrete to me that I can't tell if it fits or doesn't fit with reality.

Things I currently believe about FTX/Alameda (including from off-the-internet information):

• There was a fair amount of lying to investors from the start.
• From the start it was very chaotic, with terrible security practices and most employees not knowing the net balance of the company within like a factor of 4x, or whether net worth was increasing or decreasing from week to week.
• Massive amounts of legal risk constantly being taken intentionally without much sense of the costs.
• Most of the time making money by being crypto-long, or engaging in other deceptive practices, not by being clever.
• There were lots of bits of unrelated unethical behavior from SBF.

Do you have examples of the evaporative cooling / purges and an explanation for how they fit into your model?

I know at Alameda many of the people I think of as "sensible about risk" or at least "people who are pretty unlikely to do crazy sh*t" left either early or in a massive staff-quit at one point, but I don't have much insight into why they left, and people leave for all sorts of conflicts that aren't about 'fitting in', but due disagreements about strategy or because their boss is unfair/incompetent or for reasons better explained by lots of other models.

I suggest a more straightforward model: taking ideas seriously isn't healthy. Most of the attempts to paint SBF as not really an EA seem like weird reputational saving throws when he was around very early on and had rather deep conviction in things like the St. Petersburg Paradox...which seems like a large part of what destroyed FTX. And Ziz seemed to be one of the few people to take the decision theoretical "you should always act as if you're being simulated to see what sort of decision agent you are" idea seriously...and followed that to their downfall. I read the Sequences, get convinced by the arguments within, donate a six figure sum to MIRI...and have basically nothing to show for it at pretty serious opportunity costs. (And that's before considering Ziz's pretty interesting claims about how MIRI spent donor money.)

In all of these cases, the problem was individual confidence in ideas, not social effects.

My model is instead that the sort of people who are there to fit in aren't the people who go crazy; there are plenty of people in the pews who are there for the church but not the religion. The MOPs and Sociopaths seem to be much, much saner than the Geeks. If that's right, rationality has something much more fundamentally wrong with it.

As a final note, looking back at how AI actually developed, it's pretty striking that there aren't really maximizing AIs out there. Does a LLM take ideas seriously? Do they have anything that we'd recognize as a 'utility function'? It doesn't look like it, but we were promised that the AIs were a danger because they would learn about the world and would then take their ideas about what would happen if they did X vs Y to minmax some objective function. But errors compound.

But if group members are insecure enough, or if there is some limited pool of resources to divide up that each member really wants for themselves, then each member experiences a strong pressure to signal their devotion harder and harder, often burning substantial personal resources.

To add to this: if the group leaders seem anxious or distressed, then one of the ways in which people may signal devotion is by also being anxious and distressed. This will then make everything worse - if you're anxious, you're likely to think poorly and fixate on what you think is wrong without necessarily being able to do any real problem-solving around it. It also causes motivated reasoning about how bad everything is, so that one could maintain that feeling of distress.

In various communities there's often a (sometimes implicit, sometimes explicit) notion of "if you're not freaked out by what's happening, you're not taking things seriously enough". E.g. to take an example from EA/rationalist circles, this lukeprog post, while not quite explicitly saying that, reads to me as coming close (I believe that Luke only meant to say that it's good for people to take action, but the way it's phrased, it implies that you need to feel upset to take any action):

Over the years, my colleagues and I have spoken to many machine learning researchers who, perhaps after some discussion and argument, claim to think there’s a moderate chance — a 5%, or 15%, or even a 40% chance — that AI systems will destroy human civilization in the next few decades. ¹ However, I often detect what Bryan Caplan has called a “missing mood“; a mood they would predictably exhibit if they really thought such a dire future was plausible, but which they don’t seem to exhibit. In many cases, the researcher who claims to think that medium-term existential catastrophe from AI is plausible doesn’t seem too upset or worried or sad about it, and doesn’t seem to be taking any specific actions as a result.

Not so with Elon Musk. Consider his reaction (here and here) when podcaster Joe Rogan asks about his AI doomsaying. Musk stares at the table, and takes a deep breath. He looks sad. Dejected. Fatalistic. [...]

Moreover, I believe Musk when he says that his ultimate purpose for founding Neuralink is to avert an AI catastrophe: “If you can’t beat it, join it.” Personally, I’m not optimistic that brain-computer interfaces can avert AI catastrophe — for roughly the reasons outlined in the BCIs section of Superintelligence ch. 2 — but Musk came to a different assessment, and I’m glad he’s trying.

Whatever my disagreements with Musk (I have plenty), it looks to me like Musk doesn’t just profess concern about AI existential risk. ² I think he feels it in his bones, when he wakes up in the morning, and he’s spending a significant fraction of his time and capital to try to do something about it. And for that I am grateful.

As a separate consideration, if you consider someone an authority, then you're going to (explicitly and implicitly) trust their assessments of the world to be at least somewhat accurate. So even if you didn't experience social pressure to be distressed yourself, just the fact that they were distressed due to something you considered them an authority on (e.g. Eliezer on AI risk) suggests you might pick up some of their distress.

I feel like consequentialists are more likely to go crazy due to not being grounded in deontological or virtue-ethical norms of proper behavior. It's easy to think that if you're on track to saving the world, you should be able to do whatever is necessary, however heinous, to achieve that goal. I didn't learn to stop seeing people as objects until I leaned away from consequentialism and toward the anarchist principle of unity of means and ends (which is probably related to the categorical imperative). E.g. I want to live in a world where people are respected as individuals, so I have to respect them as individuals - whereas maximizing individual-respect might lead me to do all sorts of weird things to people now in return for some vague notion of helping lots more future people.

I think drugs and non-standard lifestyle choices are a contributing factor. Messing with ones biology / ignoring the default lifestyle in your country to do something very non-standard is riskier and less likely to turn out well than many imagine. 

Everyone, read also the comments at EA website, the top one [EA(p) · GW(p)] makes a great point:

while EA/rationalism is not a cult, it contains enough ingredients of a cult that it’s relatively easy for someone to go off and make their own.

To avoid derailing the debate towards the definition of cult etc., let me paraphrase it as:

EA/rationalism is not an evil project, but it is relatively easy for someone to start an evil project by recruiting within the EA/rationalist ecosystem. (As opposed to starting an evil project somewhere else.)

This is how.

EA/rationalism in general [...] lacks enforced conformity and control by a leader. [...]

However, what seems to have happened is that multiple people have taken these base ingredients and just added in the conformity and charismatic leader parts. You put these ingredients in a small company or a group house, put an unethical or mentally unwell leader in charge, and you have everything you need for an abusive [...] environment. [...] This seems to have happened multiple times already. 

I didn't spend much time thinking about it, but I suspect that the lack of "conformity and control" in EA/rationalism may actually be a weakness, from this perspective. Whenever a bad actor starts doing something "in the name or EA/rationality", there is no established mechanism to make public knowledge "no, you are not; this is your private project". Especially when the bad actor does it at a LW meetup, or makes it publicly known that they donate money... then it would seem quite silly to argue that they are not "an EA/rationality project".

(Compare it to other communities, which have more control and conformity, such as religion, where members clearly understand the difference between "this is just a personal opinion of someone who happens to be a Catholic" and "this is the official teaching of the Catholic church".)

[note: I'm not particularly EA, beyond the motte of caring about others and wanting my activities to be effective. ]

I think this is basically correct.  EA tends to attract outliers who are susceptible to claims of aggrandizement - telling themselves and being told they're the heroes in the story.  It reinforces this with contrarian-ness, especially on dimensions with clever, math-sounding legible arguments behind them.  And then reinforces that "effective" is really about the biggest numbers you can plausibly multiple your wild guesses out to.  

Until recently, it was all circulating in a pile of free money driven by related insanity of crypto and tech investment, which seemed to have completely forgotten that zero interest rates were unlikely to continue forever, and actually producing stuff would eventually be important.

[ epistemic status for next section: provocative devil's advocate argument ]

The interesting question is "sure, it's crazy, but is it wrong?"  I suspect it is wrong - the multiplicative factors into the future are extremely tenuous.  But in the event that this level of commitment and intensity DOES cause alignment to be solved in time, it's arguable that all the insanity is worth it.  If your advice makes the efforts less individually harmful, but also a little bit less effective, it could be a net harm to the universe.

I'm focusing on the aspects specific to rationalism and effective altruism that could lead to people who nominally are part of the rationality community being crazy at a higher rate than one would expect. From your post, I got the following list:

• Isolationist
• fewer norms
• highly ambitious
• gender-ratios
• X-risk being scary
• encourages doing big things

I may be missing some but these are all the aspects that stood out to me. From my perspective, the #1 most important cause of the craziness that sometimes occurs in nominally rationalist communities is that rationalists reject tradition. This kind of falls under the fewer norms category but I don't think 'fewer norms' really captures it.

A lot of people will naturally do crazy things without the strict social rules and guidelines that humans have operated with for hundreds of years. The same rules that have been slowly eroding since 1900. And nominally rationalist communities are kind of at the forefront of eroding those social rules. Rationalists accept as normal ideas like polyamory, group homes (in adulthood as a long term situation), drug use, atheism, mysticism, brain-hacking, transgenderism, sadomasochism, and a whole slew of other historically, socially dis-favored ways to live.

Society previously protected people who are susceptible to manipulation. Society disfavors not only the manipulation and abuse but all unusual behaviors listed above that tended to go along with manipulation and abuse. Not because they necessarily had to go together, but because people who tended to do some really weird things also tended to be manipulative and abusive. Many rationalist communities take the position that in social situations, "as long as there's consent it's not bad." And this just doesn't account for actual human behavior.

People sometimes say that cult members tend to have conflicts that lead to them joining the cult. Recently I've been wondering if this is an underrated aspect of cultishness.

Let's take the LaSota crew as an example. As I understand, they were militant vegans.

And if I understand correctly, vegans are concerned about the dynamic where, in order to obtain animal flesh to eat, people usually hire people to raise lots of animals in tiny indoor spaces, sometimes cutting off body parts such as beaks if they are too irritable. Never letting them out to live freely. Taking their children away shortly after they give birth. Breeding them to grow rapidly but also often having genetic disorders that cause a lot of pain and disability. And basically just having them live like that until they get killed and butchered.

And from what I understand, society often tries to obscure this. People get uncomfortable and try to change the subject when you talk about it. People might make laws to make it hard to film and share what's going on. People come up with convoluted denials of animals having feelings. And so on.

(I am not super certain about the above two paragraphs because I haven't investigated it myself, just picked it up by osmosis and haven't seen any serious objections to this narrative.)

I don't know much of the LaSota crew's backstory (I think there were other conflicts than this too? would have to re-read the pages), but basically I wonder if they were vegans, noticed how society (including some rationalists?) conspire to cover up these things, and how Ziz was one of the main people taking it seriously.

You can also sort of take a Bayesian view of it. Like one way to apply Bayes is that you have a set of "experts" who you have varying levels of trust in, and then as they say things, you use Bayesian updates to reallocate trust from those who make bad predictions to those who make good predictions. If you have a conflict where one side is consistently dishonest, then this procedure can lead to rapidly transfering a ton of their trust away, to the other side of the conflict.

It’s not clear to me what “crazy” means in this post & how it relates to something like raising the sanity waterline [LW · GW]. A clearer idea of what you mean by crazy would, I think, dissolve the question [LW · GW].

I have different hypotheses / framings. I will offer them. If you wish to discuss any of them in more detail, please reach out to me via email or PM. Happy to converse. ! 

// 

Mythical/Archetypal take: 

There are large-scale, old, and powerful egregores fighting over the minds of individuals and collectives. They are not always very friendly to human interests or values. In some cases, they are downright evil. (I'd claim the Marxist egregore is a pretty destructive one.)

The damage done by these egregores is multigenerational. It didn't start with just THIS generation. Shit started before any of us was born. 

It's kind of like the Iliad. Petty, powerful gods fighting over some nonsense; humans get caught in the hurricane-sized effects; chaos ensues. Sometimes humans become willing to sacrifice their souls to these egregores in exchange for the promise of power, wealth, security, sex, etc. 

It's not just unusual people like Ziz who might do this. Pretty normal-seeming, happy-seeming people have sacrificed their souls to certain egregores (e.g. progressivism, humanism, etc) and become mouthpieces for the egregores' values and agendas. When you try to have conversations with these people, it's like they're not speaking from their true beliefs or actual internal experience. They've become living propaganda. 

The rationalist / EA community attracts plenty of 'egregore activity' because of their concentration of intelligent, resourceful, good-hearted people. They are valuable to control.  They're also near the center of the actual narrative, the fight for humanity's soul and evolutionary direction. They are particularly vulnerable to egregore activity because of high rates of trauma and disembodiment and strong ideological bent, making them relatively easy to manipulate. 

OK, but how does one properly defend against these huge forces, tossing humans around like rag dolls? 

The main trick is embodiment. Being totally in the body, basically all the time. An integrated person, integration between heart, mind, body, and soul. Resolving and overcoming any addictions to anything, including seemingly innocuous ones. Resolving and healing trauma as much as possible (which is an ongoing journey). Finding a deeper, more fundamental happiness and peace that can't be disturbed by any external circumstance (thus, no longer being subject to temptations for power, wealth, security, relationship, or sense pleasure). 

//

Main other story:

Some people made some really bad choices. 

Integrity isn't something that happens to you. You have to choose it and choose it again. If you fail to choose it, and then fail to recognize and reconcile the error... and keep failing to choose it... that path leads to more slippage and things can spiral out of control. 

Ziz made certain choices, and that had consequences. Ziz didn't reform. Ziz didn't apologize. Ziz kept digging that hole. That negative-spiral path erodes one's ability to see what's happening and can lead to deep insanity. 

A person's moral system does not thrive under a guilty conscience. It gets unbearable. And you just have to keep hurting more people to justify it to yourself, and to temporarily escape the pain. This is what happens when one isn't willing to feel remorse, grieve, and acknowledge the damage they've caused. It becomes hell on earth for you, and you create hell on earth for others. 

Whatever hypothesis you come up with, don't absolve the individuals of their moral responsibility to avoid evil actions. It gets this bad when people fuck up that badly. Not due merely to external circumstances or internal drives like 'wanting to belong'. Regardless of any of that, they made choices, and they didn't have to make those choices. 

//

I could probably find more hypotheses, but I will stop there! :0 Thanks for reading. 

fwiw I think stealing money from mostly-rich-people in order to donate it isn't obviously crazy. Decouple this claim from anything FTX did in particular, since I know next to nothing about the details of what happened there. From my perspective, it could be they were definite villains or super-ethical risk-takers (low prior).

Thought I'd say because I definitely feel reluctance to say so. I don't like this feeling, and it seems like good anti-bandwagon policy to say a thing when one feels even slight social pressure to shut up.

Probably be grounded in more than one social group. Even being part of two different high-intensity groups seems like it should reduce the dynamics here a lot.

Worked well for me!

Eric Chisholm likes to phrase this principle as "the secret to cults is to be in at least two of them".

1. Don’t put yourself into positions of insecurity. […]

This seems like it points in the wrong direction to me. I'd instead say something like "look for your own insecurities and then look closely the ones you find". But the current thing you've said sounds like "avoid wherever your insecurities might manifest (because they're fixed)".

[How to resolve insecurities? Coherence Therapy [LW · GW].]

I think there's a commonly-held belief that a feeling of belonging is something that we can get from other people, but I think this is a misconception. Stable confidence doesn't come from knowing that other people like you.

Note: It's not that we can get a feeling of belonging entirely from ourselves, either— it's more nuanced than that. Alfred Adler, father of Individual Psychology, said quite a bit about this, and I'm currently drafting a sequence that touches on this.

My honest opinion is that Ziz got several friends of mine killed. So i dont exactly have a high opinion of her. But I have never heard of Ziz referring to themselves as LaSota. Its honestly toxic not to use people's preferred names. Its especially toxic if they are trans but the issue isn't restricted to trans people. So Id strongly prefer people refer to Ziz as Ziz. 

Good breakdown of one of the aspects in all this. The insecurity/desperation topic is a really hard one to navigate well, but I agree it's really important.

Hard because when someone feels like an outsider, a group of other likeminded outsiders will naturally want to help them and welcome them, and it can be an uncomplicated good to do so. Important because if someone has only one source of to supply support, resources, social needs, etc, they are far more likely to turn desperate or do desperate things to maintain their place in the community.

Does this mean we should not accept anyone into the community just because they really want a safe place to avoid broader civilization? I don't think so, but it's definitely a flag more people should be aware of, including those who are desperate to belong. Exploitation can happen on a broad and public scale, like organizations looking for volunteers or employees, but it can also be small and private, at the level of group houses and friends made in the community.

Young people in particular who join the community are of course especially at risk here, and it's a constant struggle at the rationality camps to both welcome and provide opportunities for those who do want to join the broader community rather than just enjoy the camp for its own sake, but not foster dependency.

Hot take: to the extent that EAs and rationalists turn crazy, part of the problem involves that some of their focuses include existential risk + having very low discount rates for the future.

To explain more, I think that utilitarianism is maybe a part of the problem, but it's broader than that. The bigger problem is once you fundamentally believe that we will all die of something, and your group can control the chances of being extinct, that's a fast road to craziness, given that most of these existential risks probably wouldn't materialize anyway, and importantly gives you license to justify a lot of actions that you'd normally never consider, and this is especially dangerous once you add very low discount rates to the mix, so that preventing it is our foremost priority.

This tweet below makes a harsher version of my point, and while I probably would soften it, I also suspect that something like this is true, and I actually like it both for stating something that is maybe true, and that it makes rather weak ethics assumptions on what EAs/LWers value, and what their ethical system looks like.

https://twitter.com/AGI_HeavenHell/status/1673359793078038528

Seems like the forces that turn people crazy are the same ones that lead people to do anything good and interesting at all. At least for EA, a core function of orgs/elites/high status community members is to make the kind of signaling you describe highly correlated with actually doing good. Of course it seems impossible to make them correlate perfectly, and that’s why setting with super high social optimization pressure (like FTX) are gonna be bad regardless.

But (again for EA specifically) I suspect the forces you describe would actually be good to increase on the margin for people not living in Berkeley and/or in a group house which is probably a majority of self-identified EAs but a strong minority of the people-hours OP interacts with irl.

To any individual in a group, it can easily be the case that they think the group standard seems dumb, but in a situation of risk aversion, the important part is that you do things that look to everyone like the kind of thing that others would think is part of the standard. In practice this boils down to a very limited kind of reasoning where you do things that look vaguely associated with whatever you think the standard is, often without that standard being grounded in much of any robust internal logic. And doing things that are inconsistent with the actual standard upon substantial reflection do not actually get punished, as long as they look like the kind of behavior that looks like it was generated by someone trying to follow the standard.

Here is my own spin on this idea:

We often defer at least somewhat to common sense. Unfortunately, we face uncertainty both about the question at hand, and about what our friends and neighbors consider to be common sense. What's worse is that, even though each of us has an individual perspective, we often conflate our personal observations with our deferent, all-things-considered, common-sense best guess.

The result is that what settles in as "common sense" is often very different from the average of our individual observations - it's a woefully inaccurate narrative that caught on by chance. The reason it doesn't go away is that even this inaccurate narrative does reflect just enough real wisdom that we're still better off deferring to it than just going our own way. We comply with common sense - and our own decisions turn out better than those of our neighbors who don't heed it.

Yet that only entrenches the inaccurate narrative further. We'd all be better off getting together, being open and honest about our individual perspectives, and creating a new, more accurate form of common sense. Unfortunately, this is often very difficult to do. Inaccurate narratives about common sense perpetuate themselves, and the group suffers.

Most social groups will naturally implement an "in-group / out-group" identifier of some kind and associated mechanisms to apply this identifier on their members. There are a few dynamics at play here:

1. Before this identification mechanism has been implemented, there isn't really much of a distinction between in-group and out-group. Therefore, there will be people who self-identify as being associated with the group, but who are not part of the sub-group which begins to make the identifications. Some of these members may accordingly get labeled part of the out-group by the sub-group which identifies as the in-group. This creates discord.
2. The identification method works as a cut-off, which is ultimately arbitrary. Even if the metric used to implement this cut-off is relatively valid (such as an overall measure of aptitude), the cut-off itself is technically not. 
3. There is a natural incentive structure to implement this cut-off to boost one's social rank relative to those under the cut-off. This means that there is probably a pre-existing aptitude measure of some kind (or a visible social hierarchy, which might be more correlated to this measure). Thus, the cut-off may be even be flipped-sign from whatever it is portrayed as signaling. 

We'd expect that groups which implement these cut-off strategies to be more "cult-like" than ones that do not. Groups that implement these cut-offs usually have to invent beliefs and ideologies which support the practice of doing so. Usually, these ideologies are quite outward-projected, and typically tend to consist of negative reactions to the activities of other groups. 

They probably also, in line with point 2, actually use proxy metrics for implementing the cut-off, which work as binary features, e.g. (person X has a quality we don't like, even though they are extremely good at task Y). Therefore, they promote the ideology that people with specific, ostensibly unlikable attributes need to be excluded even if they have agreed-upon displayed skill, with a visible track record of being productive for the group.

All of the above can increase the chance of internal conflict.  

As a cryptocurrency creator, I can assure you that there is something seriously wrong with nearly everyone who owns and is a fan of cryptocurrency technologies. You can't trust those kinds of people. And most crypto-chlurmcks have not even read the Bitcoin whitepaper since they are incapable of understanding it despite how easy the Bitcoin whitepaper is to read. And nearly all crypto-chlurmcks cannot seem to understand that Bitcoin has a mining algorithm that was never even designed to advance science while simultaneously establishing consensus. And unlike GPT, the crypto-chlurmcks typically do not know how to form intelligible complete sentences. They hate mathematics. And they are all-around really dysfunctional people. The cryptocurrency sector has its fair share of scandals because the people who are attracted to cryptocurrencies are not the best people.

It's simple. "You" (the rationalist community) are selected for being bad at making wisdom saving throws, so to speak.

You know, let's look at Yudkowsky, with all of his very public, very obvious character dysfunction and go "yes, this is the father figure/Pope I need to change my life."

The only surprise here is the type of stuff you are agonizing about didn't happen earlier, and isn't happening more often.