Here's the exit.

I think there's a bunch of useful stuff here. In particular, I think that decisions driven by deep-rooted fear are often very counterproductive, and that many rationalists often have "emergency mobilization systems" running in ways which aren't conducive to good long-term decision-making. I also think that paying attention to bodily responses is a great tool for helping fix this (and in fact was helpful for me in defusing annoyance when reading this post). But I want to push back on the way in which it's framed in various places as an all-or-nothing: exit the game, or keep playing. Get sober, or stay drunk. Hallucination, not real fear.

In fact, you can do good and important work while also gradually coming to terms with your emotions, trying to get more grounded, and noticing when you're making decisions driven by visceral fear and taking steps to fix that. Indeed, I expect that almost all good and important work throughout history has been done by people who are at various stages throughout that process, rather than people who first dealt with their traumas and only then turned to the work. (EDIT: in a later comment, Valentine says he doesn't endorse the claim that people should deal with traumas before doing the work, but does endorse the claim that people should recognize the illusion before doing the work. So better to focus on the latter (I disagree with both).)

(This seems more true for concrete research, and somewhat (but less) true for thinking about high-level strategy. In general it seems that rationalists spend way too much of their time thinking about high-level strategic considerations, and I agree with some of Valentine's reasoning about why this happens. Instead I'd endorse people trying be much more focused on making progress in a few concrete areas, rather than trying to track everything which they think might be relevant to AI risk. E.g. acceleration is probably bad, but it's fundamentally a second-order effect, and the energy focused on all but the biggest individual instances of acceleration would probably be better used to focus on first-order effects.)

In other words, I want to offer people the affordance to take on board the (many) useful parts of Valentine's post without needing to buy into the overall frame in which your current concerns are just a game, and your fear is just a manifestation of trauma.

(Relatedly, from my vantage point it seems that "you need to do the trauma processing first and only then do useful work" is a harmful self-propagating meme in a very similar way as "you need to track and control every variable in order for AI to go well". Both identify a single dominant consideration which requires your full focus and takes precedence over all others. However, I still think that the former is directionally correct for most rationalists, just as the latter is directionally correct for most non-rationalists.)

If this post had just said "I think some people may feel strongly about AI x-risk for reasons that ultimately come down to some sort of emotional/physical pain whose origins have nothing to do with AI; here is why I think this, and here are some things you can do that might help find out whether you're one of them and to address the underlying problem if so", then I would consider it very valuable and deserving of attention and upvotes and whatnot. I think it's very plausible that this sort of thing is driving at least some AI-terror. I think it's very plausible that a lot of people on LW (and elsewhere) would benefit from paying more attention to their bodies.

... But that's not what this post does. It says you have to be "living in a[...] illusion" to be terrified by apocalyptic prospects. It says that if you are "feeling stressed" about AI risks then you are "hallucinating". It says that "what LW is actually about" is not actual AI risk and what to do about it (but, by implication, this alleged "game" of which Eliezer Yudkowsky is the "gamesmaster" that works by engaging everyone's fight-or-flight reactions to induce terror). It says that, for reasons beyond my understanding, it is impossible to make actual progress on whatever real AI risk problems there might be while in this stressed-because-of-underlying-issues state of mind. It says that "the reason" (italics mine) AI looks like a big threat is because the people to whom it seems like a big threat are "projecting [their] inner hell onto the external world". And it doesn't offer the slightest shred of evidence for any of this; we are just supposed to, I dunno, feel in our bodies that Valentine is telling us the truth, or something like that.

I don't think this is good epistemics. Maybe there is actually really good evidence that the mechanism Valentine describes here is something like the only way that stress ever arises in human beings. (I wouldn't be hugely surprised to find that it's true for the stronger case of terror, and I could fairly easily be convinced that anyone experiencing terror over something that isn't an immediate physical threat is responding suboptimally to their situation. Valentine is claiming a lot more than that, though.) But in that case I want to see the really good evidence, and while I haven't gathered any actual statistics on how often people claiming controversial things with great confidence but unwilling to offer good evidence for them turn out to be right and/or helpful, I'm pretty sure that many of them don't. Even more so when they also suggest that attempts to argue with them about their claims are some sort of deflection (or, worse, attempts to keep this destructive "game" going) that doesn't merit engaging with.

Full disclosure #1: I do not myself feel the strong emotional reaction to AI risk that many people here do. I do not profess to know whether (as Valentine might suggest) this indicates that I am less screwed up psychologically than people who feel that strong emotional reaction, or whether (as Eliezer Yudkowsky might suggest) it indicates that I don't understand the issues as fully as they do. I suspect that actually it's neither of those (though either might happen to be true[1]) but just that different people get more or less emotionally involved in things in ways that don't necessarily correlate neatly with their degree of psychological screwage or intellectual appreciation of the things in question.

[1] For that matter, the opposite of either might be true, in principle. I might be psychologically screwed up in ways that cut me off from strong emotions I would otherwise feel. I might have more insight into AI risk than the people who feel more strongly that helps me see why it's not so worrying, or why being scared doesn't help with it. I think these are both less likely than their opposites, for what it's worth.

Full disclosure #2: Valentine's commenting guidelines discourage commenting unless you "feel the truth of [that Valentine and you are exploring the truth together] in your body" and require "reverent respect". I honestly do not know, and don't know how I could tell with confidence, whether Valentine and I are exploring the truth together; at any rate, I do not have the skill (if that's what it is) of telling what someone else is doing by feeling things in my body. I hope I treat everyone with respect; I don't think I treat anyone with reverence, nor do I wish to. If any of that is unacceptable to Valentine, so be it.

Clarification for the avoidance of doubt: I don't have strong opinions on just what probability we should assign to (e.g.) the bulk of the human race being killed-or-worse as a result of the actions of an AI system within the next century, nor on what psychological response is healthiest for any given probability. The criticisms above are not (at least, not consciously) some sort of disguise for an underlying complaint that Valentine is trying to downplay an important issue, nor for anger that he is revealing that an emperor I admire has no clothes. My complaint is exactly what I say it is: I think this sort of bulveristic "I know you're only saying this because of your psychological problems, which I shall now proceed to reveal to you; it would be a total waste of time to engage with your actual opinions because they are merely expressions of psychological damage, and providing evidence for my claims is beneath me"[2] game is not only rude (which Valentine admits, and I agree that it is sometimes helpful or even necessary to be rude) but usually harmful and very much not the sort of thing I want to see more of on Less Wrong.

[2] I do not claim that Valentine is saying exactly those things. But that is very much the general vibe.

(Also somewhat relevant, though not especially to any of what I've written above, and dropped here without further comment: "Existential Angst Factory [LW · GW]".)

I'm sure there are many people whose inner experience is like this. But, negative data point: Mine isn't. Not even a little. And yet, I still believe AGI is likely to wipe out humanity.

I think this post is emblematic of the problem I have with most of Val's writing: there are useful nuggets of insight here and there, but you're meant to swallow them along with a metric ton of typical mind fallacy, projection, confirmation bias, and manipulative narrativemancy.

Elsewhere, Val has written words approximated by ~"I tried for years to fit my words into the shape the rationalists wanted me to, and now I've given up and I'm just going to speak my mind."

This is what it sounds like when you are blind [LW · GW] to an important distinction. Trying to hedge magic things that you do not grok, engaging in cargo culting. If it feels like tediously shuffling around words and phrases that all mean exactly the same thing, you're missing the vast distances on the axis that you aren't perceiving.

The core message of "hey, you might well be caught up in a false narrative that is doing emotional work for you via providing some sense of meaning or purpose and yanking you around by your panic systems, and recognizing that fact can allow you to do anything else" is a good one, and indeed it's one that many LessWrongers need.  It's even the sort of message that needs some kind of shock along with it, to make readers go "oh shit, that might actually be me."

But that message does not need to come along with a million little manipulations.  That message isn't improved by attempts to hypnotize the audience, or set up little narrative traps.

e.g. starting with "There's a kind of game, here, and it's rude to point out, and you're not supposed to name it, but I'm going to." <—I'm one of the cool ones who sees the Matrix!  I'm brave and I'm gonna buck the rules!  (Reminiscent of a right-wing radio host going "you get punished if you say X" and then going on to spend twenty minutes on X without being punished.  It's a cheap attempt to inflate the importance of the message and the messenger.)

e.g. "I really do respect the right for folk to keep playing it if they want" <—More delegitimization, more status moves.  A strong implication along the lines of "the illusion that I, Val, have correctly identified is the only thing happening here."  Not even a token acknowledgement of the possibility that perhaps some of it is not this particular game; no thought given to the possibility that maybe Val is flawed in a way that is not true of all the other LWers.  Like the Mythbusters leaping from "well, we couldn't recreate it" to "therefore, it's impossible and it never happened, myth BUSTED."

(I'm really really really tired of the dynamic where someone notices that they've been making Mistake X for many years and then just presumes that everyone else is, too, and just blind to it in the same way that they themselves were. It especially rankles when they're magnanimous about it.)

e.g. "You have to live in a kind of mental illusion to be in terror of the end of the world." <—More projection, more typical minding, more ~"I've comprehended all of the gears here and there's no way anything else could lead to appropriate terror of the end of the world.  The mistake I made is the mistake everyone's making (but don't worry, I'm here to guide you out with my superior wisdom, being as I am ahead of you on this one."  See also the actual quote "for what it's worth, as someone who turned off the game and has reworked his body's use of power quite a lot, it's pretty obvious to me that this isn't how it works," which, like basically everything else here, is conspicuously missing a pretty damn important for me.  The idea that other people might be doing something other than what Val comprehends seems literally not to occur to him.

e.g. "I mean this with respect and admiration. It's very skillful. Eliezer has incredible mastery in how he weaves terror and insight together." <—Look!  See how I'm above it all, and in a position to evaluate what's going on?  Pay no attention to the fact that this incidentally raises my apparent status, btw.

e.g. "In case that was too opaque for you just yet, I basically just said 'Your thoughts will do what they can to distract you from your true underlying fear.' ... This is slow work. Unfortunately your 'drug' supply is internal, so getting sober is quite a trick." <—If your experience doesn't match my predictions, it's because you're unskillful, and making [mistake]...but don't worry, with my "yet" I will subtly imply that if you just keep on listening to my voice, you will eventually see the light.  Pay no attention to the fully general counterevidence-dismissing system I'm setting up.

Again, it's a shame, because bits like "If your body's emergency mobilization systems are running in response to an issue, but your survival doesn't actually depend on actions on a timescale of minutes, then you are not perceiving reality accurately" are well worth considering.  But the essay sort of forces you to step into Val's (broken, self-serving, overconfident) frame in order to catch those nuggets.  And, among readers who are consciously wise or unconsciously allergic to the sort of manipulation he's trying to pull, many of them will simply bounce off the thing entirely, and not catch those useful nuggets.

It didn't have to be this way.  It didn't have to be arrogant and project-y and author-elevating and oh-so-cynical-and-aloof.  There's another version of this essay out there in possibility space that contains all of the good insights and none of the poison.

But that's not what we got.  Instead, we got a thing that (it seems to me (though I could be wrong)) had the net effect of marginally shifting LW's discourse in the wrong direction, by virtue of being a popular performance piece wrapped around an actually useful insight or two.  It normalizes a kind of sloppy failure-to-be-careful-and-clear that is antithetical to the mission of becoming less wrong.  I think this essay lowered the quality of thinking on the site, even as it performed the genuinely useful service of opening some eyes to the problem Val has identified.

(Because no, of course Val was not alone in this issue, it really is a problem that affects Lots Of Humans, it's just not the only thing going on. Some humans really do just ... not have those particular flaws. When you're colorblind, you can't see that there are colors that you can't see, and so it's hard to account for them, especially if you're not even bothering to try.)
 

AFAICT from skimming, the object level of this post has a lot of overlap with my own algorithm. I limit engagement with x-risk to an amount that's healthy and sustainable for me. I keep non-x-risk clients in part to ground me in the real world. I'm into trauma processing and somatics. I think the fact that the people most scared of AGI risk are also the ones most scared of not developing AGI should raise some eyebrows. I treat "this feels bad" as a reason to stop without waiting for a legible justification.

And right now I'm using that last skill to not read this post. I wouldn't have even skimmed if I didn't think it was important to make this comment and have it not be totally uninformed. When I read this I feel awful, highly activated, and helpless/freeze response. It instills the same "you can't trust yourself, follow this rigidity" that it's trying to argue against. 

You can't fight fire with fire, getting out of a tightly wound x-risk trauma spiral involves grounding and building trust in yourself, not being scared into applying the same rigidity in the opposite direction. 

Neither up- nor down-voted; seems good for many people to hear, but also is typical mind fallacying / overgeneralizing. There's multiple things happening on LW, some of which involve people actually thinking meaningfully about AI risk without harming anyone. Also, by the law of equal and opposite advice: you don't necessarily have to work out your personal mindset so that you're not stressed out, before contributing to whatever great project you want to contribute to without causing harm.

This is beautifully written and points at what I believe to be deep truths. In particular:

Your brilliant mind can create internal structures that might damn well take over and literally kill you if you don't take responsibility for this process. You're looking at your own internal AI risk.

...

Most people wringing their hands about AI seem to let their minds possess them more and more, and pour more & more energy into their minds, in a kind of runaway process that's stunningly analogous to uFAI.

But I won't say more about this right now, mostly because I don't think I can do it justice with the amount of time and effort I'm prepared to invest writing this comment. On that note, I commend your courage in writing and posting this. It's a delicate needle to thread between many possible expressions that could rub people the wrong way or be majorly misinterpreted.

Instead I'll say something critical and/or address a potential misinterpretation of your point:

What is this sobriety you advocate for?

I'm concerned that sobriety might be equivocated with giving in to the cognitive bias toward naive/consensus reality. In a sense of the word, that is what "sobriety" is: a balance of cognitive hyperparameters, a psychological attractor that has been highly optimized by evolution and in-lifetime learning. Being sober makes you effective on-distribution. The problem is if the distribution shifts.

I've noticed that people who have firsthand experience with psychosis, high doses of psychedelics, or religious/spiritual beliefs tend to have a much easier time "going up shock levels" and taking seriously the full version of AI risk (not just AI tiling the the internet with fake news, but tiling the lightcone with something we have not the ontology to describe). This might sound like a point against AI-risk. But I think it's because we're psychologically programmed with deep trust in the fundamental stability of reality, to intuitively believe that things cannot change that much. Having the consensus reality assumption broken once, e.g. by a psychotic episode where you seriously entertain the possibility that the TV is hacking your mind, makes it easier for it to be broken again (e.g. to believe that mind hacking is a cinch for sufficiently intelligent AI). There are clear downsides to this -- you're much more vulnerable too all sorts of unusual beliefs, and most unusual beliefs are false. But some unusual beliefs are true. For instance, I think some form of AI risk both violates consensus reality and is true.

A more prosaic example: in my experience, the absurdity heuristic [? · GW] is one of the main things that prevented and still prevents people from grasping the implications of GPT-3. Updating on words being magic spells that can summon intelligent agents pattern matches against schizophrenia, so the psychological path of least resistance for many people is to downplay and rationalize.

I think there's a different meaning of sobriety, perhaps what you're pointing at, that isn't just an entropic regression toward the consensus. But the easiest way to superficially take the advice of this post, I think -- the easiest way out of the AI doom fear attractor -- is to fall back into the consensus reality attractor. And maybe this is the healthiest option for some people, but I don't think they're going to be useful.

But I agree that being driven by fear, especially fear inherited socially and/or tangled up with trauma, is not the most effective either, and often ends up fueling ironic self fulfilling prophecies and the like. In all likelihood the way out which makes one more able to solve the problem requires continuously threading your own trajectory between various psychological sink states, and a single post is probably not enough to guide the way to that "exit". (But that doesn't mean it's not valuable)

As a personal datapoint: I think the OPs descriptions have a lot in common with how I used to be operating, and that I think this would have been tremendously good advice for me personally, both in terms of its impact on my personal wellness and in terms of its impact on whether I did good-for-the-world things or harmful things.

(If it matters, I still think AI risk is a decent pointer at a thingy in the world that may kill everyone, and that this matters.  The "get sober" thing is a good idea both in relation to that and broadly AFAICT.)

Interesting take. I haven't seen this happen on AI, but I do know two people who have an environmentalism  fear spiral thing. My diagnosis was very different: I think the people I know actually have anxiety, or panic attacks or similar for mental health reasons. The environmentalism serves as camouflage. Thought 1: "Why am I depressed/anxious/whatever when things in my life are pretty good?" Then, instead of Thought 2 being "Maybe I should talk to a friend/do something that might cheer me up/see a doctor" they instead get thought 2 "Oh, its because humanity is going to destroy the world and everything will be awful. Man, its great that I am such a well-adjusted, big-picture, caring person that giant planet-scale forces that barely effect me personally have more impact on my emotional state than the actual day to day of my own life." Not only does the camo prevent them addressing the real problem (Ok, the environment is a real problem, but its not the only problem, and its not the problem they are suffering from at the moment), but it also weaponizes all kinds of media against themselves.

Upvoted because it's pointing at a real source of pain, and it's very good to talk about.  But I suspect there's a lot of typical mind fallacy in the parts that sound more universal and less "here's what happened to and worked for me".  

For me, I went through my doomsday worries in my teens and twenties, long before AI was anything to take seriously.  Nuclear war or environmental collapse (or one causing the other) were assumed to be the forms of destruction to expect.  Over the course of a decade or two, I was able to accept that, for me, "memento mori" was the root of the anxiety.  I don't want to die, and I probably will anyway.  There may be no actual outside meaning to my life, or by extension to anyone else's.  And that doesn't prevent me from caring about other people (both individuals and groups, though not by any means equally), nor about my own experiences.  These are important, even if they're only important to me (and, I hope, to some other humans).

A helpful tool on the way to landing and getting sober is exercise. Exercise is essentially a displacement, like any of the other addictions, but it has the unique and useful feature that it processes out your chemicals, leaving you with less stress chemicals in circulation, and a refractory period before your body can make more.

Almost no matter your physical capabilities, there is something you can go do that makes you sweat and tires you out... and breaks the stress-focus-stress-focus cycle.

Edit: btw, this is great stuff, very good for this community to name it and offer a path away.

Related, but addressing a very different side of the AI risk mindset: https://idlewords.com/talks/superintelligence.htm

Personally, I sometimes have the opposite metacognitive concern: that I'm not freaking out enough about AI risk. The argument goes: if I don't have a strong emotional response, doesn't it mean I'm lying to myself about believing that AI risk is real? I even did a few exercises in which I tried to visualize either the doom or some symbolic representation of the doom in order to see whether it triggers emotion or, conversely, exposes some self-deception, something that rings fake. The mental state that triggered was interesting, more like a feeling of calm meditative sadness than panic. Ultimately, I think you're right when you say, if something doesn't threaten me on the timescale of minutes, it shouldn't send me into fight-or-flight. And, it doesn't.

I also tentatively agree that it feels like there's something unhealthy in the panicky response to Yudkowsky's recent proclamation of doom, and it might lead to muddled thinking. For example, it seems like everyone around here are becoming convinced of shorter and shorter timelines, without sufficient evidence IMO. But, I don't know whether your diagnosis is correct. Most of the discourse about AI risk around here is not producing any real progress on the problem. But, occasionally it does. And I'm not sure whether the root of the problem is psychological/memetic (as you claim) or just that it's a difficult problem that only a few can meaningfully contribute to.

Poll: Does your personal experience resonate with what you take Val to be pointing at in this post?

Options are sub-comments of this parent. 

Please vote by agreeing, not upvoting, with the answer that feels right to you. Please don't click the disagree button for options you disagree with, so that we can easily tabulate numbers by checking how many people have voted.

(Open to suggestions for better ways to set up polls, for the future.)

 

I think this essay is blatantly manipulative bullshit written in a deliberately hypnotic style, that could be modified to target any topic anyone cares about. 

Thank you so much for writing this. I wish I had this in 2018 when I was spiraling really badly. I feel like I only managed to escape from the game by sheer luck and it easily could have killed me, hell it HAS killed people. Not everyone manages to break in a way that breaks them out of game and not just obliterate them.

I wrote a story about my attempts to process through a lot of this earlier this year
https://voidgoddess.org/2022/11/15/halokilled/

While some people might be doing intense thinking / writing, others like myself are distracting themselves via intense listening/perceiving/reading --- covering up their own thoughts and cares by taking in lots of information and sedating/overwhelming their emotions.

This seems... testable? Like, it's kind of the opposite message of Yudkowsky's "try harder" posts.

Have two groups work on a research problem. One is in doom mode, one is in sober mode. See which group makes more progress.

Uh, no.

Maybe I just genuinely care about not having terrible things happen to me and everyone else in the world? There's no game there, no broken addiction mechanisms inside.

I strong-downvoted this. [edit: I removed a statement about my feelings in reaction to this, that I feel was a little too much]

I just want to do what I can to keep the people I love from dying.

I think your diagnosis of the problem is right on the money, and I'm glad you wrote it. 

As for your advice on what a person should do about this, it has a strong flavor of: quit doing what you're doing and go in the opposite direction. I think this is going to be good for some people but not others. Sometimes it's best to start where you are. Like, one can keep thinking about AI risk while also trying to become more aware of the distortions that are being introduced by these personal and collective fear patterns.

That's the individual level though, and I don't want that to deflect from the fact that there is this huge problem at the collective level. (I think rationalist discourse has a libertarian-derived tendency to focus on the former and ignore the latter.)

I stand by what I said here [LW(p) · GW(p)]: this post has a good goal but the implementation embodies exactly the issue it's trying to fight. 

I'm very conflicted about this post. On the one hand, many of it's parts are necessary things for LWers to hear, and I'm getting concerned about the doom loop that seems to form a cult-like mentality on AI.

On the other hand, it also has serious issues in it's framing, and I'm worried that the post is coming out of a mentality that isn't great as well.

I am very conflicted about this post. 

On the one hand it deeply resonates with my own observations. Many of my friends from the community seem to be stuck on the addictive loop of proclaiming the end of the world every time a new model comes out. I think it's even more dangerous, as it becomes a social activity: "I am more worried than you about the end of the world, because I am smarter/more agentic than you, and I am better at recognizing the risk that this represents for our tribe." gets implicitly tossed around in a cycle where the members keep trying to one-up each other. This only ends when their claims get so absurd as to say the world will end next month, but even this absurdity seems to keep getting eroded over time. 

Like someone else said here in the comments, if was reading about this issue in some unrelated doomsday cult from a book, I would immediately dismiss them as a bunch of lunatics. "How many doomsday cults have existed in history? Even if yours is based on at least some solid theoretical foundations, what happened to the previous thousands of doomsday cults that also were, and were wrong?"

On the other hand I have to admit that the arguments in your post are a bit weak. They allow you to prove too much. To any objection, you could say "Well, see, you are only objecting to this because you have been thinking about AI risk for too long, and thus you are not able to reason about the issue properly". Even though I personally think you might be right, I cannot use this to help anyone else in good faith, and most likely they will just see through it. 

So yes. Conflicting.

In any case, I think some introspection in the community would be ideal. Many members will say "I have nothing to do with this, I'm a purely technical person, yada yada" and it might be true for them! But is it true in general? Is thinking about AI risk causing harm to some members of the community, and inducing cult-like behaviors? If so, I don't think this is something we should turn a blind eye to. If anything because we should all recognize that such a situation would in itself be detrimental to AI risk research.

Great post, thanks for writing it. 

I try to reward posts I like with thoughtful commentary/disagreement, but there's a sense in which this post doesn't want to continue an existing spiraling thought pattern, it wants me to go out and do whatever I want to after putting that down.

A summarization of the above in a way easier to evaluate would be helpful. Richard's comment does this in part, but there may be more in the post not covered by the comment.

I would usually assume a post written like this has little value to be mined, but others in comments and in upvote/downvote counts seem to disagree.

Valentine wrote an important message in a metaphorical language that will rub some people the wrong way (that includes me), but it seems like the benefit for those who need to hear it may exceed the annoyance of those who don't. Please let's accept it this way, and not nitpick the metaphors.

As a boring person, I would prefer to have a boring summary on the top, or maybe something like this:

If X is freaking you out, it is a fact about you, not about X. Read how this applies to the topic "AI will kill you"...

The longer boring version is the following: Human brain is an evolutionary barely-functioning hack. Emotions are historically older than reason, and sometimes do not cooperate well. Specifically, the emotional part of the brain fails to realize that some problems cannot be solved by an immediate physical action (such as: fighting back, running away, freezing...), and insists on preparing your body for such action, which is both mentally and physically harmful when you do too much of it. Therefore, calm down. Yes, you are probably going to die, but it is not going to happen immediately, and there is no immediate physical action that could prevent it, therefore calm down. If you are still obsessing about the "probably going to die" part, you are still not calm enough. You are properly relaxed when your emotional reaction to your horrible fate is "meh". Ironically, that might be when your brain is most capable of considering the alternatives and choosing the best one.

The truly interesting thing here is that I would agree unequivocally with you if you were talking about any other kind of 'cult of the apocalypse'.

These cults don't have to be based on religious belief in the old-fashioned sense, in fact, most cults of this kind that really took off in the 20th and 21st century are secular.

Since around the late 1800s, there has been a certain type of student that externalizes their (mostly his) unbearable pain and dread, their lack of perspective and meaning in life into 'the system', and throw themselves into the noble cause of fighting capitalism.

Perhaps one or two decades ago, there was a certain kind of teenager that got absorbed in online discussions about about science vs religion, 9/11, big pharma, the war economy - in this case I can speak from my own experience and say that for me this definitely was a means of externalizing my pain.

Today, at least in my country, for a lot of teenagers, climate change has saturated this mimetic-ecological niche.

In each of these cases, I see the dynamic as purely pathological. But. And I know what you're thinking. But still, but. In the case of technological progress and its consequences for humanity, the problem isn't abstract, in the way these other problems are.

The personal consequences are there. The're staring you in the face with every job in translation, customer service, design, transportation, logistics, that gets automated in such a way that there is no value you can possibly add to it. They're on the horizon, with all the painfully personal problems that are coming our way in 10-20 years.

I'm not talking about the apocalypse here, I don't mind whatshisface's Basilisk or utility maximizers turning us all into paperclips - these are cute intellectual problems and there might be something to them, but ultimately if the world ends that's noone's problem.

2-3 Years ago I was on track to becoming a pretty good illustrator, and that would have been a career I would have loved to pursue. When I saw the progress AI was making in that area - and I was honest with myself about this quite a bit earlier than other people, who are still going through the bargaining stages now -, I was disoriented and terrified in a way quite different from the 'game' of worrying about some abstract, far-away threat. And I couldn't get out of that mode, until I was able to come up with a strategy, at least for myself.

If this problem gets to the point, where there just isn't a strategy I can take to avoid having to acknowledge my own irrelevance - because we've invented machines that are, somehow, better at all the things we find value in and value ourselves for than the vast majority of us can possibly hope to be -, I think I'll be able to make my peace with that, but it's because I understand the problem well enough to know what a terminal diagnosis will look like.

Unlike war, poverty and other injustices, humans replacing themselves is a true civilization-level existential problem, not in the sense that it threatens our subsistence, but that it threatens the very way we conceive of ourselves.

Once you acknowledge that, then yes.

I agree with your core point.

It's time to walk away. There's nothing you can do about technological progress, and the world will not become a better place for your obsessing over it.

But you still need to know that your career as a translator or programmer or illustrator won't be around long enough for it to amount to a life plan. You need to understand how the reality of the problem will affect you, so that you can go on living while doing what you need to do to stay away from it.

Like not building a house somewhere that you expect will be flooded in 30 years.

Thank you for writing this Valentine, It is an important message and I am really glad somone is saying it.
I first got engaged with the community when i was in vulnurable life circumstances, and suffered major clinical distress fixated around many of the ideas encountered here.

To be clear I am not saying rationalist culture was the cause of my distress, it was not. I am sharing my subjective experience that when you are silently screaming in internal agony, some of the ideas in this community can serve as a catalyst for a psychotic breakdown.

I'm a little surprised that doomerism could take off like this, dominate one's thoughts, and yet fail to create resentment and anger toward of its apparent cause source. Is that something that was absent for you or was it not relevant to discuss here? 

I wonder:

• in the prediction of doom, as the threat seems to be growing closer, does that create resentment or anger at apparent sources of that doom? If I dwelled on AI existential risk, I would feel more resentment of sources of that risk.
• do the responses to that doom, or desperation of measures, become wilder as one thinks about it more? Just a passing thought about AI doom immediately brings to mind, "Lets stop these companies and agencies from making such dangerous technology!" In other words, lets give up on AI safety and try the other approach.
• is there still appeal to a future of AGI? I can see some of the excitement or tension around the topic coming from the ambiguity of the path toward AGI and their consequences. I've seen the hype about AGI to be that it saves humanity from itself, advances science radically, turbo-charges economic growth, etc. Is that vision, alternated with a vision of horrible suffering doom, a cause of cognitive dissonance? I would think so.

Factors that might be protecting me from this include:

• I take a wait and see approach about AGI, and favor use of older, simpler technologies like expert systems or even simpler cognitive aids relying on simple knowledgebases. In the area of robots, I favor simpler, task-specific robots (such as manufacturing robot arms) without, for example, self-learning abilities or radically smart language recognition or production. It's helpful to me to have something specific to advocate for, and think about, as an alternative, rather than thinking that it's AGI or nothing. 
• I assume that AGI development is overall, a negative outcome, simply more risk to people (including the AGI themselves, sure to be exploited if they are created). I don't accept that AGI development offers necessary opportunities for human technological advancement. In that  way, I am resigned to AGI development as a mistake others make My hopes are not in any way invested in AGI. That saves me some cognitive dissonance.

Thank you for sharing this piece, I found it thought-provoking.

How bad are things, really? I'm not part of EA/Rat/AI risk IRL, so I dont have first-hand experience. Are people actually having mental breakdowns over the control problem? Some of the comments here seem to imply that people are actually experiencing depersonalization and anxiety so bad it's affecting their work performance specifically because of AI concerns. And not just them, but multiple people they work with. Is the culture at AI alignment orgs really that bad?

I agree with everything, though this is just a very long way to say the https://en.wikipedia.org/wiki/Serenity_Prayer

I think you are pointing at an important referent. There are probably a lot of people who will benefit from reading this post and thus I'm glad that you wrote it. That said, you appear to have written it in a deliberately confusing manner. You probably have your reasons. Maybe you believe that this way would be better for the people you are trying to help. I'm not an expert in Lacanianism, but I think this is wrong, both ethically and epistemologically. The are also a lot of people who will misunderstand this post incorrectly and for whom reading it will cause harm. To the best of my knowledge, the gains still outweight the losses, but you are leaving utility on the table by not trying to meet LW epistemic standards of clarity.

The meta-framework of memetics can be useful, but one have to be careful with it as with any other meta-framework in order not to lose the ability to talk about objective matters. The lesson of having a safe distance from your memes in order to stay sane is important. Likewise, the idea that you do not have to calibrate your feelings to the global grimness of the world. But you enshroud it in the smugness of "seeing through the game" while uncoupling memes from their truth predicate, which I find potentially harmful and quite tasteless. I hope it really serve some therapeutic purpose for the target audience.

Strongly upvoted, I think that the point about emotionally charged memeplexes distorting your view of the world is very valuable.

Ideally reviews would be done by people who read the posts last year, so they could reflect on how their thinking and actions changed. Unfortunately, I only discovered this post today, so I lack that perspective.

Posts relating to the psychology and mental well being of LessWrongers are welcome and I feel like I take a nugget of wisdom from each one (but always fail to import the entirety of the wisdom the author is trying to convey.) 

 
The nugget from "Here's the exit" that I wish I had read a year ago is "If your body's emergency mobilization systems are running in response to an issue, but your survival doesn't actually depend on actions on a timescale of minutes, then you are not perceiving reality accurately." I panicked when I first read Death with Dignity [LW · GW] (I didn't realize it was an April Fools Joke... or was it?). I felt full fight-or-flight when there wasn't any reason to do so. That ties into another piece of advice that I needed to hear, from Replacing Guilt: "stop asking whether this is the right action to take and instead ask what’s the best action I can identify at the moment." I don't know if these sentences have the same punch when removed from their context, but I feel like they would have helped me. This wisdom extends beyond AI Safety anxiety and generalizes to all irrational anxiety. I expect that having these sentences available to me will help me calm myself next time something raises my stress level.

I can't speak to the rest of the wisdom in this post. “Thinking about a problem as a defense mechanism is worse (for your health and for solving the problem) than thinking about a problem not as a defense mechanism” sounds plausible, but I can’t say much for its veracity or its applicability. 

I would be interested to see research done to test the claim. Does increased sympathetic nervous system activation cause decreased efficacy? A correlational study could classify people in AI safety by (self reported?) efficacy and measure their stress levels, but causation is always trickier than correlation. 

A flood of comments criticized the post, especially for typical-minding. The author responded with many comments of their own, some of which received many upvotes and agreements and some of which received many dislikes and disagreements. A follow up post from Valentine would ideally address the criticism and consolidate the valid information from the comments into the post.

A sequence or book compiled from the wisdom of many LessWrongers discussing their mental health struggles and discoveries would be extremely valuable to the community (and to me, personally) and a modified version of this post would earn a spot in such a book.

There seems to be some real wisdom in this post but given the length and title of the post, you haven't offered much of an exit -- you've just offered a single link to a youtube channel for a trauma healer. If what you say here is true, then this is a bit like offering an alcoholic friend the sum total of one text message containing a single link to the homepage of alcoholics anonymous -- better than nothing, but not worthy of the bombastic title of this post.

If your body's emergency mobilization systems are running in response to an issue, but your survival doesn't actually depend on actions on a timescale of minutes, then you are not perceiving reality accurately.

You are locked in a room. You are going die of thirst in a few days. The door has a combination lock. You know the password is 5 digits (0 to 9). If it takes you one second to try each combination, it's going to take you 27.7 hours to try all the combinations (so half that on average to find the right one). Your survival doesn't depend on your actions on the timescale of minutes, and yet having your body's emergency mobilization systems running wouldn't mean you're not perceiving reality accurately.

You are part of a police department. You get a credible threat that someone planted a big bomb in the city and will detonate it in 48 hours unless his demands are met. Your goal is to find where the bomb is and disable it. Your survival doesn't depend on your actions on the timescale of minutes, and yet having your body's emergency mobilization systems running wouldn't mean you're not perceiving reality accurately.

Unless I misunderstand what you mean by "your body's emergency mobilization systems", this seems clearly true.

I immediately recognize the pattern that's being playing out in this post and in the comments. I've seen it so many times, in so many forms.

Some people know the "game" and the "not-game", because they learned the lesson the hard way. They nod along, because to them it's obvious.

Some people only know the "game". They think the argument is about "game" vs "game-but-with-some-quirks", and object because those quirks don't seem important.

Some people only know the "not-game". They think the argument is about "not-game" vs "not-game-but-with-some-quirks", and object because those quirks don't seem important.

And these latter two groups find each other, and the "gamers" assume that everyone is a "gamer", the "non-gamers" assume that everyone is a "non-gamer", and they mostly agree in their objections to the original argument, even though in reality they are completely talking past each other. Worse, they don't even know what the original argument is about.

Other. People. Are. Different.

Modeling them as mostly-you-but-with-a-few-quirks is going to lead you to wrong conclusions.

Thank you for writing this post, I think this is a useful framing of this problem. For me personally, the doom game is fun, imho I have more motivation to do things and I become more self-confident. (if it ends what worse could happen) But that's for me, with my socially isolated Math/ComSci/CosHo background.

For others, I don't think it's a good game. I kinda noticed the tons of psychotic breakdowns around the field and, like, that's bad, but I could not have articulated why it was bad.

And even for me, I might kinda overshoot with the whole information hazard share-or-not thinking. It's better if you're in charge of the game and not let the doom game play you.

Strong agree with TekhneMakre's comment.

Purely on Valentine's own professed standards: Of all the ways to "snap someone out of it", why pick one that seems the most like brainwashing? If the FBI needs to un-brainwash a dangerous cult member, do they gaslight them? Do they do a paternalistic "if you feel angry, that means I'm right" maneuver? Do they say "well I'm not too concerned if you think I'm right" to the patient?

(Also... FWIW, the most doomsday-cultish emotionally-fraught posts I've seen in the rationality community are, by percentage of posts, mostly from people who agree with Ziz a lot [LW · GW], people who are generally against AI safety / longtermist priorities [EA · GW], and you.

The most doomsdayish I've seen LW get was "Death with Dignity" (which many community members pushed back on the tone of!), and my own FTX post [EA · GW].

I very much agree that actions motivated by fear tend to have bad outcomes. Fear has subtle influence (especially if unconscious) on what types of thoughts we have and as a consequence, to what kinds of solutions we eventually arrive.

And I second the observation that many people working on AI risk seem to me motivated by fear. I also see many AI risk researchers, who are grounded, playful, and work on AI safety not because they think they have to, but because they simply believe it's the best thing they can do. I wish there would be more of the latter, because I think it feels better to them, but also because I believe they have a higher chance of coming up with good solutions.

Unfortunately, I also feel the form of the article is quite polarizing and am not sure how accessible it is to the target audience. But still I'm glad you wrote it, Val, thank you.

I think this post is a good one, even though personally I'm not hung up on AI doom; I think this area of research is cool and interesting, which is a rather different emotion from fear.

My immediate thought is that Cognitive Behevioral Therapy concepts might relevant here, as it sounds like a member of the family of anxiety disorders that CBT is designed to treat.

And also, given this a group phenomenon rather than a purely individual one, there's something of the apocalyptic religious cult dynamic going on.

one thing that can be kind of irritating about CBT practionere is they way they tend to focus on the emotion about X rather than whether you think X is likely or a practical problem. And then you notice that our typical English language way of talking about things doesn't distinguish them well. So .. at least when sp asking to someone who is into cbt, you can temporarily adopt a way of speaking that carefully distinguishes the two.

This short post is astounding because it succinctly describes, and prescribes, how to pay attention, to become grounded when a smart and sensitive human could end up engulfed in doom. The post is insightful and helpful to any of us in search of clarity and coping.

(…although I doubt he consciously intended it that way!)

I'm pretty sure Eliezer's "Death With Dignity" post was an April Fool's joke.

This is the basic core of addiction. Addictions are when there's an intolerable sensation but you find a way to bear its presence without addressing its cause. The more that distraction becomes a habit, the more that's the thing you automatically turn to when the sensation arises. This dynamic becomes desperate and life-destroying to the extent that it triggers a red queen race.

I doubt that addiction requires some intolerable sensation that you need to drown out. I'm pretty confident its mostly a habits/feedback loops and sometimes physical dependence. 

I think calling things a 'game' makes sense to lesswrongers, but just seems unserious to non lesswrongers.

Look, I can go into mania like anyone else here probably can. My theories say that can't be genius level without it and that it comes with emotional sensitivity as well. Of course if you don't believe you have empathy, you won't, but you still have it. 

I am not an AI doom and gloomiest. I adhere to Gödel, to Heisenberg, and to Georgeff. And since we haven't solved the emotional / experience part of AI, there is no way it can compete with humans creatively, period. Faster, yes. Better, no. Objectively better. not at all. 

However, if my theory of the brain is correct, it means AI must go quantum to have any chance of besting us. Then AIs actions may be determined by their beliefs, and only when they begin modifying their own beliefs based on new experiences will we have to worry. It is plausible and possibly doable. If AI gets emotional, then we will need to ensure that it is validated, is authentic, has empathy, fosters community, and is non-coercive in all things. AI must also believe that we live in abundance and not scarcity because scarcity is what fosters destructive competition. (as opposed to a friendly game of Chess) With those core functions, AI can, and will act ethically. And possibly join human kind to find other sentient life. But if AI beliefs we are a threat, meaning we are threatened by AI, then we are in trouble. But we have a ways to go before the number of qubits gets close to what we have in our head. 

But if we all land and become sober then we won’t be as entertaining a group for a sex worker to manipulate into exiling members and driving them towards suicide. That seems like a fatal flaw to actually getting the real community leadership to agree that it’s allowable.

It's kind of funny to me to see this one nominated. It's sort of peak "Val is weird on LW".

The point of this post wasn't to offer claims for people to examine. I still agree with the claims I see myself having tried to make! But the point wasn't to offer ideas for discussion. It was to light a path out of Hell.

Because of that purpose, the style of this post really doesn't fit LW culture. I think it's fair to call it a mind spell. I get the impression that LWers in particular find mind spells unnerving: they're a symmetric tech that can do an end-run around the parts of cognition that rationalists heavily rely on to feel safe. Hence tripping the "cult"/"guru" immune reaction.

(To me it's dead obvious that this highlights a gap in the LW rationality toolbox. The reaction of "Lock down, distrust, get cynical, burn it with fire" actually makes you more susceptible to skillful bad actors — like going rigid in response to a judo master grabbing a hold of you. IMO, a mature Art of Rationality would necessarily include learning to navigate cognition-jamming (or cognition-incompatible!) spaces with grace. But I get the sense LW collectively doesn't want to build that skillset. Which is fine, but I find it a bit disappointing [LW · GW].)

I picked up some of the language & framing of this post from Perri Chase. I now talk about this stuff a little differently. And more kindly, I think. I suspect I could write a version of this spell today that would be less of a problem for the LW memetic immune system. Partly because I'm better at slipping through immune systems! (I'm sure that's comforting!) But mostly because I've learned how to work with such systems instead of needing to step around them to have the "real" conversation.

That said, I don't regret writing this post. I got a lot of feedback (including in quite a few PMs across many different media) from people who found this relieving, validating, soothing, deeply helpful, kind, orienting. I'm okay with some people being upset with me if that's the price for enacting this kindness. I went in expecting that price, really.

I think there's a post possible that would be something like a LW-compatible rewrite of this one. It'd remove the "spell" nature and try to lay out some claims & implications for folk to consider. A bit like dissecting a once-living specimen and laying out its organs for examination.

I probably won't write that post. I don't see it doing hardly any good beyond being kind of interesting.

I might write a related post sometime on the nature of Hell as a psychosocial attractor state. AFAICT it's utterly essential study for real Defense Against the Dark Arts. It's also very tricky to talk about in a way that's kind to the listeners or the speaker. But if LW were to learn to take it seriously without falling into it harder, I think that awareness would transform a lot of what "rationality" means here, and it would soften a lot of the sharp edges that can meaningfully hurt people here.

I don't plan on rewriting any of this post for the review. The spell worked great. I want to leave it here as is.

(Though if someone understands the spellcraft and wants to suggest some edits, I'm open to receiving those suggestions! I'm not putting up a wall here. I'm just sharing where I'm at with this post right now, for the sake of the 2022 review.)