AI x-risk is high, which makes cryonics less attractive (because cryonics doesn't protect you from AI takeover-mediated human extinction). But on the flip side, timelines are short, which makes cryonics more attractive (because one of the major risks of cryonics is society persisting stably enough to keep you preserved until revival is possible, and near term AGI means that that period of time is short).

Cryonics is more likely to work, given a positive AI trajectory, and less likely to work given a negative AI trajectory. 

I agree that it seems less likely to work, overall, than it seemed to me a few years ago.

yeahh i'm afraid I have too many other obligations right now to give a elaboration that does it justice. 

Fair enough!

otoh i'm in the Bay and we should definitely catch up sometime!

Sounds good.

Frankly, it feels more rooted in savannah-brained tribalism & human interest than a evenkeeled analysis of what factors are actually important, neglected and tractable.

Um, I'm not attempting to do cause prioritization or action-planning in the above comment. More like sense-making. Before I move on to the question of what should we do, I want to have an accurate model of the social dynamics in the space.

(That said, it doesn't seem a foregone conclusion that there are actionable things to do, that will come out of this analysis. If the above story is true, I should make some kind of update about the strategies that EAs adopted with regards to OpenAI in the late 2010s. Insofar as they were mistakes, I don't want to repeat them.)

It might turn out to be right that the above story is "naive /misleading and ultimately maybe unhelpful". I'm sure not an expert at understanding these dynamics. But just saying that it's naive or that it seems rooted in tribalism doesn't help me or others get a better model.

If it's misleading, how is it misleading? (And is misleading different than "false"? Are you like "yeah this is technically correct, but it neglects key details"?)

Admittedly, you did label it as a tl;dr, and I did prompt you to elaborate on a react. So maybe it's unfair of me to request even further elaboration.

@Alexander Gietelink Oldenziel, you put a soldier mindset react on this (and also my earlier, similar, comment this week). 

What makes you think so?

Definitely this model posits that adversariality, but I don't think that I'm invested in "my side" of the argument winning here, FWTIW. This currently seems like the most plausible high level summary of the situation, given my level of context.

Is there a version of this comment that would regard as better?

I don't dispute that he never had any genuine concern. I guess that he probably did have genuine concern (though not necessarily that that was his main motivation for founding OpenAI).

In a private slack someone extended credit to Sam Altman for putting EAs on the on the OpenAI board originally, especially that this turned out to be pretty risky / costly for him.

I responded:

It seems to me that there were AI safety people on the board at all is fully explainable by strategic moves from an earlier phase of the game.

Namely, OpenAI traded a boardseat for OpenPhil grant money, and more importantly, OpenPhil endorsement, which translated into talent sourcing and effectively defused what might have been vocal denouncement from one of the major intellectually influential hubs of the world.

No one knows how counterfactual history might have developed, but it doesn’t seem unreasonable to think that there is an external world in which the EA culture successfully created a narrative that groups trying to build AGI were bad and defecting.

He’s the master at this game and not me, but I would bet at even odds that Sam was actively tracking EA as a potential social threat that could dampen OpenAI’s narrative flywheel.

I don’t know that OpenPhil’s grant alone was sufficient to switch from the “EAs vocally decry OpenAI as making the world worse” equilibrium to a “largely (but not universally) thinking that OpenAI is bad in private, but mostly staying silent in public + going to work at OpenAI” equilibrium. But I think it was a major component. OpenPhil’s cooperation bought moral legitimacy for OpenAI amongst EAs.

In retrospect, it looks like OpenAI successfully bought out the EAs through OpenPhil, to a lesser extent through people like Paul. 

And Ilya in particular was a founder and one of the core technical leads. It makes sense for him to be a board member, and my understanding (someone correct me) is that he grew to think that safety was more important over time, rather than starting out as an “AI safety person”.

And even so, the rumor is that the thing that triggered the Coup is that Sam maneuvered to get Helen removed. I highly doubt that Sam planned for a situation where he was removed as CEO, and then did some crazy jujitsu move with the whole company where actually he ends up firing the board instead. But if you just zoom out and look at what actually played out, he clearly came out ahead, with control consolidated. Which is the outcome that he was maybe steering towards all along?

So my first pass summary of the situation is that when OpenAI was small and of only medium fame and social power, Sam maneuvered to get the cooperation of EAs, because that defused a major narrative threat, and bought the company moral legitimacy (when that legitimacy was more uncertain). Then after ChatGPT and GPT-4, when OpenAI is rich and famous and has more narrative power than the EAs, Sam moves to remove the people that he made those prestige-trades with in the earlier phase, since he no longer needs their support, and doesn’t have any reason for them to have power over the now-force-to-be-reckoned-with company.

Granted, I’m far from all of this and don’t have confidence about any of these political games. But it seems wrong to me to give Sam points for putting “AI safety people” on the board.

But it is our mistake that we didn't stand firmly against drugs, didn't pay more attention to the dangers of self-experimenting, and didn't kick out Ziz sooner.

These don't seem like very relevant or very actionable takeways.

1. we didn't stand firmly against drugs - Maybe this would have been a good move generally, but it wouldn't have helped with this situation at all. Ziz reports that they don't take psychedelics, and I believe that extends to her compatriots, as well.
2. didn't pay more attention to the dangers of self-experimenting - What does this mean concretely? I think plenty of people did "pay attention" to the dangers of self experimenting. But "paying attention" doesn't automatically address those dangers. 
   
   What specific actions would you recommend by which people? Eliezer telling people not to self experiment? CFAR telling people not to self experiment? A blanket ban on "self experimentation" is clearly too broad ("just don't ever try anything that seems like maybe a good idea to you on first principles"). Some more specific guidelines might have helped, but we need to actually delineate the specific principles.
3. didn't kick out Ziz sooner - When specifically is the point when Ziz should have been kicked out of the community? With the benefit of hindsight bias, we can look back and wish we had separated sooner, but that was not nearly as clear ex ante. 
   
   What should have been the trigger? When she started wearing black robes? When she started calling herself Ziz? When she started writing up her own homegrown theories of psychology? Weird clothes, weird names, and weird beliefs are part and parcel of the rationalist milieu.
   
   As it is, she was banned from the alumni reunion at which she staged the failed protest (she bought tickets in advance, CFAR told her that she was uninvited, and returned her money). Before that, I think that several community leaders had grey-listed her as someone not to invite to events. Should something else have happened, in addition to that? Should she have been banned from public events or private group houses entirely? On what basis? On who's authority?

[For some of my work for Palisade]

Does anyone know of even very simple examples of AIs exhibiting instrumentally convergent resource aquisition? 

Something like "an AI system in a video game learns to seek out the power ups, because that helps it win." (Even better would be a version in which, you can give the agent one of several distinct-video game goals, but regardless of the goal, it goes and gets the powerups first).

It needs to be an example where the instrumental resource is not strictly required for succeeding at the task, while still being extremely helpful.

Is this taken to be a counterpoint to my story above? I'm not sure exactly how it's related.

My model is that Sam Altman regarded the EA world as a memetic threat, early on, and took actions to defuse that threat by paying lip service / taking openphil money / hiring prominent AI safety people for AI safety teams.

Like, possibly the EAs could have crea ed a widespread vibe that building AGI is a cartoon evil thing to do, sort of the way many people think of working for a tobacco company or an oil company. 

Then, after ChatGPT, OpenAI was a much bigger fish than the EAs or the rationalists, and he began taking moves to extricate himself from them.

My read:

"Zizian ideology" is a cross between rationalist ideas (the historical importance of AI, a warped version timeless decision theory, that more is possible with regards to mental tech) and radical leftist/anarchist ideas (the state and broader society are basically evil oppressive systems, strategic violence is morally justified, veganism), plus some homegrown ideas (all the hemisphere stuff, the undead types, etc).

That mix of ideas is compelling primarily to people who are already deeply invested in both rationality ideas and leftist / social justice ideas, an demographic which is predominantly trans women.

Further, I guess there's a lot of bigoted / oppressive societal dynamics that are more evident to trans people than they are to, say, me, because they have more direct experience with those dynamics. If you personally feel marginalized and oppressed by society, it's an easier sell that society is broadly an oppressive system.

Plus very straightforward social network effects, where I think many trans rationalists tend to hang out with other trans rationalist (for normal "people like to hang out to people they relate to" reasons), and so this group initially formed from that social sub-network.

(I endorse personal call outs like this one.)

Why? Forecasting the future is hard, and I expect surprises that deviate from my model of how things will go. But o1 and o3, seem like pretty blatant evidence that reduced my uncertainty a lot. On pretty simple heuristics, it looks like earth now knows how to make a science and engineering superintelligence: by scaling reasoning modes in a self-play-ish regime.

I would take a bet with you about what we expect to see in the next 5 years. But more than that, what kind of epistemology do you think I should be doing that I'm not?

Have the others you listed produced insights on that level? What did you observe that leads you to call them geniuses, "by any reasonable standard"?

It might help if you spelled it as LSuser. (I think you can change that in the settings).

In that sense, for many such people, short timelines actually are totally vibes based.

I dispute this characterization. It's normal and appropriate for people's views to update in response to the arguments produced by others.

Sure, sometimes people most parrot other people's views, without either developing them independently or even doing evaluatory checks to see if those views seem correct. But most of the time, I think people are doing those checks?

Speaking for myself, most of my views on timelines are downstream of ideas that I didn't generate myself. But I did think about those ideas, and evaluate if they seemed true.

I find your commitment to the basics of rational epistemology inspiring.

Keep it up and let me know if you could use support.

I currently believe it's el-es-user, as in LSuser. Is that right?

Can you operationalize the standard you're using for "genius" here? Do you mean "IQ > 150"?

I think that Octavia is confused / mistaken about a number of points here, such that her testimony seems likely to be misleading to people without much context.

[I could find citations for many of my claims here, but I'm going to write and post this fast, mostly without the links, for the time being. I am largely going off of my memory of blog post comments that I read months to years ago, and my memory is fallible. I'll try to accurately represent my epistemic status inline. If anyone knows the links that I'm referring to, feel free to put them in the comments. Same if you think that I'm misremembering something.

To Octavia, if I've gotten any of the following wrong, I encourage you to correct it. I apologize for any rudeness. I'm speaking somewhat more bluntly here than I often would, because it seems more important than usual to help people get clear models of the situation, urgently.]

Octavia is not a Zizian in the relevant sense

Most importantly, I think she is mistaken about whether or not she is "a Zizian".

There are at least types of people that the term "Zizian" might refer to:

1. Someone who has read Sinceriously.fyi and is generally sympathetic to Ziz's philosophy.
2. A member of a relatively tightly-coordinated anarchist conspiracy, that has (allegedly) planned and carried out a series of violent crimes.

Octavia is a Zizian in the first sense, but is not (to my knowledge) a Zizian in the second sense. In fact, she seems unaware or disbelieving that a network of Zizians of the second sense exists. She appears to think that there are only 'people who have benefited from reading Ziz's blog', and no coordinated criminal network to speak of. ^[1]

Because she claims to be a Zizian, one might reasonably expect that she's an authority on what Zizians believe or do. Insofar as people are interested in what the members of the criminal conspiracy believe, I currently think that she is not much of an authority. (Though again, I don't know what kind of contact she's had with who, and maybe they're closer than I know.)

I don't know, but I would guess that Octavia has either not spoken to Ziz at all since Ziz faked her death in 2022, or that the two have minimally conversed. (Octavia obviously has more info about this than I do, and is welcome to correct me.) 

Based on comments that I saw on Sinceriously.fyi, when it was up, I guess that Ziz does not endorse Octavia's take on her philosophy, or regard Octavia as a member of her Vegan Sith crew (though I may be misremembering, and their relationship may have changed since the blog was taken down). 

Octavia gets a lot wrong about what Ziz wrote

Furthermore, Octavia says a number of things that are, by my memory, either outright contradicted by the text of sinceriously.fyi, or seem to me to be importantly mistaken misreadings.

For instance, 

Does Ziz think that core values can change?

Octavia mostly seems to miss the point of Ziz’s arguments that core values are immutable. She says that Ziz never stated explicitly that core values don't change (and that JD tries to heavily imply this without justification), or that Ziz is only making a technical point that if you choose good than that means you were good all along. (Although a few minutes later she does agree that core is "the aspect of yourself that doesn't change, and if you can change it's not the core", so I'm not totally sure what she's saying and maybe I'm just misunderstanding her.)

Ziz does say in the first line of Choices Made Long Ago, "I don’t know how mutable core values are. My best guess is, hardly mutable at all or at least hardly mutable predictably." and goes on to elucidate why apparent changes in values are actually not that. 

Additionally, in her glossary, Ziz defines core: "Core is something in the mind that has infinite energy. Contains terminal values you would sacrifice all else for, and then do it again infinity times with no regret. Seems approximately unchanging across lifespan. Figuratively, the deepest frame in the call stack of the mind, capable of aborting any train of thought, everything the mind does is because it decided for it to happen."

I don't think it's correct to say that Ziz never explicitly said that core values couldn't change. 

(Furthermore, Octavia states in the interview that Ziz sometimes dares the reader to stop being evil. At the end of Choices Made Long Ago, Ziz says "If you have done do lamentable things for bad reasons (not earnestly misguided reasons), and are despairing of being able to change, then either embrace your true values, the ones that mean you’re choosing not to change them, or disbelieve." This is just about the opposite of daring the reader to stop being evil. It's more like daring the reader, who's done bad things and is horrified by that, to stop rationalizing and just admit that they're actually evil.)

This is an extremely key piece of Ziz's moral philosophy. According to my understanding, Ziz and co. feel justified in taking violent action against most people, not just because they happen to do bad things (but could be redeemed), but because they have fundamentally evil values. The Zizians sidestep a bunch of conventional ethical dilemmas, because in their view, almost everyone is an irredeemable moral monster, that not just kills and eats animals, but ultimately desires the destruction of the multiverse.

I've also seen Octavia post elsewhere that if you're evil you can just choose to not to be evil anymore (and change your actions). I believe she's aware that this a deviation from Ziz's view, but she seems to understate how big a difference it makes to the whole worldview.

Does Zizian "debucketing" involve unihemspheric sleep?

She seems to think that her style of "parts work" practice is the same kind of thing that Ziz and Gwen were doing with "debucketing", then says that she doesn’t do any weird unihempishpheric sleep stuff when she’s working with people, suggesting that reports of weird cult-like sleep deprivation practices are false or exaggerations. She says "it's so goofy, it's kind of woo, and unnecessarily cult ritual vibes".

I strongly suspect Octavia's  parts work practice is not at all like the debucketing process that Ziz and Gwen used, and that Ziz and Gwen would not endorse the conflation between them. Trying to draw conclusions about the one based on the other is probably an apples to oranges comparison.

Furthermore, Ziz and Gwen were experimenting with “sleep tech”. That's reported in the blog—even the interviewer points that out!

And in Punching Evil, Ziz writes "Humans are weak creatures; we spend third of our lives incapacitated. (Although, I stumbled into using unihemispheric sleep as a means of keeping restless watch while alone)." 

She claims that JD Pressman made up the specific procedure for unihemispheric sleep in zizians.info, which I have no particular reason to doubt, but I don't think it's valid to claim that the Zizians didn't do anything like that.

Is Ziz a Benthemite utilitarian?

She derides Zizians.info as a hit piece, but then says that she doesn’t disagree with any of the specific claims, she just dislikes the framing. Which is all the weirder, because at least some of it is wrong. According to my memory, Ziz explicitly stated in some comment that she’s not a Benthemite, and never said that she was, whereas Octavia thinks that [paraphrased] ’it’s a reductive simplification to call her a Benthemite”. It’s not a reductive simplification. It’s just false.

Overall, it seems to me that Octivia has her own take on Ziz's philosophy, which is different in several crucial aspects, and that she is either confused about how much her take differ's from those expressed by Ziz, or is (by my lights) underestimating how important those differences are.

I don't think that bystanders should regard her as representing the views of Ziz or the others that are alleged to have been involved in various crimes.
 

1. ^{^}

   Additionally, it's suggestive to me that she offers that Youngblut and Bauckholt were wearing tactical gear because they're autistic nerds who thought that it looked cool. 
   
   That hypothesis is inconsistent with other details about the situation—that they were carrying guns and that the wrapped their phones in aluminum foil (presumably to prevent government authorities from tracking them via their phones). Those details make it seem likely to me that they were attempting oppose or circumvent government authorities, either because they were planning to commit a crime, or because they were generally paranoid of being persecuted. The "maybe just wanted to look cool" hypothesis, in contrast, to suggests that Octavia is very much out of the loop regarding the activities of the hardcore criminal Zizians. 

Somewhat. Not as well as a thinking assistant. 

Namely, the impetus to start still needed to come from inside of me in my low efficacy state.

I thought that I should do a training regime where I took some drugs or something (maybe mega doses of carbs?) to intentionally induce low efficacy states and practice executing a simple crisp routine, like triggering the flowchart, but I never actually got around to doing that.

I maybe still should?

Here's an example. 

This was process I tried for a while to make transitioning out of less effective states easier, by reducing the cognitive overhead. I would basically answer a series of questions to navigate a tree of possible states, and then the app would tell me directly what to do next, instead of my needing to diagnose what was up with me free-form, and then figure out how to respond to that, all of which was unaffordable when I was in a low-efficacy state.

• State modulation process:
  • Start: #[[state modulation notes]]
    • Is this a high activation state or a low activation state?
    • {{[[TODO]]}} {{[[High activation]]}}
      • Am I currently compulsive / stimulation hungry / reactive / urgey?
      • {{[[TODO]]}} {{[[yes]]}}
        • {{[[TODO]]}} Generate a random number between 0 and 2
        • {{[[TODO]]}} {{[[Untitled]]}}
          • {{[[TODO]]}} 0
            • [Hypothesized approach]
            • {{[[TODO]]}} **Do 3 minutes of cardio to clear cache. **
            • {{[[TODO]]}} {{[[Untitled]]}}
              • {{[[TODO]]}} **Take some breaths to clear space and then check in with what's present for me. **
          • {{[[TODO]]}} 1
            • [Hypothesized approach]
            • {{[[TODO]]}} Meditate into the sensation for 4 minutes
          • {{[[TODO]]}} 2
            • [Hypothesized approach]
            • {{[[TODO]]}} Alternating warm-cold shower
      • {{[[TODO]]}} {{[[no]]}}
        • Activated - What is the flavor of the activation?
        • {{[[TODO]]}} {{[[Overwhelm / bursting with threads]]}}
          • **Write out all my threads on note-cards. **
        • {{[[TODO]]}} {{[["Left over activation"]]}}
          • One of:

            • {{[[TODO]]}} Just sit with the sensation, observing it, and feeling the space around it.

            • {{[[TODO]]}} Just focus on my breath.

            • {{[[TODO]]}} Intentionally lower my arousal with biofeedback

        • {{[[TODO]]}} {{[["Psychological"]]}}
          • Do Focusing

          • {{[[Focusing checklist]]}}

        • {{[[TODO]]}} {{[[I'm not sure]]}}
          • Meditate, alternating with Journaling
          • Set a 5 minute timer

          • [[journal]]

    • {{[[TODO]]}} {{[[Low activation]]}}
      • Am I...?
      • {{[[TODO]]}} {{[[Sleep deprived]]}}
        •  
          •  
            • Does it feel like if I tried to nap that I would fall asleep?
            • {{[[TODO]]}} {{[[yes]]}}
              • Do I have at least 2 hours before my next commitment?
              • {{[[TODO]]}} {{[[yes]]}}
                • Do yoga nidra:
                • {{[[nap checklist]]}}
                  1. Put my phone on silent
                  2. Generate a random number between 0 and 3
                     • 0 - nothing
                     • 1 - Turn on pink noise
                     • 2 - Turn on binaural beats in the alpha range
                  3. Take a moment to feel the appeal of getting cozy and going to sleep
                  4. Turn on the air conditioner
                  5. Start a randomized yoga nidra audio
              • {{[[TODO]]}} {{[[no]]}}
                • Do a 20 minute yoga-nidra, then amp up with bellows breathing
            • {{[[TODO]]}} {{[[no]]}}
              • Have I exercised already today?
              • {{[[TODO]]}} {{[[yes]]}}
                • Can I easily play beatsaber, and do I feel like it?
                • {{[[TODO]]}} {{[[yes]]}}
                  • go play beatsaber
                • {{[[TODO]]}} {{[[no]]}}
                  • Do a 20 minute yoga nidra
                  • [note that I need to make a separate playlist for these]
              • {{[[TODO]]}} {{[[no]]}}
                • Go exercise:
                • What is smallest next step for going to exercise?
                  • .
      • {{[[TODO]]}} {{[[Cognitively drained]]}}
        • Does it feel like if I tried to nap that I would fall asleep?
        • {{[[TODO]]}} {{[[yes]]}}
          • Do I have at least 2 hours before my next commitment?
          • {{[[TODO]]}} {{[[yes]]}}
            • Lie down for a 40 minute yoga-nidra
            • {{[[nap checklist]]}}
              1. Put my phone on silent
              2. Generate a random number between 0 and 3
                 • 0 - nothing
                 • 1 - Turn on pink noise
                 • 2 - Turn on binaural beats in the alpha range
              3. Take a moment to feel the appeal of getting cozy and going to sleep
              4. Turn on the air conditioner
              5. Start a randomized yoga nidra audio
          • {{[[TODO]]}} {{[[no]]}}
            • Do a 20 minute yoga-nidra, then amp up with bellows breathing
        • {{[[TODO]]}} {{[[no]]}}
          • Have I exercised already today?
          • {{[[TODO]]}} {{[[yes]]}}
            • Can I easily play beatsaber, and do I feel like it?
            • {{[[TODO]]}} {{[[yes]]}}
              • go play beatsaber
            • {{[[TODO]]}} {{[[no]]}}
              • Do a 20 minute yoga nidra
              • [note that I need to make a separate playlist for these]
          • {{[[TODO]]}} {{[[no]]}}
            • Go exercise:
            • What is smallest next step for going to exercise?
              • .
      • {{[[TODO]]}} {{[[Physically exhausted]]}}
        • [duno]
      • {{[[TODO]]}} {{[[Slugish]]}}
        • Am I slugish from just having eaten?
        • {{[[TODO]]}} Yes, I just ate
          • Have I done my anki review for the day?
            • {{[[TODO]]}} Yes
              • Go read saved articles
              • And if I bounce off of that, go lie down with my eyes closed.
            • {{[[TODO]]}} No
              • Do an anki review
              • If I bounce off, lie down with my eyes closed.
        • {{[[TODO]]}} No, that's not the cause
          • Drink a glass of water
          • {{[[TODO]]}} {{[[Untitled]]}}
            • Do I want to just lie down with my eyes closed?
              • {{[[TODO]]}} Yeah, that sounds good
                • Lay down with my eyes closed.
              • {{[[TODO]]}} No, not really
                • Go exercise,__ even if I already have for the day__.

                • What's the first, smallest step for going to exercise?

      • {{[[TODO]]}} {{[[Fuzzy]]}}
        • {{[[TODO]]}} Drink a tall glass of water
        • {{[[TODO]]}} {{[[Untitled]]}}
          • Go exercise,__ even if I already have for the day__.

          • What's the first, smallest step for going to exercise?

      • {{[[TODO]]}} {{[[Feeling stuck or cloged or weighed down]]}}
        • Do Focusing
        • {{[[Focusing checklist]]}}
      • {{[[TODO]]}} {{[[in a slow-mode]]}}
        • Do I want to be in a slow mode?
        • {{[[TODO]]}} {{[[yes]]}}
          • Great!
        • {{[[TODO]]}} {{[[no]]}}
          • Shift state:
          • Start by imagining yourself from the outside, and imagining yourself in a different state, and feeling the the appeal of that different state.
      • {{[[TODO]]}} {{[[urgy, stimulation hungry]]}}
        • Hypothesized approach:
        • Do 3 minutes of cardio to clear cache.
        • {{[[Untitled]]}}
          • Orient on what's next.
      • {{[[TODO]]}} {{[[Unmotivated]]}}
        • Am I dressed?
        • {{[[TODO]]}} {{[[yes]]}}
          • Feel into my chest. Is there a feeling of "stuckness"?
          • {{[[TODO]]}} {{[[yes]]}}
            • Do focusing.
            • [Focusing checklist]
              •  
                1. Clear a space
                •  
              •  
                1. Delineate all of the parts that I can feel
                •  
              •  
                1. Ask what most wants my attention
                •  
            • {{[[TODO]]}} {{[[Untitled]]}}
              • Did it work?
              • {{[[yes]]}}
                • Yay!
              • {{[[TODO]]}} {{[[no]]}}
                • Lie down with my eyes closed
          • {{[[TODO]]}} {{[[no]]}}
            •  
              •  
                • Am I averse to a specific task or just generally uninterested?
                • {{[[TODO]]}} {{[[Averse to a specific task]]}}
                  • Is it an ambiguity aversion, a "ugh" aversion, or active opposition?)
                  • {{[[TODO]]}} {{[[ambiguity aversion]]}}
                  • What is the absolute minimum next step?
                  • .
                  • {{[[TODO]]}} {{[next]}}
                  • Simulate that next step.
                  • {{[[TODO]]}} {{[["ugh" aversion, it feels like a slog]]}}
                  • Did I precommit to do this as one of my main goals for the day?
                  • {{[[TODO]]}} {{[[yes]]}}
                  • {{[[TODO]]}} Take one deep breath.
                  • {{[[TODO]]}} Start a Focusmate session.
                  • {{[[TODO]]}} {{[[no]]}}
                  • Dialog with my motivation
                  • Any of:

                  • {{[[TODO]]}} Goal factor

                  • {{[[TODO]]}}** Iterate up the motivation chain**
                  • [[Goal-chain inspection]]

                  • {{[[TODO]]}} Fermi estimate of it it seems worth it to do

                  • {{[[TODO]]}} {{[[active opposition]]}}
                  • {{[[TODO]]}} Do Focusing
                  • {{[[Pull up Focusing checklist]]}}
                • {{[[TODO]]}} {{[[generally uninterested]]}}
                  • OR:

                  • {{[[TODO]]}} Journal: What do I want?

                  • {{[[TODO]]}} Lie down with my eyes closed until something seems worth getting up for.
        • {{[[TODO]]}} {{[[no]]}}
          • {{[[TODO]]}} get dressed.
          • {{[[TODO]]}} {{[[Untitled]]}}
            • Am I averse to a specific task or just generally uninterested?
            • {{[[TODO]]}} {{[[Averse to a specific task]]}}
              • Is it an ambiguity aversion, a "ugh" aversion, or active opposition?)
              • {{[[TODO]]}} {{[[ambiguity aversion]]}}
                • What is the absolute minimum next step?
                  • .
                • {{[[TODO]]}} {{[next]}}
                  • Simulate that next step.
              • {{[[TODO]]}} {{[["ugh" aversion, it feels like a slog]]}}
                • Did I precommit to do this as one of my main goals for the day?
                • {{[[TODO]]}} {{[[yes]]}}
                  • {{[[TODO]]}} Take one deep breath.
                  • {{[[TODO]]}} Start a Focusmate session.
                • {{[[TODO]]}} {{[[no]]}}
                  • Dialog with my motivation
                  • Any of:
                  • {{[[TODO]]}} Goal factor
                  • {{[[TODO]]}}** Iterate up the motivation chain**
                  • [[Goal-chain inspection]]
                  • {{[[TODO]]}} Fermi estimate of it it seems worth it to do
              • {{[[TODO]]}} {{[[active opposition]]}}
                • {{[[TODO]]}} Do Focusing

                • {{[[Pull up Focusing checklist]]}}

            • {{[[TODO]]}} {{[[generally uninterested]]}}
              • OR:

                • {{[[TODO]]}} Journal: What do I want?

                • {{[[TODO]]}} Lie down with my eyes closed until something seems worth getting up for.
      • {{[[TODO]]}} {{[[otherwise low energy]]}}
        • Am I dressed?
        • {{[[TODO]]}} {{[[yes]]}}

          • Check in with myself, to see if there's something that I need.

        • {{[[TODO]]}} {{[[no]]}}
          • {{[[TODO]]}} Get dressed.
          • {{[[Untitled]]}}
            • Am I feelingbetter?
              • {{[[yes]]}}
                • Great!
              • {{[[no]]}}

                • Check in with myself, to see if there's something that I need.

A friend of mine once told me "if you're making a decision that depends on a number, and you haven't multiplied two numbers together, you're messing up." I think this is basically right, and I've taken it to heart.

Some triggers for me:

Verbiage

When I use any of the following words, in writing or in speech, I either look up an actual number, or quickly do a fermi estimate in a spreadsheet, to check if my intutitive idea is actually right.

• "Order of magnitude"
• "A lot"
• "Enormous" / "enormously"

Question Templates

When I'm asking a question, that effectively reduces to one of the following forms:

• Is it worth it to [take some action]? (Including an internal conflict about whether something is worth doing.)
• Is [a specific idea] feasible? Does it pencil?
• Is [an event] probable?

One thing that's been critical for me is having a hotkey that opens a new spreadsheet. I want "open a spreadsheet" to be in muscle memory and take litterally less than a second.

I'm open to hiring people remotely. DM me.

Then, since I've done the upfront work of thinking through my own metacognitive practices, the assistant only has to track in the moment what situation I'm in, and basically follow a flowchart I might be too tunnel-visioned to handle myself.

In the past I have literally used flowcharts for this, including very simple "choose your own adventure" templates in roam.

The root node is just "something feels off, or something", and then the template would guide me through a series of diagnostic questions, leading me to root nodes with checklists of very specific next actions depending on my state.

FYI: I'm hiring for basically a thinking assistant, right now, for I expect 5 to 10 hours a week. Pay depending on skill-level. Open to in-person or remote.

If you're really good, I'll recommend you to other people who I want boosted, and I speculate that this could easily turn into a full time role.

If you're interested or maybe interested, DM me. I'll send you my current writeup of what I'm looking for (I would prefer not to post that publicly quite yet), and if you're still interested, we can do a work trial.

However, fair warning: I've tried various version of hiring people to support my metacognition over the past 5 years, and so far none of them have worked well enough that it was worth continuing. I've learned a bit about what I need each time, but a lot of this will probably come down to personal fit.

A different way to ask the question: what, specifically, is the last part of the text that is spoiled by this review?

Can someone tell me if this post contains spoilers?

Planecrash might be the single work of fiction for which I most want to avoid spoilers, of either the plot or the finer points of technical philosophy.

I've sometimes said that dignity in the first skill I learned (often to the surprise of others, since I am so willing to look silly or dumb or socially undignified). Part of my original motivation for bothering to intervene on x-risk, is that it would be beneath my dignity to live on a planet with an impending intelligence explosion on track to wipe out the future, and not do anything about it.

I think Ben's is a pretty good description of what it means for me, modulo that the "respect" in question is not at all social. It's entirely about my relationship with myself. My dignity or not is often not visible to others at all.

I use daily checklists, in spreadsheet form, for this.

Was this possibly a language thing? Are there Chinese or Indian machine learning researchers who would use a different term than AGI in their native language?

If your takeaway is only that you should have fatter tails on the outcomes of an aspiring rationality community, then I don't object.

If "I got some friends together and we all decided to be really dedicatedly rational" is intended as a description of Ziz and co, I think it is a at least missing many crucial elements, and generally not a very good characterization. 

 

I think this post cleanly and accurately elucidates a dynamic in conversations about consciousness. I hadn't put my finger on this before reading this post, and I noe think about it every time I hear or participate in a discussion about consciousness.

Short, as near as I can tell, true, and important. This expresses much of my feeling about the world.

Perhaps one of the more moving posts I've read recently, of direct relevance to many of us.

I appreciate the simplicity and brevity in expressing a regret that resonate strongly with.

The general exercise of reviewing prior debate, now that ( some of ) the evidence is come in, seems very valuable, especially if one side of the debate is making high level claims that their veiw has been vindicated.

That said, I think there were several points in this post where I thought the author's read of the current evidence is/was off or mistaken. I think this overall doesn't detract too much from the value of the post, especially because it prompted discussion in the comments.

I don't remember the context in detail, so I might be mistaken about Scott's specific claims. But I currently think this is a misleading characterization.

Its conflating two distinct phenomena, namely non-mystical cult leader-like charisma / reality distortion fields, on the one hand, and metaphysical psychic powers, on the other, under the label "spooky mind powers", to imply someone is reasoning in bad faith or at least inconsistently.

It's totally consistent to claim that the first thing is happening, while also criticizing someone for believing that the second thing is happening. Indeed, this seems like a correct read of the situation to me, and therefore a natural way to interpret Scott's claims.

I think about this post several times a year when evaluating plans.

(Or actually, I think about a nearby concept that Nate voiced in person to me, about doing things that you actually believe in, in your heart. But this is the public handle for that.)

I don't understand how the second sentence follows from the first?

Disagreed insofar by "automatically converted" you mean "the shortform author has no recourse against this'".

No. That's why I said the feature should be optional. You can make a general default setting for your shortform, plus there should and there should be a toggle (hidden in the three dots menu?) to turn this on and off on a post by post basis.

I agree. I'm reminded of Scott's old post The Cowpox of Doubt, about how a skeptics movement focused on the most obvious pseudoscience is actually harmful to people's rationality because it reassures them that rationality failures are mostly obvious mistakes that dumb people make instead of hard to notice mistakes that I make.

And then we get people believing all sorts of shoddy research – because after all, the world is divided between things like homeopathy that Have Never Been Supported By Any Evidence Ever, and things like conventional medicine that Have Studies In Real Journals And Are Pushed By Real Scientists.

Calling groups cults feels similar, in that it allows one to write them off as "obviously bad" without need for further analysis, reassures one that their own groups (which aren't cults, of course) are obviously unobjectionable.

Read ~all the sequences. Read all of SSC (don't keep up with ACX).

Pessimistic about survival, but attempting to be aggresively open-minded about what will happen instead of confirmation biasing my views from 2015. 

your close circle is not more conscious or more sentient than people far away, but you care about your close circle more anyways

Or, more specifically, this is a non-sequitor to my deonotology, which holds regardless of whether I personally like or privately wish for the wellbeing of any particular entity.

Well presumably because they're not equating "moral patienthood" with "object of my personal caring". 

Something can be a moral patient, who you care about to the extent you're compelled by moral claims, or who's rights you are deontologically prohibited from trampling on, without your caring about that being in particular.

You might make the claim that calling something a moral patient is the same as saying that you care (at least a little bit) about its wellbeing, but not everyone buys that calim.

An optional feature that I think LessWrong should have: shortform posts that get more than some amount of karma get automatically converted into personal blog posts, including all the comments.

It should have a note at the top "originally published in shortform", with a link to the shortform comment. (All the copied comments should have a similar note).

What would be the advantage of that?

There's some recent evidence that non-neural cells have memory like functions. This doesn't, on its own, entail that non-neural cell are maintaining personality-relevant or self-relevant information.

I got it eventaully!

Shouldn't we expect that ultimately the only thing selected for is mostly caring about long run power?

I was attempting to address that in my first footnote, though maybe it's too important a consideration to be relegated to a footnote. 

To say it differently, I think we'll see selection evolutionary fitness, which can take two forms:

• Selection on AIs' values, for values that are more fit, given the environment.
• Selection on AIs' rationality and time preference, for long-term strategic VNM rationality.

These are "substitutes" for each other. An agent can either have adaptive values, adaptive strategic orientation, or some combination of both. But agents that fall below the Pareto frontier described by those two axes^[1], will be outcompeted.

Early in the singularity, I expect to see more selection on values, and later in the singularity (and beyond), I expect to see more selection on strategic rationality, because I (non-confidently) expect the earliest systems to be myopic and incoherent in roughly similar ways to humans (though probably the distribution of AIs will vary more on those traits than humans). 

The fewer generations there are before strong, VNM agents with patient values / long time preferences, the less I expect small amounts of caring for human in AI systems will be eroded. 

1. ^{^}

   Actually, "axes" are a bit misleading since the space of possible values is vast and high dimensional. But we can project it onto the scalar of "how fit are these values (given some other assumptions)?"

[I can imagine this section being mildly psychologically info-hazardous to some people. I believe that for most people reading this is fine. I don't notice myself psychologically affected by these ideas, and I know a number of other people who believe roughly the same things, and also seem psychologically totally healthy. But if you are the kind of person who gets existential anxiety from thought experiments, like from thinking about being a Boltzmann-brain, then you should consider skipping this section, I will phrase the later sections in a way that they don't depend on this part.]

Thank you for the the warning!

To whom are are you talking?