LessWrong 2.0 Reader

Welcome! Don't be too worried, you can try posting some stuff and see how it's received. Based on how you wrote this comment, I think you won't have much trouble. The New User Guide and other stuff gets worded a bit sternly because of the people who tend not to put in much effort at all and expect to be well received – which doesn't sound like you at all. It's hard hard to write one document that's stern to those who need it and more welcoming to those who need that, unfortunately.

I assume that at [year(TAI) - 3] we'll have a decent idea of what's needed

Why?? What happened to the bitter lesson?

My gut says it's now at least 5%, which seems easily high enough to start putting together an emigration plan. Is that alarmist?

That's a crazy low probability.

More generally, what would be an appropriate smoke alarm for this sort of thing?

You're already beyond the "smoke alarm" stage and into the "worrying whether the fire extinguisher will work" stage.

Thanks, seems pretty good on a quick skim, I'm a bit less certain on the corrigibility section, also more issues might become apparent if I read through it more slowly.

Anyone have a logical solution to exactly why we should act altruistically? I know it makes sense evolutionarily through game theory and statistics, but human decision making is still controlled by emotions, and it's still most advantageous for an individual actor to follow their own self-interest to a degree in a social community. I know how altruistic actors develop, but not why unconstrained intelligences should choose to do so. 

I am very much not a fan of this project's website linking to it's leaders' coaching program. I'm sorry, but that just screams grift. James Norris: If you are even a real person (your personal website screams SEO-optimized LLM slop), please do better. You are being actively unhelpful.

if there are any survivors, you solved alignment

I believe deploying the Observer [LW · GW] satisfies this requirement. The Observer is an ASI that's interested in the continuation of humanity's story. It will intervene and not let humanity get wiped out, though it gets to choose how many casualties there are before it intervenes, which could well be in the billions.

Apparently people have been trying to do such comparisons:

Hugging Face researchers aim to build an ‘open’ version of OpenAI’s deep research tool

In hindsight, I over-updated on my previous success with a poorly-written angry short post with a clickbait title and lots of inline links criticizing the rationality community [LW · GW]. Oops.

You might be interested in this paper by Wolfgang Spohn on auto-epistemology and Sleeping Beauty (and related) problems (Sleeping Beauty starts on p. 388). Auto-epistemic models have more machinery than the basic model described in this post has, but I'm not sure there's anything special about your example that prevents it being modeled in a similar way.