I don't agree that the probability of alignment research succeeding is that low. 17 years or 22 years of trying and failing is strong evidence against it being easy, but doesn't prove that it is so hard that increasing alignment research is useless.

People worked on capabilities for decades, and never got anywhere until recently, when the hardware caught up, and it was discovered that scaling works unexpectedly well.

There is a chance that alignment research now might be more fruitful than alignment research earlier, though there is uncertainty in everything.

We should have uncertainty in the Ten Levels of AI Alignment Difficulty [LW · GW].

The comparison

It's unlikely that 22 years of alignment research is insufficient but 23 years of alignment research is sufficient.

But what's even more unlikely, is the chance that $200 billion on capabilities research plus $0.1 billion on alignment research is survivable, while $210 billion on capabilities research plus $1 billion on alignment research is deadly.

In the same way adding a little alignment research is unlikely to turn failure into success, adding a little capabilities research is unlikely to turn success into failure.

It's also unlikely that alignment effort is even deadlier than capabilities effort dollar for dollar. That would mean reallocating alignment effort into capabilities effort paradoxically slows down capabilities and saves everyone.

Even if you are right that delaying AI capabilities is all that matters

Anthropic still might be a good thing, because Dario Amodei alone isn't responsible for Anthropic's capabilities progress.

Even if Anthropic disappeared, or never existed in the first place, the AI investors will continue to pay money for research, and the AI researchers will continue to do research for money. Anthropic was just the middleman.

If Anthropic never existed, the middlemen would consist of only OpenAI, DeepMind, Meta AI, and other labs. These labs will not only act as the middle man, but lobby against regulation far more aggressively than Anthropic, and may discredit the entire "AI Notkilleveryoneism" movement.

To continue existing at one of these middlemen, you cannot simply stop paying the AI researchers for capabilities research, otherwise the AI investors and AI customers will stop paying you in turn. You cannot stem the flow, you can only decide how much goes through you.

It's the old capitalist dilemma of "doing evil or getting out-competed by those who do."

For their part, Anthropic redirected some of that flow to alignment research, and took the small amount of precautions which they could afford to take. That may be the best one can hope to accomplish against this unstoppable flow from the AI investors to AI researchers.

The small amount of precautions Anthropic did take may have already costed them their first mover advantage. Had Anthropic raced ahead before OpenAI released ChatGPT, Anthropic may have stolen the limelight, got the early customers and investors, and been bigger than OpenAI.

A trivial inconvenience of my gym occasionally not having a barbell cover to protect my back during squats prevented me from going to the gym consistently. I didn't do probably around 10 workouts just because I got an ugh field around my back being in minor pain while the barbell was on it.

No, there is nothing wrong with the referents in the Gettier examples.

The problem is not that the proposition refers to Jones. Within the universe of the scenario, it in fact did not. Smith's mental model implied that the proposition referred to Jones, but Smith's mental model was incorrect in this important respect. Due to this, the fact that the model correctly predicted the truth of the proposition was an accident.

duplicate with Hyperstitions 

This reminded me to the Goldfish reading [LW · GW] post, and it turns out it's the same author.

Images here won't load, but can be seen in the archived version: https://web.archive.org/web/20221107200157/https://www.lesswrong.com/posts/EEZsTatSoJz4CDvAc/distillation-experiment-chunk-knitting

Another interpretation of the no free lunch theorem by @davidad [LW · GW] is that learning/optimization is too trivial under worst-case conditions, but also impractical, so you need to put more constraints to have an interesting solution:

https://www.lesswrong.com/posts/yTvBSFrXhZfL8vr5a/worst-case-thinking-in-ai-alignment#N3avtTM3ESH4KHmfN [LW(p) · GW(p)]

Also, it helps taboo your words. For example, "Toki Pona helps taboo your words" would be rendered as
tenpo toki pi toki pona li sama e tenpo toki pi ni: jan li ken kepeken ala e nimi pi ken ala sona pi pali lili.
"(the) speech-time related to Toki Pona is similar or the same as (the) speech-time with this quality: (the) person cannot use word(s) which cannot be known via small effort." 

Before you complain that this is too long a phrase to be used practically, try to explain the concept of rationalist taboo in less syllables than I did in Toki Pona, whilst not relying on other rationalist jargon.

Let's just think about the pros and cons of picking another forum, vs. continuing to comment on LessWrong, but only being visible by others who choose to see you.

Picking another forum:

• They fit better in other forums than LessWrong. For most rate-limited users, this is true, but they can go to other forums on their own without being forced.
• Less need for LessWrong to write code and increase bandwidth to accommodate them.
• Less chance they say really bad things (neoreactionary content) which worsens the reputation of LessWrong? This doesn't apply to most rate-limited users.

Continuing to comment but only visible to those interested:

• They get to discuss the posts and topics they find engaging to talk about.
• They don't feel upset at LessWrong and the rationalist community.

I think whether it's worth it depends on how hard it is to write the code for them.

I agree that it's not looking good, if all you know is SBF and Zizians. But as far as I know, Annie Altman is not related to either rationalism or effective altruism.