As a final note: the term "Butlerian Jihad" is taken from Dune and describes the shunning of "thinking machines" by mankind.

In Dune, "thinking machines" are shunned because of a very longstanding taboo that was pretty clearly established in part by a huge, very bloody war. The intent was to make that taboo permanent, not a "pause", and it more or less succeeded in that.

It's a horrible metaphor and I strongly suggest people stop using it.

the Culture ending, where CEV (or similar) aligned, good ASI is created and brings us to some hypothetical utopia. Humanity enjoys a rich life in some manner compatible with your personal morals.

santa claus to 11 ending: ASI solves our problems and human development stagnates; ASI goes on to do its own thing without killing humans - but without human influence on the lightcone

Um, humans in the Culture have no significant influence on the lightcone (other than maybe as non-agentic "butterfly wings"). The Minds decide what's going to happen. Humans opposed to that will be convinced (often via manipulation subtle enough that they don't even know about it) or ignored. Banks struggled to even find reasons to write stories about the humans, and sometimes had to cheat to do so.

I have come to accept that some people have an attachment to the whole "human influence" thing, but how can you believe that simultaneously say the Culture is a good outcome?

"Likewise, risks from competing nation states (e.g China) could be mitigated via existing intentional collaboration strategies - nuclear proliferation management techniques like inspections & intelligence agencies keeping check on each other could feasibly serve as a means for the world to prevent the development of AI. "

This is a word salad that has zero empirical or theoretical foundation. Gunpowder, greenhouse gases, virus pathology and many other fields have shown this to be empirically false. We'd all be better off is there wasn't arms races and runaway selection (though would we have evolved in the first place?), but denying this fact gets us nowhere. 

I mostly agree with the body of this post, and think your calls to action make sense. 

On your title and final note: Butlerian Jihad feels out of place.  It’s catchy, but it seems like you are recommending AI concerned people more or less do what AI concerned people already do.  I feel like we should save our ability to use words that are a call to arms for a time when that is what we are doing.

We find ourselves at the precipice of

tAI;dr.

The phone seems to be off the hook for most of public on AI danger, perhaps a symptom of burnout from numerous other scientific Millenialist scares - people have been hearing of imminent dangers of catastrophe for decades that have failed to impact the lives of 95%+ of population in any significant way and now just write it all off as more of the same.

I am sure that most LW readers find little in the way of positive reception for our concerns amongst less technologically engaged family members and acquaintances.  There are just too many comforting techno-utopian narratives that we are still having to compete with informed by the superficially positive representations in movies and TV, and and most people bias towards optimism in relatively good/comfortable times like these.  We are dealing with emotional reactions and the sheeply 'vibe' of population rather than thoughtfully considered positions.

But I am pretty sure that will all change as soon as we see significant AI competition/undercutting for white collar professions.  Those affected will quickly start baying for blood, and the electorally dominant empathic response to those sad stories of the economically impacted will rapidly swing the more emotively governed wider population against AI.  OECD Democratic govts will inevitably then move to ban AI taking jobs of particularly politically protected classes of people (might still leave some niches vulnerable - like medicine where there is always a shortage of service supply causing ridiculously high prices, and perhaps tech for vengeful reasons).  It will be Butlerian Jihad lite, aimed at symptoms rather than causes, and will likely buy us a few years of relative normalcy as more dangerous ASI is developed in govt approved labs and by despotic regimes.

I doubt it will save us in 50 year time frame, but will perhaps make the economic disruption less severe for 5-10 years.

The way to have a bigger impact in the shorter term would be to buy AI-danger editorial support from influencers, particularly those with young female audiences that form the core of environmental and other popular protest movements.  They are by temperament the easiest to bring on board, and have outsized poltical influence.