against "AI risk"

Speaking only for myself, most of the bullets you listed are forms of AI risk by my lights, and the others don't point to comparably large, comparably neglected areas in my view (and after significant personal efforts to research nuclear winter, biotechnology risk, nanotechnology, asteroids, supervolcanoes, geoengineering/climate risks, and non-sapient robotic weapons). Throwing in all x-risks and the kitchen sink in, regardless of magnitude, would be virtuous in a grand overview, but it doesn't seem necessary when trying to create good source materials in a more neglected area.

bio/nano-tech disaster

Not AI risk.

I have studied bio risk (as has Michael Vassar, who has even done some work encouraging the plucking of low-hanging fruit in this area when opportunities arose), and it seems to me that it is both a smaller existential risk than AI, and nowhere near as neglected. Likewise the experts in this survey, my conversations with others expert in the field, and reading their work.

Bio existential risk seems much smaller than bio catastrophic risk (and not terribly high in absolute terms), while AI catastrophic and x-risk seem close in magnitude, and much larger than bio x-risk. Moreover, vastly greater resources go into bio risks, e.g. Bill Gates is interested and taking it up at the Gates Foundation, governments pay attention, and there are more opportunities for learning (early non-extinction bio-threats can mobilize responses to guard against later ones).

This is in part because most folk are about as easily mobilized against catastrophic as existential risks (e.g. Gates thinks that AI x-risk is larger than bio x-risk, but prefers to work on bio rather than AI because he thinks bio catastrophic risk is larger, at least in the medium-term, and more tractable). So if you are especially concerned about x-risk, you should expect bio risk to get more investment than you would put into it (given the opportunity to divert funds to address other x-risks).

Nanotech x-risk would seem to come out of mass-producing weapons that kill survivors of an all out war (which leaves neither side standing), like systems that could replicate in the wild and destroy the niche of primitive humans, really numerous robotic weapons that would hunt down survivors over time, and such like. The FHI survey gives it a lot of weight, but after reading the work of the Foresight Institute and Center for Responsible Nanotechnology (among others) from the last few decades since Drexler's books, I am not very impressed with the magnitude of the x-risk here or the existence of distinctive high-leverage ways to improve outcome around the area, and the Foresight Institute continues to operate in any case (not to mention Eric Drexler visiting FHI this year).

Others disagree (Michael Vassar has worked with the CRN, and Eliezer often names molecular nanotechnology as the x-risk he would move to focus on if he knew that AI was impossible), but that's my take.

Malthusian upload scenario

This is AI risk. Brain emulations are artificial intelligence by standard definitions, and in articles like Chalmers' "The Singularity: a Philosophical Analysis."

highly destructive war

It's hard to destroy all life with a war not involving AI, or the biotech/nanotech mentioned above. The nuclear winter experts have told me that they think x-risk from a global nuclear war is very unlikely conditional on such a war happening, and it doesn't seem that likely.

bad memes/philosophies spreading among humans or posthumans and overriding our values

There are already massive, massive, massive investments in tug-of-war over politics, norms, and values today. Shaping the conditions or timelines for game-changing technologies looks more promising to me than adding a few more voices to those fights. On the other hand, Eliezer has some hopes for education in rationality and critical thinking growing contagiously to shift some of those balances (not as a primary impact, and I am more skeptical). Posthuman value evolution does seem to sensibly fall under "AI risk," and shaping the development and deployment of technologies for posthumanity seems like a leveraged way to affect that.

upload singleton ossifying into a suboptimal form compared to the kind of superintelligence that our universe could support

AI risk again.

(Are there any doomsday cults that say "doom is probably coming, we're not sure how but here are some likely possibilities"?)

Probably some groups with a prophecy of upcoming doom, looking to every thing in the news as a possible manifestation.

SI/LW sometimes gives the impression of being a doomsday cult...

I certainly never had this impression. The worst that can be said about SI/LW is that some use inappropriately strong language with respect to risks from AI.

What I endorse:

• Risks from AI (including WBE) are an underfunded research area and might currently be the best choice for anyone who seeks to do good by contribute money to an important cause.

What I think is unjustified:

• This is crunch time. This is crunch time for the entire human species. And it’s crunch time not just for us, it’s crunch time for the intergalactic civilization whose existence depends on us.

I would have to assign a +90% probability to risks from AI, to pose an existential risk, to endorse the second stance. I would further have to be highly confident that we will have to face associated risks within this century and that the model uncertainty associated with my estimates is low.

You might argue that I would endorse the second stance if NASA told me that there was a 20% chance of an asteroid hitting Earth and that they need money to deflect it. I would indeed. But that seems like a completely different scenario to me.

That intuition might stem from the possibility that any estimates regarding risks from AI are very likely to be wrong, whereas in the example case of an asteroid collision one could be much more confident in the 20% estimate. As the latter is based on empirical evidence while the former is inference based and therefore error prone.

What I am saying is that I believe that SI is probably the top charity right now but that it is not as far ahead of other causes as some people here seem to think. I don't think that the evidence allows anyone to claim that trying to mitigate risks from AI is the best one could do and be highly confident about it. I think that it is currently the leading cause, but only slightly. And I am highly skeptical about using the expected value of a galactic civilization to claim otherwise.

Are there any doomsday cults that say "doom is probably coming, we're not sure how but here are some likely possibilities"?

No, but there are lots of cults that say "we are the people to solve all the world's problems." Acknowledging the benefits of Division of Labour is un-cult-like.

Why does SI/LW focus so much on AI-FOOM disaster, with apparently much less concern for things like

• Malthusian upload scenario

For my part I consider that scenario pretty damn close to the AI-FOOM. ie. It'll quite probably result in a near equivalent outcome but just take slightly longer before it becomes unstoppable.

Personally, I care primarily about AI risk for a few reasons. One is that it is an extremely strong feedback loop. There are other dangerous feedback loops, including nanotech, and I am not confident which will be a problem first. But I think AI is the hardest risk to solve, and also has the most potential for negative utility. I also think that we are relatively close to being able to create AGI.

As far as I know, the SI is defined by its purpose of reducing AI risk. If other risks need long-term work, then each risk needs a dedicated group to work on it.

As for LW, I think it's simply that people read EY's writing on AI risk, and those that agree tend to stick around and discuss it here.

To me it seems reasonable to focus on self-improving AI instead of wars and nanotechnology. If we get the AI right, then we can give it a task to solve our problems with wars, nanotechnology, et cetera (the "suboptimal singleton" problem is included in "getting the AI right"). One solution will help us with other solutions.

As an analogy, imagine yourself as an intelligent designer of your favorite species. You can choose to give them an upgrade: fast feet, thick fur, improved senses, or human-like brain. Of course you should choose a human-like brain, because this allows them to also fix their problems with feet, fur and senses. Now when you have an opportunity to give them Friendly AI as a next upgrade, you should do it, because it will help them fix many other problems too.

This reasoning does not work if chance to make the Friendly AI is extremely low, and the chances of fixing other problems are much higher. Then it makes sense to fix the other problems first. Important thing is, in long term we want to fix all these problems, so it's not about whether "A" is better than "B", but whether "A, then B" is better than "B, then A".

The answer to your initial question is that Eliezer and Luke believe that if we create AI, the default result is itkills us all.or does.something else equally unpleasant. And also that creating Friendly AI will be an extraordinarily good thing, in part (and only in part) because it would be excellent protection against other risks.

That said, I think there is a limit to how confident anyone ought to be in that view, and it is worth trying to prepare for other scenarios.

What does "doomsday cult" mean? I had been under the impression that it referred to groups like Heaven's Gate or Family Radio which prophesied a specific end-times scenario, down to the date and time of doomsday.

However, Wikipedia suggests the term originated with John Lofland's research on the Unification Church (the Moonies):

Doomsday cult is an expression used to describe groups who believe in Apocalypticism and Millenarianism, and can refer both to groups that prophesy catastrophe and destruction, and to those that attempt to bring it about. The expression was first used by sociologist John Lofland in his 1966 study of a group of Unification Church members in California, Doomsday Cult: A Study of Conversion, Proselytization, and Maintenance of Faith. A classic study of a group with cataclysmic predictions had previously been performed by Leon Festinger and other researchers, and was published in his book When Prophecy Fails: A Social and Psychological Study of a Modern Group that Predicted the Destruction of the World.

(This is the same When Prophecy Fails that Eliezer cites in Evaporative Cooling of Group Beliefs, by the way. Read the Sequences, folks. Lotsa good stuff in there.)

Wikipedia continues, describing some of the different meanings that "doomsday cult" has held:

Some authors have used "doomsday cult" solely to characterize groups that have used acts of violence to harm their members and/or others, such as the salmonella poisoning of salad bars by members of the Bhagwan Shree Rajneesh group, and the mass murder/suicide of members of the Movement for the Restoration of the Ten Commandments of God group. Others have used the term to refer to groups which have made and later revised apocalyptic prophesies or predictions, such as the Church Universal and Triumphant led by Elizabeth Clare Prophet, and the initial group studied by Festinger, et al. Still others have used the term to refer to groups that have prophesied impending doom and cataclysmic events, and also carried out violent acts, such as the Aum Shinrikyo sarin gas attack on the Tokyo subway and the mass murder/suicide of members of Jim Jones' Peoples Temple group after similar types of predictions.

So, "doomsday cult" seems to have a lot to do with repeated prophecies of doom, even in the face of past prophecies being overtaken by events. So far as I know, SIAI seems more to err on the side of not making specific predictions, and thus risking running afoul of getting evicted for not paying rent in anticipated experiences, than in giving us a stream of doomish prophecies and telling us to forget about the older ones when they fail to come true.

Reading on:

While a student at the University of California, Berkeley Lofland lived with Unification Church missionary Young Oon Kim and a small group of American church members and studied their activities in trying to promote their beliefs and win new members for their church. Lofland noted that most of their efforts were ineffective and that most of the people who joined did so because of personal relationships with other members, often family relationships. Though Lofland had made his sociological interests clear to Kim from the outset, when she determined that he was not going to convert to their religion he was asked to move out of their residence. [...] Lofland laid out seven conditions for a doomsday cult, including: acutely felt tension, religious problem-solving perspective, religious seekership, experiencing a turning point, development of cult affective bonds, and neutralization of extracult attachments. He also suggests that individuals who join doomsday cults suffer from a form of deprivation.

I haven't been able to get a hold of a greppable copy of Lofland's book. I'd be interested to see how he expands on these seven conditions. Some of them very well may apply, in some form, to our aspiring rationalists ... I wonder to what extent though they apply to aspirants to any group at some ideological variance from mainstream society, though.

SI/LW sometimes gives the impression of being a doomsday cult

To whom? In the post you linked, the main source of the concern (google hits) turned out not to mean the thing the author originally thought (edit: this is false. Sorry). Merely "raising the issue" is merely privileging the hypothesis.

Anywho, is the main idea of this post "this other bad stuff is similarly bad, and SI could be doing similar amounts to reduce the risk of these bad things?" I seem to recall their justification for focus on AI was that with self-improving AI, you only need to get it right the first time - one person could eliminate the risk if they could solve the right technical problems. With preventing war or preventing upload labor, on the other hand, you need all or most people to cooperate with you, making the marginal effect of one group smaller.

bad memes/philosophies spreading among humans or posthumans and overriding our values

Well,

I am going to assert that the fear of unfriendly AI over the threats you mention is a product of the same cognitive bias which makes us more fascinated by evil dictators and fictional dark lords than more mundane villains. The quality of "evil mind" is what really frightens us, not the impersonal swarm of "mindless" nanobots, viruses or locusts. However, since this quality of "mind," which encapsulates such qualities as "consciousness" and "volition," is so poorly understood by science and so totally undemonstrated by our technology, I would further assert that unfriendly AI is pure science fiction which should be far down the list of our concerns compared to more clear and present dangers.

SI/LW sometimes gives the impression of being a doomsday cult,

Because it fits the pattern exactly. If you have top astronomers worrying about meteorite hitting earth, that is astronomy. If you have nonastronomers (with very few astronomers) worrying about meteorite hitting earth, that's doomsday cult. Or at very best, a vague doomsday cult. edit: Just saying, that's how I classify, works for me. If you have instances (excluding SIAI) where this method of classification fails in damaging way, I am very interested to hear of them, to update my classification method. I might be misclassifying something. I might just go through the list of things that i classified as cults, and classify some items on that list as non-cult, if the classification method fails.