Some Considerations Against Short-Term and/or Explicit Focus on Existential Risk Reduction

not so much existential risk due to asteroid strike or Venusian global warming

This is fairly true of asteroids with a short enough time horizon, but there are still uncertainties over the future costs of asteroid defense and detection, military applications of asteroid defense interceptors, and so forth.

Standard cost-benefit analysis on non-Venusian global warming involves (implicit or explicit) projections of climate sensitivity, technological change, economic and population growth, risks of nuclear war and other global catastrophic risks, economic damages of climate change, and more 90 (!!!) years into the future or even centuries. There are huge areas where subjective estimates play big roles there. .

Venusian runaway surprise climate change of the kind discussed by Martin Weitzman or Broome, the sort threatening human extinction (through means other than generic social collapse non-recovery) involves working with right-tail outcomes and small probabilities of big impacts, added on to all the other social, technological, and other uncertainties. Nonetheless, one can put reasonable probability distributions on these and conclude that there are low-hanging fruit worth plucking (as part of a big enough global x-risk reduction fund). However, it can't be done without dealing with this uncertainty.

Regarding social costs and being an "odd duck": note that Weitzman, in his widely celebrated article on uncertainty about seemingly implausible hard-to-analyze high-impact climate change, also calls for work on risks of AI and engineered pathogens as some of the handful of serious x-risks demanding attention.

Likewise judge-economist Richard Posner called for preliminary work on AI extinction risk in his book Catastrophe. Philosopher John Leslie in his book on human extinction discussed AI risk at length. Bill Gates went out of his way to mention it as a possibility.

Regarding Fermi calculations, the specific argument in the post is wrong for the reasons JGWeissman mentions: Drake Equation style methods are the canonical way of combining independent probabilities of failure to get the correct (multiplicative) penalty for a conjunction (although one should also remember that some important conclusions are disjunctive, and one needs to take all the disjuncts into account). The conjunction fallacy involves failing to multiply penalties in that way.

With respect to sacrifice/demandingness, that's pretty orthogonal to efficacy. Whether the most effective cause is contributing to saving local children in your own community, deworming African children, paying for cleaning asteroid-watcher telescopes, or researching AI risks, one could make large or small sacrifices. GiveWell-type organizations can act as "utility monsters," a never-ending maw to consume all luxuries, but even most efficient charity enthusiasts are able to deal with that adequately. One can buy fuzzies and utilons and personal indulgences separately, compensating for social weirdness of more effective causes by retaining more for other pursuits.

Regarding Pascal's Mugging, that involves numbers much more extreme than show up in the area of x-risk, by many orders of magnitude.

Intuitively, the position "it doesn't matter how well executed charity X's activities are; since charity X is an existential risk reduction charity, charity X triumphs non-existential risk charities" is for me a reductio ad absurdem for adopting a conscious, explicit, single-minded focus on existential risk reduction.

Sure, most people are not unitary total utilitarians. I certainly am not one, although one of my motives is to avert "astronomical waste" style losses. But this is a bit of an irrelevant comparison: we have strong negative associations with weak execution, which are pretty well grounded, since one can usually find something trying to do the same task more efficiently. That applies to x-risk as well. The meaningful question is: "considering the best way to reduce existential risk I can find, including investing in the creation or identification of new opportunities and holding resources in hope of finding such in the future, do I prefer it to some charity that reduces existential risk less but displays more indicators of virtue and benefits current people in the near-term in conventional ways more?"

I feel right to restrict support to existential risk reduction opportunities that meet some minimal standard for "sufficiently well-conceived and compelling" well above that of multiplying the value of ensuring human survival by a crude guess as to the probability that a given intervention will succeed.

Does the NTI pass your tests? Why?

As humans are systematically biased toward believing that conjunctions are more likely than their conjuncts (at least in certain setting),

The experimental evidence for the conjunction fallacy does not support "systematically". It looks to me like the effect comes from one of the conjoined events acting as an argument in favor of the other that the subjects had not considered when asked about it alone. If the experiment had asked for the probability that diplomatic relations break down between the Soviet Union and America and that there will be major flooding in California, I predict you would not see the effect. Really, I think that "conjunction fallacy" is a bad name, because it describes how the researchers can see that something must have gone wrong in the subjects' reasoning, but it doesn't describe what went wrong.

there's a strong possibility of exponentially overestimating probabilities in the course of Fermi calculations.

Fermi calculations are done explicitly to account for the conjunction of individual events in the correct way. This again is not supported by the experiment which did not involve the subjects using Fermi calculations to prevent errors in probability theory.

Multifoliaterose, if you weren't explicitly focused on existential risk reduction in the short term, what would you be doing instead?

I am interested in the trade-off between directing funds/energy towards explicitly addressing existential risk and directing funds/energy towards education. In anything but the very near term, the number of altruistic, intelligent rationalists appears to be an extremely important determinant of prosperity, chance of survival, etc. There also appears to be a lot of low hanging fruit, both related to improving the rationality of exceptionally intelligent individuals and increasing the number of moderately intelligent individuals who become exceptionally intelligent.

Right now, investment (especially of intelligent rationalist's time) in education seems much more valuable than direct investment in existential risk reduction.

Eliezer's assessment seems to be that the two projects have approximately balanced payoffs, so that spending time on either at the expense of the other is justified. Is this correct? How do other people here feel?

I feel right to restrict support to existential risk reduction opportunities that meet some minimal standard for "sufficiently well-conceived and compelling" well above that of multiplying the value of ensuring human survival by a crude guess as to the probability that a given intervention will succeed.

You don't need a "crude guess" but rather a "crude lower-bound".

The biggest problem with attempting to work on existential risk isn't the expenditure of resources. The problem is that our beliefs on the matter are - more or less necessarily - far mode. And the problem with far mode beliefs is that they have no connection to reality. To us smart, rational people it seems as though intelligence and rationality should be sufficient to draw true conclusions about reality itself even in the absence of immediate data, but when this is actually tried outside narrow technical domains like physics, the accuracy of the results is generally worse than random chance.

The one thing that protects us from the inaccuracy of far mode beliefs is compartmentalization. We believe all sorts of stories that have no connection to reality, but this is usually quite harmless because we instinctively keep them in a separate compartment, and only allow near mode beliefs (which are usually reasonably accurate, thanks to feedback from the real world) to influence our actions.

Unfortunately this can break down for intellectuals, because we import the idea of consistency - which certainly sounds like a good thing on the face of it - and sometimes allow it to persuade us to override our compartmentalization instincts, so that we start actually behaving in accordance with our far mode beliefs. The results range from a waste of resources (e.g. SETI) to horrific (e.g. 9/11).

I've come to the conclusion that the take-home message probably isn't so much hold accurate far mode beliefs - I have no evidence, after all, that this is even possible (and yes, I've looked hard, with strong motivation to find something). The take-home message, I now think, is compartmentalization evolved for good reason, it is a vital safeguard, and whatever else we do we cannot afford to let it fail.

In short, the best advice I can give is, whatever you do, do it in a domain where there is some real world feedback on the benefits, a domain where you can make decisions on the basis of probably reasonably accurate near mode beliefs. Low-leverage benefit is better than high-leverage harm.