Don't Condition on no Catastrophes

I suspect the intuition behind asking these questions in this way is this: People mentally have a model of "AGI", and maybe a model of "nuclear war" or other disasters, and these models are mostly separate (that is, when thinking about AGI timelines, most people "don't worry about" nuclear war and other disasters, whatever that means. So if you don't ask people to exclude those considerations, SOME people will exclude them anyway (because they don't come to mind), while others (call them "pedants" perhaps) will try to manually adjust their model for those considerations, which is technically correct but isn't really what you wanted, and which means you will get heterogenous answers.

Here is another illustration of how catastrophe-censoring gives bad results.

Suppose you ask me when I expect (say) the first human to set foot on another planet. And suppose I think that this is likely to happen in 50 years' time if everything goes well, but that there's some class of Bad Thing that might happen that would delay it -- economic disaster, pandemic, environmental collapse, etc. Now, imagine what happens as we ramp up the severity I think this Bad Thing would have, while keeping everything else constant. My estimate of that date will get later and later, as it obviously should ... until the point where the Bad Thing counts as a catastrophe, when (if we're conditioning on no catastrophe) suddenly my estimated date gets much earlier because now the possible worlds where the Bad Thing happens are being excluded from consideration.

(With a more realistic model of Bad Things, rather than a nice monotonic increase followed by an absurd backward jump we get some more complicated curve that eventually turns downward. But it's the same problem.)

Maybe I should write down the theorem that is implicit here, since it wasn't obvious to me at first: if you ask the example Alice your question, then she will answer 50. So it is recovering the AGI distribution from before nuclear war was added to the model. However, the new type of conditional still doesn't seem very natural to me: if Alice has a model in which the processes leading to AGI and nuclear war are correlated with each other in a more substantive way than just precluding each other at completion, then it is not clear what the new conditional is supposed to be measuring about her model. For example, maybe she thinks that both AGI and nuclear war times will be affected by variables like the rate of technological progress, political developments etc. So it seems simpler just to ask the question about what she would predict about AGI if other considerations were removed from her model.

This reminds me of Gaussian Processs Memoization https://arxiv.org/abs/1512.05665

I strongly suspect this is close to what most people who answer these kinds of questions are doing already (provided they think about it at all). I'd thus be surprised if rephrasing it leads to significantly different responses in surveys. But I agree it should be rephrased regardless.

How about weighting each future f by the inverse of the probability of nuclear war before the time of AI in f (and then re-normalising)?

I firmly agree that ignoring other catastrophes is a mistake, and that including more catastrophes in our estimates is necessary.

I don't believe that independence is a valid assumption. Intuitively I expect shared causal impacts of one catastrophic risk on others, which causes me to suspect they will cluster. For example, consider the following chain:

Climate change results in a drought in Syria -> rebellion in Syria -> civil war in Syria -> Russia, Iran, and the United States get involved in the Syrian conflict -> the risk of nuclear catastrophe increases

I break this down like so:

climate change -> military conflict

military conflict -> nuclear risk

The early developments of computing were driven by military conflict; ENIAC was for artillery firing tables and nuclear weapon simulations. I expect military conflict to increase the risk of AGI by driving incentives for it similarly, so I also have:

military conflict -> AGI risk

So just from the cases of climate change, nuclear risk, and AGI risk I have a causal relationship that looks like this:

climate change -> nuclear risk & AGI risk

This isn't enough for me to put sensible numbers on the problem, but it is enough for me to be suspicious of treating them as independent variables. So currently I cringe at questions of the first type, and also of the second type, but I haven't developed any meaningful improvements.

That being said, this was helpful to me in thinking about the counterfactual case, and for some reason this is also the first time I have ever seen the idea of 'cruxing on the wrong question' pointed to, which is very interesting.