My prior on humans going extinct in the next century is less than .05%, because in the last 2,000 centuries, we haven't. My prior on civilization ending in the next century is less than 1.6%, because in the last 60 centuries, it hasn't.

I have, however, absolutely no faith in those priors, because in any universe in which either of those things happened, nobody is asking those questions.

As for evidence that would update those priors? Well... I see no evidence of anything that could or would end our species, so I have nothing to update on there. I see some evidence of things that could end our civilization, so that could be updated slightly higher. But my error bars dominate the question.

So I'm going to say <1% odds of humans (or rather, human minds) going extinct in the next century. Indeed, I'd bet my life it won't happen.

Estimating the probability of human extinction

You need to be clear about whether you're talking about the extinction of the human species, or the extinction of all descendants of the human species. Does death by Unfriendly AI count as extinction, or evolution?

I like that both Bostrom and Rees attached dates to their estimates, though Rees talked about a civilization destroying event, not an extinction event.

since Earth-originating life has existed for some 3.5 billion years without an existential catastrophe having happened.

Human life is only a blink in the eye of that 3.5billion years. How much of those 3.5billion years would be compatible with human life? How many catastrophes have occurred in that time which would wipe us out today? Would wipe out civilization today?

Great overall, but I disagree with this "while colonization would insulate us against a number of potential existential risks, there are some risks that it wouldn’t stop. A physics disaster on Earth, for example, could have consequences that are cosmic in scope. For example, the universe might not be in its most stable state. Consequently, a high-powered particle accelerator could tip the balance, resulting in a 'catastrophic vacuum decay, with a bubble of the true vacuum expanding at the speed of light.”'

If a positive singularity occurs and the solution to the Fermi paradox is that we are alone I would like to make a copy of myself and put that copy on a spaceship that travels fast enough away from earth so that (given sufficient time) when you add in the expansion of the universe something starting at earth and traveling at the speed of light would not be able to reach me. As I understand it, once I have traveled far enough from earth it will be impossible for something from earth to reach me regardless of my speed.

Thanks so much for these incredibly thoughtful responses. Very, very helpful.

In sum, just as a dog can’t possibly comprehend any of the natural and anthropogenic risks mentioned above, so too could there be risks that forever lie beyond our epistemic reach

This seems to prove too much. For any outcome X, there may be ways of reaching it that are "beyond our epistemic reach", therefore the true probability must be higher than we think.

This also holds for ~X (not necessarily with the same evidential weight as for X). For instance, there may be events we are incapable of imagining which would drastically reduce existential risk, just like FAI would.

One might argue that such risks, if they exist at all, must be highly improbable, since Earth-originating life has existed for some 3.5 billion years without an existential catastrophe having happened. But this line of reasoning is deeply flawed: it fails to take into account that the only worlds in which observers like us could find ourselves are ones in which such a catastrophe has never occurred.

K-T, the extinction event famous for killing the dinosaurs, was only 66 Mya (million years ago), and would pretty certainly kill all humans if it were to reoccur today. So could other big extinction events that occurred in the past, like snowball earth scenarios.

The anthropic argument is misapplied here. Evolution has been extremely slow and 'undirected' in producing humans. If planetary sterilization events were much more common, then we would expect to observe a much shorter evolutionary past: tens of millions of years, not billions of them.