Existential Risk and Existential Hope: Definitions

Definition (iii): An existential catastrophe is an event which causes the loss of most expected value.

Can you name any past existential (or nearly so) catastrophies?

I'll note that I don't claim much justification for my views, I'm mostly just stating them to promote some thought:

• The E(V) definition is definitionally true, but needs translation to gain any practical meaning, whereas Bostrom's definition of loss of potential is theoretically reasonable kosher, works across a lot of value systems, and has the additional virtue of being more practically meaningful. Extinction misses a bunch of bad fail-scenarios, but is at least concrete. So it's better to use the latter two by default, and it's rare that we'd need to use the E(V) definition.

• The main other type of definition for existential risk, which Daniel Dewey had suggested to me is existential risk as the reduction of the number of options available to you. This definition seems kind of doomed to me, because I have a feeling it collapses into some kind of physical considerations. e.g. how many particles you have / how many conformations they can take / how much energy or entropy they have - something like that. I think you either need to anchor your definition on extinction, or expected value, or something in-between like "potential", but if you try to define 'options' objectively, I think you end up with something we don't care about anymore.

• Existential eucatastrophes are interesting. Perhaps making a simulation would count. However, they might not exist if we believe the technological maturity conjecture. The notions of expanding cubically / astronomical waste feel kinda relevant here - if failing some extinction event, everything's going to end up expanding cubically anyway, then you have to decide how you can counterfactually stuff extra value in there. It still seems like the main point of leverage over the future would be when an AI is created. And in that context, I think we want people to cooperate and satisfice (reduce existential risk) rather than quibble and all try to create personal eucatastrophes. So I'm not sure the terminology helps.

Anyhow, it's all interesting to think about.

I like the definition of eucatastrophe, I think it's useful to look at both sides of the coin when assessing risk.

Far out example: we receive a radio transmission from an alien craft that passed by our solar system a few thousand years ago looking for intelligent life. If we fire a narrow beam message back at them in the next 10 years they might turn back, after that they'll be out of range. Do we call them back? It's quite likely that they could destroy Earth, but we also need to consider the chance that they'll "pull us up" to their level of civilization, which would be a eucatastrophe.

More relevant example: a child is growing up, his g factor may be the highest ever measured and he's talking his first computer science class at 8 years old. Certainly, if anyone in our generation is going to be give the critical push towards AGI it's likely to be him. But what if he's not interested in AI friendliness and doesn't want to hear about values or ethics?

Definition (iii): An existential catastrophe is an event which causes the loss of most expected value.

A good place to start, but I don't know about the heavy emphasis on expectation. The problems due to skewed distributions are ever-present. An event with small probability but high value will skew expected value. If a second event were to occur that rendered this impossible, we'd lose a lot of expected value. I'm not sure I'd call that a catastrophe, though.

Not sure I like the (iii) definition (" the loss of most expected value"). It just transfers all the burden onto the word "value" which is opaque, slippery, and subject to wildly different interpretation.

Consider that e.g. for all the Christians an irrefutable discovery that the whole Jesus thing was a fake and a hoax would count as an existential catastrophe.

I haven't had a chance to read the report fully yet but my immediate reaction from looking at the first two definitions given is that they don't rely on a specific moral framework or class of moral framework whereas the proposed definition seems to rely on some utilitarian or close to utilitarian notion.

Edit: Now read the whole thing, and it would be nice to have some substantial address to the issue raised in the above paragraph. Also, calling this a technical report seems a little overblown given how short it is. And as a matter of signaling a formal bibliography would be nice.