Timothy M. (timothy-bond)
A Proof Against Oracle AI
I have a few objections here:
Even when objectives aren't aligned, that doesn't mean the outcome is literally death. No corporation I interact with us aligned with me, but in many/most cases I am still better off for being able to transact with them.
I think there are plenty of scenarios where "humanity continues to exist" has benefits for AI - we are a source of training data and probably lots of other useful resources, and letting us continue to exist is not a huge investment, since we are mostly self-sustaining. Maybe this isn't literally "being aligned" but I think supporting human life has instrumental benefits to AI.
I think the formal claim is only true inasmuch as it's also true that the space of all objectives that align with the AI's continued existence is also incredibly small. I think it's much less clear how many of the objectives that are in some way supportive of the AI also result in human extinction.