Posts
Comments
It's hardly a wilful ignorance, it's a deliberate rejection. A good decision theory, by nature, should produce results that don't actually depend on visible precommitments to achieve negotiation equilibrium, since an ideal agent negotiating ought to be able to accept postcommitment to things you would predictably wish you'd precommitted to. And if a decision theory doesn't allow you to hold out for fairness in the face of an uneven power dynamic, why even have one?
There's a difference between "technology that we don't know how to do but it's fine in theory", "technology that we don't even know if it's possible in principle" and "technology that we believe isn't possible at all". Uploading humans is the former; we have a good theoretical model for how to do it and we know physics allows objects with human brain level computing power.
Time travel is the latter.
It's perfectly reasonable for a civilisation to estimate that problems of the first type will be solved without becoming thereby committed to believing in time travel. Being ignorant of a technology isn't the same as being ignorant of the limits of physics.
But this prediction market is exactly the one case where, if the Keepers are concerned about AGI existential risk, signalling to the market not to do this thing is much much more important than preserving the secret. Preventing this thing is what you're preserving the secret for; if Civilization starts advancing computing too quickly the Keepers have lost.
To deceive in a prediction market is to change the outcome. In this case in the opposite of the way the Keepers want. The whole point of having the utterly trustworthy reputation of the Keepers is so that they can make unexplained bids that strongly signal "you shouldn't do this and also you shouldn't ask too hard why not" and have people believe them.
Given the sheer complexity of the human brain, it seems very unlikely anyone could possibly assess a 97% probability of revival without conditioning on AI of some strong form, if not a full AGI.
In a sane society it would be a task that an average citizen understood and could take on if necessary.
No, for a similar reason: to be the sort of person who gives in to threats is to motivate threats against you.
You should only negotiate for the things that you predict are actually part of an agent's utility function, not things that you believe to be part of a hostile function adopted only to impose utility costs on you.