Posts
Comments
The vast majority of people who read about Pascal's Mugging won't actually be convinced to give money to someone promising them ludicrous fulfilment of their utility function. The vast majority of people who read about Roko's Basilisk do not immediately go out and throw themselves into a research institute dedicated to building the basilisk. However, they also do not stop believing in the principles underpinning these "radical" scenarios/courses of action (the maximization of utility, for one). Many of them will go on to affirm the very same thought processes that would lead you to give all your money to a mugger or build an evil AI, for instance by donating money to charities they think will be most effective.
This suggests that most people have some innate way of distinguishing between "good" and "bad" implementations of certain ideas or principles that isn't just "throw the idea away completely". It might* be helpful if we could dig out this innate method and apply it more consciously.
*I say might because there's a real chance that the method turns out to be just "accept implementations that are societally approved of, like giving money to charity, and dismiss implementations that are not societally approved of, like building rogue AIs". If this is the case, then it's not very useful. But it's probably worth investigating some amount at least.
It confuses me a little that you're proposing a broad question ("is it okay for X to do Y?"/"should X feel free to do Y?") and then answering with a personal preference ("I personally would enjoy it more if X did Y"). Surely there are a lot of people who don't mind or even prefer inconsistent axioms in the literature they read, and it seems fine for writers to create content catering to their desires.
"Is it optimal, according to my values, to devote my time and energy to working on [thing I am devoting my time and energy to working on]?" is probably going to be the right question to ask for quite a lot of people.
This is a clever answer, but we don't know how the oracle works - they're supposedly omniscient and there's some chance that they can pull the answer magically from thin air (or some other clever method of derivation that doesn't require any simulation), in which case you just wasted a very valuable question.