Posts
Comments
Thanks for the pointers, I should have done more research before writing this up. After a quick glance my initial impression is that there still isn't a good solution (even an uncomputable one) to the problems such as the one I describe at the end, or the one AlexMennen mentions.
I agree with the first point, and I don't have solid solutions to this. There's also the fact that some games are easier to optimize than others (name a number game I described at the end vs. chess), and this complexity is impossible to capture while staying computation-agnostic. Maybe one can use the length of the shortest proof that taking action a leads to utility u(a) to account for these issues..
The second point is more controversial, my intuition is that first agent is an equally good optimizer, even if it did better in terms of payoffs. Also, at least in the setting of deterministic games, utility functions are arbitrary up to encoding the same preference orderings (once randomness is introduced this stops being true)
I agree with the point that we should be investing more into research on direct reduction of suffering(as a phenomenon that happens in brains), rather than reducing the proxies for it.
This is true for humans as well as for animals: e.g. investing into discovering direct stimulation/surgery approaches to reducing or even turning off pain (or just the painfullness of pain, see pain asymbolia) might have greater impact on life satisfaction than it's opportunity cost for say cancer research.
I am not at all knowledgeble on the subject (and would love to be corrected), but I suspect that ever since lobotomy was declared unethical no interventions for pain other than chemical ones have been seriously investigated.