RFC: Philosophical Conservatism in AI Alignment Research

1. AI alignment is a problem with bimodal outcomes, i.e. most of the probability distribution is clustered around success and failure with very little area under the curve between these outcomes.

2. Thus, all else equal, we would rather be extra cautious and miss some paths to success than be insufficiently cautious and hit a path to failure.

At first I didn't understand the connection between these two points, but I think I get it now. (And so I'll spell it out for others who might be like me.)

If there are really only two outcomes and one is much better than the other, then when you're choosing which path to take you want to maximize the probability that it leads you to the good outcome. (Aka the maxipok rule.)

This would be different from a situation where there's many different possible outcomes of varying quality. In that case, if you think a path leads you to the best outcome, and you're wrong, maybe that's not so bad, because it might still lead you somewhere pretty good. And also if the difference between outcomes is not so great, then maybe you're more okay selecting your path based in part on how easy or convenient it is.

Whereas back in the bi-modal world, if you have a path that you're pretty sure leads you to the good outcome, if you've got the time you might as well look for a path you can be more confident in.

(Is that a fair summary, @gworley?)

But Statement 1 doesn't imply caution is always best. A plausibly friendly AI based on several dubious philosophical assumptions (50% chance of good outcome) is better than taking our time to get it right(99%) if someone else will make a paperclip maximizer in the mean time. We want to maximize the likelyhood of the first AI being good, which may mean releasing a sloppily made, potentially friendly AI in race conditions. (assuming the other side can't be stopped)

I like the general thrust here, although I have a different version of this idea, which I would call "minimizing philosophical pre-commitments". For instance, there is a great deal of debate about whether Bayesian probability is a reasonable philosophical foundation for statistical reasoning. It seems that it would be better, all else equal, for approaches to AI alignment to not hinge on being on the right side of this debate.

I think there are some places where it is hard to avoid pre-commitments. For instance, while this isn't quite a philosophical pre-commitment, it is probably hard to develop approaches that are simultaneously optimized for short and long timelines. In this case it is probably better to explicitly do case splitting on the two worlds and have some subset of people pursuing approaches that are good in each individual world.

I'm not really sure what it would mean in practice to operate under the assumption that "there is no discoverably correct ethics or metaethics the AI can learn". The AI still has to do something. The idea of CEV seems to be something like: have the AI act according to preference utilitarianism. It seems to me that this would result in a reasonably good outcome even if preference utilitarianism isn't true (and in fact I think that preference utilitarianism isn't true). So the only ethical assumption is that creating a preference utilitarian AI is a good idea. What would be an example of a weaker ethical assumption that could plausibly be sufficient to design an AI?

Isn't the least convenient world a world where FAI is outright impossible and all AGIs are Unfriendly ?