Aligned AI May Depend on Moral Facts

The trouble is that moral facts do not just mean realist notions of moral facts, but include anti-realist and non-cognitivist notions of moral propositions like "do what happens when you extrapolate my preferences in a certain way". In fact the closer I look the more it seems like moral facts are a different sort of confusion than the one I originally thought they were.

And yes we may be able to build AGI we consider aligned without fully resolving issues of meta-ethical uncertainty (in fact, I'm counting on it, because meta-ethical uncertainty cannot be fully resolved!), but I'd just like to address your claim that driving to the store does not require moral facts. There is a weak sense in which it doesn't because you can just drive to the store and do what you want to do this, but so can we also build AGI that just does what it wants without need of moral facts. I doubt we would call such an AGI aligned, though, because its behavior is not constrained by the wants of others.

We might say "well, let's just build an AGI that does what we want not what it wants", but then we have to decide what we want and, importantly, figure out what to do about places where what we want comes into conflict with itself. Because we're not trying to build AGI that is aligned with just a single human but with all of humanity (and maybe with all moral agents) we're forced to figure out how to resolve those conflicts in a way independent of any individual, and this is solidly in the realm of what people consider ethics.

Perhaps we can approach this in a new way, but we'd be foolish to fail to notice and appreciate the many ways people have already tried to address this, and these approaches are grounded in the notion of moral facts even if moral facts are a polymorphic category that is metaphysically quite different depending on your ethics.

Maybe. My guess is not because you probably make a choice somewhere where the AI would resolve conflicts using it's own norm confirming values, but thanks for the suggestion and I'm going to read through carefully both because my initial impression may be wrong and because if I turn out to be right identifying these sort if things is exactly the sort if task I want to enable people to do with my present work.

Well if I'm totally honest I agree with you, but I'm using analytic methods here since it's what my intended audience prefers. I'm working around this by engaging deeply with epistemic uncertainty as it's approached within the analytic tradition, but I also think it smells right now and I haven't totally figured out how to make my intended cuts using the method.

Moral facts are true moral propositions, where we'd a consider a proposition to be a moral proposition if it makes a claim about something that normally fall under the field of ethics. It's hard to be more precise without speculating about the nature of moral facts and picking up a particular system of ethics. For example, moral facts in an anti-realist, non-cognitivist sense may be something like reflexive relationships between moral agents (something like the golden rule) or even values an individual agents holds as true.

I agree we can talk about, say, an AGI having preferences for how it will resolve value conflicts, but I don't think we can leave the issue there because the AGI's preferences now take on the functional role of moral facts in its reasoning about conflicting human values. This does not mean the AGI is necessarily certain about these moral facts and may change its preferences so that the moral facts effectively change as it learns more.

Perhaps you've revealed some of the trouble, though, in that "moral facts" are too fuzzy a category for the sort of analysis I was attempting and my confusion results from the willingness to consider some things moral facts that are not and I would be better served by more carefully engaging with the terminology.