You Provably Can't Trust Yourself

A puzzle: How can one rigorously construct Self-PA as recursively axiomatized first order theory in the language of PA?

Eliezer, I have an objection to your metaethics and I don't think it's because I mixed levels:

If I understood your metaethics correctly, then you claim that human morality consists of two parts: a list of things that we value(like love, friendship, fairness etc), and what we can call "intuitions" that govern how our terminal values change when we face moral arguments. So we have a kind of strange loop (in the Hofstadterian sense); our values judge if a moral argument is valid or not, and the valid moral arguments change our terminal values. I think I accept this. It explains quite nicely a lot of questions, like where does moral progress comes from. What I am skeptic about is the claim that if a person hears enough moral arguments, their values will always converge to a single set of values, so you could say that his morality approximates some ideal morality that can be found if you look deep enough into his brain. I think it's plausible that the initial set of moral arguments that the person hears will change considerably his list of values, so that his morality will diverge rather than converge, and there won't be any "ideal morality" that he is approximating.

Note that I am talking about a single human that hears different sets of moral arguments, and not about the convergence of moralities across all humans (which is a different matter altogether)

Also note that this is a purely empirical objection; I am asking for empirical evidence that supports your metaethics

IL My understanding was that Terminal Values are not something you ever observe directly (nobody can simply list their Terminal Values). Moral arguments change what use as our approximation to the Moral Calculation. However, if moral arguments did make our actual moral calculations diverge (that is, if our actual moral calculation is not a state function with respect to moral arguments) then that does disprove Eliezer's meta-ethics (along with any hope for a useful notion of morality it seems to me).

I agree— and I balk at the concept of "the" Coherent Extrapolated Volition precisely because I suspect there are many distinct attractors for a moral framework like ours. Since our most basic moral impulses come from the blind idiot god, there's no reason for them to converge under extrapolation; we have areas of agreement today on certain extrapolations, but the convergence seems to be more a matter of cultural communication. It's not at all inconceivable that other Everett branches of Earth have made very different forms of moral progress from us, no less consistent with reason or consequences or our moral intuitions.

I'd be very interested, of course, to hear Eliezer's reasons for believing the contrary.

Patrick,

It looks doubtful that different moral attractors would look equally right. You don't just dive into an attractor, ending up in whatever one you happened to come across, walking it all the way. The decision to advance from right to right1 and not right2 is invalid if it is not right, and will be corrected as possible. Premeditation should minimize such errors.

Vladimir,

Just to clarify (perhaps unnecessarily): by an attractor I mean a moral framework from which you wouldn't want to self-modify radically in any direction. There do exist many distinct attractors in the space of 'abstracted idealized dynamics', as Eliezer notes for the unfortunate Pebblesorters: they might modify their subgoals, but never approach a morality indifferent to the cardinality of pebble heaps.

Eliezer's claim of moral convergence and the CEV, as I understand it, is that most humans are psychologically constituted so that our moral frameworks lie in the 'basin' of a single attractor; thus the incremental self-modifications of cultural history have an ultimate destination which a powerful AI could deduce.

I suspect, however, that the position is more chaotic than this; that there are distinct avenues of moral progress which will lead us to different attractors. In your terms, since our current right is after all not entirely comprehensive and consistent, we could find that both right1 and right2 are both right extrapolations from right, and that right can't judge unequivocally which one is better.

Especially given that exposure to different fact patterns could push you in different directions. E.g. suppose right now I try to do what is right_1 (subscripts on everything to avoid appearance of claim to universality). Now, suppose that if I experience fact pattern facts_1 I conclude that it is right_1 to modify my 'moral theory' to right_2. but if I experience fact pattern facts_2 I conclude that it is right_1 to modify to right_3.

Now, that's all well and good. Eliezer would have no problem with that, as long as the diagram commutes: that is, if it's true that ( if I've experienced facts_1 and moved to right_2, and then I experience facts_2, I will move to right_4), it must also be true that ( if I've experienced facts_2 and moved to right_3, and then experience facts_1, I will move to right_4).

I suppose that at least in some cases this is true, but I see no reason why in all cases it ought to be. Especially if you allow human cognitive biases to influence the proceedings; but even if you don't (and I'm not sure how you avoid it), I don't see any argument why all such diagrams should commute. (this doesn't mean they don't, of course. I invite Eliezer to provide such an argument).

I still hold that Eliezer's account of morality is correct, except his claim that all humans would reflectively arrive at the same morality. I think foundations and priors are different enough that, functionally, each person has his own morality.

Jadagul: I suppose that at least in some cases this is true, but I see no reason why in all cases it ought to be.

A particular property of moral progress is a property of algorithm that is morality. You can implement many possible algorithms, and for most of them any given property won't hold. If you consider the order in which moral arguments are presented an arbitrary factor, the outcome shouldn't depend on it. If it's found that it does, it is an error that should be corrected, the same way that your morality should be reverted back if it was changed by external factor which you didn't approve, by a red pill that makes you want to kill people that you swallowed by mistake.

Here is a construction of a theory T with the properties of Self-PA. That is, 1) T extends PA and 2) T can prove the consistency of T. Of course, by Godel's second incompleteness theorem, T must be inconsistent, but it is not obviously inconsistent.

In addition to the axioms of PA, T will have one additional axiom PHI, to be chosen presently.

By the devices (due to Godel) used to formalize "PA is consistent" in PA we can find a formula S(x) with one free variable, x, in the language of PA which expresses the following:

a) x is the Godel number of a sentence, S, of the language of PA;

b) The theory "PA + S" is consistent.

By Godel's self-referential lemma, there is a sentence PHI, with Godel number q such that PA proves:

PHI if and only if S([q]).

(Loosely speaking, PHI says the theory obtained by adjoining me to PA is consistent.)

If we take T to be the theory "PA + PHI" then T has the properties stated at the start of this posting.

Of course, to fully understand the argument just given one needs some familiarity with mathematical logic. Enderton's text "A Mathematical Introduction to Logic" covers all the necessary background material.

If you go back and check, you will find that I never said that extrapolating human morality gives you a single outcome. Be very careful about attributing ideas to me on the basis that others attack me as having them.

The "Coherent" in "Coherent Extrapolated Volition" does not indicate the idea that an extrapolated volition is necessarily coherent.

The "Coherent" part indicates the idea that if you build an FAI and run it on an extrapolated human, the FAI should only act on the coherent parts. Where there are multiple attractors, the FAI should hold satisficing avenues open, not try to decide itself.

The ethical dilemma arises if large parts of present-day humanity are already in different attractors.

I think this is your most important post.

@ eli: nice series on lob's theorem, but I still don't think you've added any credibility to claims like "I favor the human one because it is h-right". You can do your best to record exactly what h-right is, and think carefully about convergence (or lack of) under self modification, but I think you'd do a lot better to just state "human values" as a preference, and be an out-of-the-closet-relativist.

We've got moral intuitions; our initial conditions.

We've got values, which are computed, and morality, our decision making computation.

Our morality updates our values, our values inform our morality, and the particular framework we use, evolves over time.

Do all locally-moral explorations of framework-space converge, even assuming the psychological unity of humans? Our morals influence our psychology; can we ignore the effect of our local morality on the psychology of our future-selves?

Eliezer isn't tempting us with a perfect morality; he's unraveling a meta-ethics, a computation for COMPUTING the evolution of morality, i.e. a framework for iteratively building better moralities.

Why assume that even with this meta-ethics, our morality-evolution converges, rather than diverges (or merely remaining as diverse as it currently is)? Maybe it doesn't matter. We've already been warned against the dichotomy between "morality-as-given" and "morality-as-preference". Morality is not a fixed, immutable structure to which our moral utility-functions will all inevitably converge. But there is a general framework within which we can evaluate moralities, analogous to the framework within which mathematicians explore various formal theories (which seems basically correct). But neither is morality merely a preference, again analogous in my mind to the fact that not all possible mathematical theories are 'interesting'. I think Eliezer needs to fill us in on what makes a morality 'interesting'. Oddly enough, in mathematics at least, there is an intuitive notion of 'interesting' based on the consequences of a formal theory; what theorems does said theory generate?

Certainly, we should be able to exclude certain moralities easily; we've got bedrock 'neath us, right?

My previous post was actually written last night. I was unable to post it until just now, and unfortunately had not read the most recent comments ...

Ah, thanks Eliezer, that comment explains a lot. I think I mostly agree with you, then. I suspect (on little evidence) that each one of us would, extrapolated, wind up at his own attractor (or at least at a sparsely populated one). But I have no real evidence for this, and I can't imagine off the top of my head how I would find it (nor how I would find contradictory evidence), and since I'm not trying to build fAI I don't need to care. But what you've just sketched out is basically the reason I think we can still have coherent moral arguments; our attractors have enough in common that many arguments I would find morally compelling, you would also find morally compelling (as in, most of us have different values but we (almost) all agree that the random slaughter of innocent three-year-olds is bad). Thanks for clearing that up.

Am I correct in supposing that you can substitute PA for any consistent calculus such as ZF set theory?

What about a Turing-complete system such as SKI calculus?

How comes anthropomorphizing axioms sounds so weird to me? I don't get the same feeling when anthropomorphizing software, or even stones.

Gödelian arguments are inescapable; you can always isolate the framework-of-trusted-arguments if a mathematical system makes sense at all. Maybe the adding-up-to-normality-ness of my system will become clearer, after it becomes clear that you can always isolate the framework-of-trusted-arguments of a human having a moral argument.

If you hadn't qualified the two statements beginning with, "you can always isolate the framework..." then it seems they would not escape Gödelian arguments. In other words, there is no reason to believe that there isn't a non-isolate-able, general moral Framework, but I suspect that you are right that it would have to be neither mathematical (small 'm') nor of-a-human. ^^

Do I represent well the principles discussed when I say this? ;)