Conservation of expected moral evidence, clarified

The canonical example is that of a child who wants to steal a cookie. That child gets its morality mainly from its parents. The child strongly suspects that if it asks, all parents will indeed confirm that stealing cookies is wrong. So it decides not to ask, and happily steals the cookie.

I find this example confusing. I think what it shows is that children (humans?) aren't very moral. The reason the child steals instead of asking isn't anything to do with the child's subjective moral uncertainty -- it's that the penalty for stealing-before-asking is lower than stealing-after-asking, and the difference in penalty is enough to make "take the cookie and ask forgiveness if caught" better than "ask permission".

I suspect this is related to our strong belief in being risk-averse when handing out penalties. If I think there's a 50% chance my child misbehaved, the penalty won't be 50% of the penalty if they were caught red-handed. Often, if there's substantial uncertainty about guilt, the penalty is basically zero -- perhaps a warning. Here, the misbehavior is "doing a thing you knew was wrong;" even if the child knows the answer in advance, when the child explicitly asks and is refused, the parent gets new evidence about the child's state of mind, and this is the evidence that really matters.

I suspect this applies to the legal system and society more broadly as well -- because we don't hand out partial penalties for possible guilt, we encourage people to misbehave in ways that are deniable.

A child will eventually learn the independence. It's a very different thing to hardcode assume the independence from the start. Values are also complex and there might not be a natural line on what is natural to care about. If I take the the morality of the agent to be independent of it's decisions really far it will mean the AI believes it's a good boy or a bad boy already no matter what it does (and it just doesn't know which). This seems to be opposite what moral behavior is about. There is also no telling that the valued actions need to simple concepts. After all cake is good unless diabetes and death might not be all bad if euthanasia.

Also if I am inconsistent or plain out make an error on assigning the labels I would rather have the AI have a dialogue back to me about it than make silently a really convoluted utility function.

We could off course think that a value loader is a two step process where the values loaded are not taken as is as the actual utility function but modified to make sense in a "non-cheating way". That is the actual utility function always contains terms about moral fairness even if not explicitly input. This reduces the amount of values softcoded as meta-morality is hardcoded. But this layering also solves the original problem. The agent might evaluate the new situation with lower expected utility and would choose not to go there - if it had a choice. But by layering we are taking the choice away so the wrong valuation doesn't matter. By enforcing the hand-inputted conservation of moral evidence we tie the AIs hand to form a moral strategy. "We know better" before hand.

So conservation of expected moral evidence is something that would be automatically true if morality were something real and objective, and is also a desiderata when constructing general moral systems in practice.

Yes, but usual learning and prediction algorithms deal entirely with things that are "real and objective", in the sense that you simply cannot change them (ie: laws of science).

This is yet another domain where my intuitions are outpacing my ability to learn mathematics. For domains where my actions can affect the experiment, I know damn well I should avoid affecting the experiment. The justification is damn simple when you think of data/information as a substance: experiments/learning are done to gain information, and if I alter the outcome of the experiment I gain information only about my own decisions, which I already had, thus rendering the experiment/learning pointless.

This leads to a question of how to model value learning as collecting moral information, and thus make the conclusion epistemically natural to the agent that biasing its own learning process is yielding falsehoods.

This area (or perhaps just the example?) is complicated somewhat because for authority-based moral systems (parental, religious, legal, professional...) directly ignoring a command/ruling is in itself considered to be an immoral act: on top of whatever the content of said act was. And even if the immorality of the act is constant, most of those systems seem to recognise in principle and/or in practice that acting when you suspect you'd get a different order is different to direct breaking of orders.

This makes sense for all sorts of practical reasons: caution around uncertainty, the clearer Schelling point of 'you directly disobeyed', and, cynically, the fact that it can allow those higher in the authority chain plausible deniability (the old "I'm going to pay you based on your sales. Obviously I won't tell you to use unscrupulous methods. If you ask me directly about any I will explicitly ban them. But I might not ask you in too much detail what you did to achieve those sales: that's up to you")

"So conservation of expected moral evidence is something that would be automatically true if morality were something real and objective, and is also a desiderata when constructing general moral systems in practice."

This seems to go against your pulsar example... I guess you mean something like: "if [values were] real, objective, and immutable"?