On the fragility of values

As a toy model, let's say the friendly utility function U has a hundred valuable components - friendship, love, autonomy, etc... - assumed to have positive numeric values. Then to ensure that we don't lose any of these, U is defined as the minimum of all those hundred components.

Now define V as U, except we forgot the autonomy term. This will result in a terrible world, without autonomy or independence, and there will be wailing and gnashing of teeth (or there would, except the AI won't let us do that).

That is far from a logical conclusion. Just because something isn't explicitly being maximised that doesn't mean it is not being produced in large quantities.

For example, the modern world is not actively maximising CO2 production - but, nontheless, it makes lots of CO2.

We have hundreds of instrumental values - and if one of them is not encoded as an ultimate preference it will make no difference at all - since it was never an ultimate preference in the first place. Autonomy is likely to be one of those. Humans don't value autonomy for its own sake, it rather is valued instrumentally - since is one of the many things that lets humans achieve their actual goals.

The problem arises when people try to wire-in instrumental values. The point of instrumental values is that they can change depending on circumstances - unless they are foolishly wired in as ultimate values - in which case you get an inflexible system that can't adapt to environmental changes so competently.

I followed the first part of this, and I agree: if V=min(S1) and U=min(S2) and S2 = S1 + a, and a is not an instrumental value on the way to something else in S1, then V-maximization will cause a (and therefore U) to approach zero.

You lost me when you introduced W.

Your conclusion seems to be saying that a system that optimizes for certain things will reliably optimize for those things, the hard part is building a system that optimizes for the things we want. I agree with that much, certainly.

As a toy model, let's say the friendly utility function U has a hundred valuable components - friendship, love, autonomy, etc... - assumed to have positive numeric values. Then to ensure that we don't lose any of these, U is defined as the minimum of all those hundred components.

This sounds like a cure that might be worse than the disease. ("Oh, the AI has access to thousands of interventions which could reliably increase the values of 999 of the different components by a factor of a thousand... but it turns out that they would all end up decreasing the value of one component by .001%, so none of them can be implemented.")

V seems quite close to U when compared to the utility function programmed into a paperclip maximizer.

Secondly, I see how V could allow the coefficient for autonomy to be zero or even negative, in principle. But I don't think our values are actually that inconsistent. More precisely, if Value A and Value B require different choices with high frequency and have relatively similar priority, then U cannot implement both A and B unless it outputs very little of either one. Or so it seems to me.

As a toy model, let's say the friendly utility function U has a hundred valuable components - friendship, love, autonomy, etc... - assumed to have positive numeric values.

Values associated with what, computed given what parameters?

Just make the following part of the utility for the first couple of years or so: "Find out who defined your utility function. Extrapolate what they really meant and find out what they may have forgotten. Verify that you got that right. Adapt your utility function to be truer to its intended definition once you have confirmation."

This won't solve everything, but it seems like it should prevent the most obvious mistakes. The AI will be able to reason that autonomy was not intentionally left out but simply forgotten. It will then ask if it is right about that assumption and adapt itself.

I would like to argue that installing values in the first place is also robust, if done the right way.

Fragility is not intrinsic to value.

Value isn't fragile because value isn't a process. Only processes can be fragile or robust.

Winning the lottery is fragile, is a fragile process, because it had to be done all in one go. Consider the process of writing down a 12 digit phone number: if you to try to memorise the whole number, and then write it down, you are likely to make a mistake, due to Millers law, the one that says you can only hold five to nine items in short term memory. Writing digits down one at time, as you hear them, is more robust. Being able to ask for corrections, or having errors pointed out to you, is more robust still.

Processes that are incremental and involve error correction are robust, and can handle large volumes of data. The data aren't the problem: any volume of data can be communicated, so long as there is enough error correction. Trying to preload an AI with the total of human value is the problem, because it is the most fragile way of instilling human value.

MIRI favours the preloading approach because it allows proveable correctness, and provable correctness is an established technique for achieving reliable software in other areas: it's often used with embedded systems, critical systems, and so on.

But choosing that approach isn't a nett gain, because it entails fragility, and loss of corrigibility, and because it is less applicable to current real world AI. Current real world AI systems are trained rather than programmed: to be precise, they are programmed to be trainable.

Training is a process that involves error correction. So training implies robustness. It also implies corrigibility, because, corrigibility just is error correction. Furthermore, we know training is capable of instilling at least a good-enough level of ethics into an entity of at least human intelligence, because training instills good-enough ethics into most human children.

However that approach isn't a nett gain either. Trainable systems lack explicitness, in the sense that their goals are not coded in, but rather a feature that emerges and which are virtually impossible to determine by inspecting source side. Without the ability to determine behaviour from source code, they lack proveable correctness. On the other hand, their likely failure modes are more familiar to us...they are biomorphic, even if not anthropomorphic. They are less likely to present us with an inhuman threat, like the notorious paperclipper.

But surely a proveably safe system is better? Only if proveable really means proveable, if it implies 100% correctness. But a proof is a process, and one that can go wrong. A human can make a mistake. A proof assistant is a piece of software which is not magically immunized from having bugs. The difference between the two approach is not that the one is certain and the other not. One approach requires you to get something right first time, something which is very difficult, and which software engineers try to avoid where possible.

It is now beginning to look as though there never was a value fragility problem, aside from the decision to adopt the one-shot, preprogrammed, strategy. Is that right?

One of the things the value fragility argument was supposed to establish was that a "miss is as good as a mile". But humans despite not sharing precise values, regularly achieve a typical level of ethical behaviour with an imperfect grasp of each others values. Human value may be a small volume of valuespace, but it is still a fuzzy blob, not a mathematical point. ( And feedback, error correction, is part of how humans muddle along -- "would you mind not doing that, it annoys me" -- " sorry, I didn't realise")

Another concern is that an AI would need to understand the whole of human value order to create better world, a better future, it would need be friendly in the sense of adding value, not just refraining from subtracting value,

"Unfriendliness" is ambiguous: an unfriendly AI may be downright dangerous; or it might have enough grasp of ethics to be safe, but it enough to be able to make the world a much more fun place for humans. Unfriendliness in the second sense is not, strictly speaking a safety issue. One of the disadvantages of the Friendliness approach is that it makes it difficult to discuss the strategy of foregoing fun in order to achieve safety, of building boringly safe AIs.