Posts
Comments
I think this post is basically correct. You don't, however, give an argument that most minds would behave this way. However, here is a brief intuitive argument for it. A "utility function" does not mean something that is maximized in the ordinary sense of maximize; it just means "what the thing does in all situations." Look at computers: what do they do? In most situations, they sit there and compute things, and do not attempt to do anything in particular in the world. If you scale up their intelligence, that will not necessarily change their utility function much. In other words, it will lead to computers that mostly sit there and compute, without trying to do much in the world. That is to say, AIs will be weakly motivated. Most humans are weakly motivated, and most of the strength of their motivation does not come from intelligence, but from the desires that came from evolution. Since AIs will not have that evolution, they will be even more weakly motivated than humans, assuming a random design.
Part of the problem with the usual LW position on this is that it is based on two mistakes:
1) Eliezer's mistaken idea that good and evil are arbitary in themselves, and therefore to be judged by human preference alone.
2) Eliezer's excessive personal preference for life (e.g. his claim that he expects that his extrapolated preference would accept the lifespan dilemma deal, even though such acceptance guarantees instant death.)
These two things lead him to judge the matter by his excessive personal preference for life, and therefore to draw the erroneous conclusion that living forever is important.
Good and evil are not arbitrary, and have something to do with what is and what can be. In particular, what cannot be, cannot be good. But living forever cannot be. Therefore living forever is not good, and should not be desired. In a sense this is similar to saying that hoping to win the lottery is a waste of hope, because you won't actually win. The difference is that it is at least possible to win the lottery, whereas it is entirely impossible to live forever.
This theory is mostly true, but rather than being cynical about people caring about poor people, we should be cynical about the more general concept of people caring about stuff.
This is all basically right.
However, as I said in a recent comment, people do not actually have utility functions. So in that sense, they have neither a bounded nor an unbounded utility function. They can only try to make their preferences less inconsistent. And you have two options: you can pick some crazy consistency very different from normal, or you can try to increase normality at the same time as increasing consistency. The second choice is better. And in this case, the second choice means picking a bounded utility function, and the first choice means choosing an unbounded one, and going insane (because agreeing to be mugged is insane.)
You don't have any such basic list.
I don't think you understood the argument. Let's agree that an electron prefers what it is going to do, over what it is not going to do. But does an electron in China prefer that I write this comment, or a different one?
Obviously, it has no preference at all about that. So even if it has some local preferences, it does not have a coherent preference over all possible things. The same thing is true for human beings, for exactly the same reasons.
I don't know why you think I am assuming this. Regardless of the causes of your opinions, one thing which is not the cause is a coherent set of probabilities. In the same way, regardless of the causes of your actions, one thing which is not the cause is a coherent set of preferences.
This is necessarily true since you are built out of physical things which do not have sets of preferences about the world, and you follow physical laws which do not have sets of preferences about the world. They have something similar to this, e.g. you could metaphorically speak as if gravity has a preference for things being lower down or closer together. But you cannot compare any two arbitrary states of the world and say "Gravity would prefer this state to that one." Gravity simply has no such preferences. In a similar way, since your actions result from principles which are preference-like but not preferences, your actions are also somewhat preference-like, but they do not express a coherent set of preferences.
All that said, you are close to a truth, which is that since the incoherence of people's lives bothers them (both in thoughts and in actions), it is good for people to try to make those both more coherent. In general you could make incoherent thoughts and actions more coherent in two different directions, namely "more consistent with themselves but less consistent with the world" and "more consistent with themselves and also more consistent with the world". The second choice is better.
" It's a list of all our desires and preferences, in order of importance, for every situation ."
This is basically an assertion that we actually have a utility function. This is false. There might be a list of pairings between "situations you might be in" and "things you would do," but it does not correspond to any coherent set of preferences. It corresponds to someone sometimes preferring A to B, and sometimes B to A, without a coherent reason for this.
Asserting that there is such a coherent list would be like asserting that you have a list of probabilities for all statements that are based on a coherent prior and were coherently updated in their current state using Bayesian updating. This is nonsense: there is no such thing as "the actual probability that you really truly assign to the claim that you are about to change your name to Thomas John Walterson and move to Australia." You never thought of that claim before, and although you think it very unlikely when you think of it, this is not intrinsically numerical in any way. We assign probabilities by making them up, not by discovering something pre-existent.
In exactly the same way, coherent sets of preferences are made up, not discovered.
I predict the video was faked (i.e. that everyone in it knows what is happening and that in fact there was not even a test like this.)
Most people, most of the time, state their beliefs as binary propositions, not as probability statements. Furthermore, this is not just leaving out an actually existing detail, but it is a detail missing from reality. If I say, "That man is about 6 feet tall," you can argue that he has an objectively precise height of 6 feet 2 inches or whatever. But if I say "the sky is blue," it is false that there is an objectively precise probability that I have for that statement. If you push me, I might come up with the number. But I am basically making the number up: it is not something that exists like someone's height.
In other words, in the way that is relevant, beliefs are indeed binary propositions, and not probability statements. You are quite right, however, that in the process of becoming more consistent, you might want to approach the situation of having probabilities for your beliefs. But you do not currently have them for most of your beliefs, nor does any human.
This seems obviously circular, since you depend on using induction based on human languages to conclude that humans were produced by an evolutionary process.
"1) the paperclip maximizer is not a paperclip maximizer but a different kind of unfriendly AI"
Being a paperclip maximizer is about values, not about decision theory. You can want to maximize paperclips but still use some of acausal decison theory that will cooperate with decision makers that would cooperate with paperclippers, as in cousin_it's response.
It's curious that you assume in this discussion that the one is right, and the other is wrong. The opposite happens with equal frequency (this is mathematically necessary, since each is one and each is other.)
" Similarly, this argument for averaging could only make sense if the other person would consider averaging beliefs with you for the same reason. "
No. This is exactly the situation where it would not make sense. If the person has already averaged his belief with others they met, you will end up with a distorted belief. It is percisely when you have two "natural" beliefs that averaging them offers some potential gain.
" I don't see how that conjunction is logically inconsistent?" Suppose you have beliefs A, B, C, and belief D: "At least one of beliefs A, B, C is false." The conjunction of A, B, C, and D is logically inconsistent. They cannot all be true, because if A, B, and C are all true, then D is false, while if D is true, at least one of the others is false. So if you think that you have some false beliefs (and everyone does), then the conjunction of that with the rest of your beliefs is logically inconsistent.
" I think consistency is good. " I agree.
" A map that is not consistent with itself cannot be used for the purposes of predicting the territory." This is incorrect. It can predict two different things depending on which part is used, and one of those two will be correct.
" And inconsistent map (especially one where the form and extent of inconsistency is unknown (save that the map is inconsistent)) cannot be used for inference." This is completely wrong, as we can see from the example of recognizing the fact that you have false beliefs. You do not know which ones are false, but you can use this map, for example by investigating your beliefs to find out which ones are false.
" An inconsistent map is useless. " False, as we can see from the example.
" I don't want consistency because consistency is desirable in and of itself—I want consistency because it is useful. " I agree, but I am pointing out that it is not infinitely useful, and that truth is even more useful than consistency. Truth (for example "I have some false beliefs") is more useful than the consistent but false claim that I have no false beliefs.
" An example please? I cannot fathom a reason to possess inconsistent preferences." It is not a question of having a reason to have inconsistent preferences, just as we were not talking about reasons to have inconsistent beliefs as though that were virtuous in itself. The reason for having inconsistent beliefs (in the example) is that any specific way to prevent your beliefs from being inconsistent will be stupid: if you arbitrarily flip A, B, or C, that will be stupid because it is arbitrary, and if you say "all of my beliefs are true," that will be stupid because it is false. Inconsistency is not beneficial in itself, but it is more important to avoid stupidity. In the same way, suppose there is someone offering you the lifespan dilemma. If at the end you say, "Nope, I don't want to commit suicide," that will be like saying "some of my beliefs are false." There will be an inconsistency, but getting rid of it will be worse.
(That said, it is even better to see how you can consistently avoid suicide. But if the only way you have to avoid suicide is an inconsistent one, that is better than nothing.)
" Consistent preferences are strictly more useful than inconsistent preferences." This is false, just as in the case of beliefs, if your consistent preferences lead you to suicide, and your inconsistent ones do not.
" If you are inconsistent, then please fix that. :) "
G.K. Chesterton made a lot of errors which he always managed to state in interesting ways. However, one thing he got right was the idea that a lot of insanity comes from an excessive insistence on consistency.
Consider the process of improving your beliefs. You may find out that they have some inconsistencies between one another, and you might want to fix that. But it is far more important to preserve consistency with reality than interal consistency, and an inordinate insistence on internal consistency can amount to insanity. For example, the conjunction of all of your beliefs with "some of my beliefs are false" is logically inconsistent. You could get more consistency by insisting that "all of my beliefs are true, without any exception." But that procedure would obviously amount to an insane arrogance.
The same sort of thing happens in the process of making your preferences consistent. Consistency is a good thing, but it would be stupid to follow it off a cliff. That does not mean that no possible set of preferences would be both sane and consistent. There are surely many such possible sets. But it is far more important to remain sane than to adopt a consistent, but insane, set of preferences.
"arbitrary limit"
What would constitute a non-arbitrary limit? A utility function just describes the behavior of a physical being. It is not surprising that a physical being has limits in its behavior -- that follows from the fact that physical law imposes limits on what a thing does. This is why e.g. transhumanist hope for real immortality is absurd. Even if you could find a way around entropy, you will never change the fact that you are following physical laws. The only way you will exist forever is if current physical laws extrapolated from your current situation imply that you will exist forever. There is no reason to think this is the case, and extremely strong reasons to think that it is not.
In the same way, everything I do is implied by physical laws, including the fact that I express a preference for one thing rather than another. It may be that a good conman will be able to persuade some people to order their preferences in a way that gets them to commit suicide (e.g. by accepting the lifespan offer), but they will be very unlikely to be able to persuade me to order my preferences that way. This is "arbitrary" only in the sense that in order for this to be true, there have to be physical differences between me and the people who would be persuaded that in theory neither of us is responsible for. I don't have a problem with that. I still don't plan to commit suicide.
" Also the lifespan dilemma shows that for many people the answer can’t be just a matter of expected value, otherwise everyone would agree on reducing the probability of success to values near to 0. "
This is a delusion that stems from using the formulation of "expected value" without understanding it. The basic idea of expected value derives from a utility function, which is the effect of being able to give consistent answers to every question of the form, "Would you prefer X to Y, or Y to X? Or does it just not matter?" Once you have such a set of consistent answers, a natural way to use a real numbering measuring system falls out, which is such that something is called "twice as good" when it just doesn't matter whether you get a 50% chance of the "twice as good" thing, or 100% chance of the "half as good" thing.
But this idea of "twice as good" is just defined in this way because it works well. There is no reason whatsoever to assume "twice as much stuff by some other quantitative measure" is twice as good in this sense; and the lifespan dilemma and all sorts of other things definitively proves that it is not. Twice as much stuff (in the measurement sense) is just not twice as good (in the expected value sense), period, not even when you are talking about lifespan or lives or whatever else.
In this sense, if you can give consistent answers to every pair of preference options, you will want to maximize expected utility. There is no reason whatsoever to think you will be willing to drive the probability of success down to values near zero, however, since there is no reason to believe that value of lifespan scales in that way, and definitive reason to believe otherwise.
" that doesn't make their strongest beliefs the most trustworthy -- indeed, they're the ones we ought to downgrade the most"
Not likely. Their strongest beliefs will be the most trustworthy, even though they are downgraded the most, because they start start out higher. It would be a very unlikely calibration graph indeed which assigned a lower probability to 99.9% assigned probabilities than to 95% assigned probabilities.
Great post. I think Robin Hanson would accept that "older people are wiser" should tend to be true, even in relation to himself, but he also accepts that number 9 applies to himself, so it wouldn't bother him that he hasn't corrected his beliefs in particular ways yet.
I agree with most of this post, and in fact my recent posts (at my blog, not here) imply something similar. But I think there is a mistaken idea in this particular statement: "We want a safe way to resolve the under-determination in human values, a task that gets more and more difficult as we move away from the usual world of today and into the hypothetical world that a superpowered AI could build."
It looks like you are saying that we need a way to make sure that the future, however distant, will always be somewhat acceptable to current humans. But this is impossible in principle, given the fact that things are tending towards the heat death of the universe. What we actually should want is that the universe should move at any particular time towards things that the beings in existence value at that time. Obviously creatures in the future will have different values, and given a long enough time period, a future will therefore come into existence that we as we are would have no particular interest in. But we also should no particular interest in preventing it from coming into being; that interest comes from a positively unreasonable extrapolation of your current interests.
I agree with this, although it might not work for some theoretically possible games that humans would not actually play.
Life in the real world, however, is not a perfect information zero-sum game, or even an approximation of one. So there is no reason to suppose that the techniques use will generalize to a fooming AI.
(Obviously I meant if you are not Chinese and have not been in China.)
"However, if even with the human special sauce we're to expect AGI capabilities to be slow, domain-specific, and requiring feed-in from a big market ecology, then the situation we see without human-equivalent generality special sauce should not look like this. "
Actually, most of what humans know is from other humans. This is not unrelated to generality: you cannot deduce facts about China from your experiences if you exclude what you have heard and read from other humans. Any general AI will be in the same situation. Most of what they know will have to come from the testimony of human beings. This is unlike the situation with Go, where everything is already available by considering the board and the moves.