What makes us think _any_ of our terminal values aren't based on a misunderstanding of reality?

I, for one, have "terminal value" for traveling back in time and riding a dinosaur, in the sense that worlds consistent with that event are ranked above most others. Now, of course, the realization of that particular goal is impossible, but possibility is orthogonal to preference.

The fact is, most things are impossible, but there's nothing wrong with having a general preference ordering over a superset of the set of physically possible worlds. Likewise, my probability distributions are over a superset of the actually physically possible outcomes.

When all the impossible things get eliminated and we move on like good rationalists, there are still choices to be made, and some things are still better than others. If I have to choose between a universe containing a bilion paperclips and a universe containing a single frozen dinosaur, my preference for ice cream over dirt is irrelevant, but I can still make a choice, and can still have a preference for the dinosaur (or the paperclips, whatever I happen to think is best).

I actually don't know what you even mean by my values dissolving, though. Sometimes I learn things that change how I would make choices. Maybe some day I will learn something that turns me into a nihilist such that I would prefer to wail about the meaninglessness of all my desires, but it seems unlikely.

But, how do we know that anything we value won't similarly dissolve under sufficiently thorough deconstruction?

Experience.

I was once a theist. I believed that people were ontologically fundamental, and that there was a true morality written in the sky, and an omniscient deity would tell you what to do if you asked. Now I don't. My values did change a little, in that they're no longer based on what other people tell me is good so I don't think homosexuality is bad and stuff like that, but it wasn't a significant change.

The only part that I think did change because of that was just that I no longer believed that certain people were a good authority on ethics. Had I not believed God would tell us what's right, I'm not sure there'd have been any change at all.

Learning more physics is a comparatively small change, and I'd expect it to correspond to a tiny change in values.

In regards to your Bob example, if I had his values, I'd expect that after learning that someone in the past is by definition not a future evolution of me, I'd change my definition to something closer to the "naive" definition, and ignore any jumps in time so long as the people stay the same when deciding of someone is a future evolution of me. If I then learn about timeless quantum physics and realize there's no such thing as the past anyway, and certainly not pasts that lead to particular futures, I'd settle for a world with a lower entropy, in which a relatively high number of Feynman paths reach here.

What makes us think any of our terminal values aren't based on a misunderstanding of reality?

Much the same thing that makes me think my height isn't based on a misunderstanding of reality. Different category. I didn't understand my way into having terminal values. Understanding can illuminate your perceptions of reality and allow you to better grasp what is, but I don't think that your terminal values were generated by your understanding. Trying to do so is a pathological tail biting exercise.

everything we want is as possible and makes as much sense as wanting to hear the sound of blue or taste the flavor of a prime number

We know it isn't because most of the time we get what we want. You want chocolate, so you go and buy some and then eat it, and the yummy chocolatey taste you experience is proof that it wasn't that futile after all for you to want chocolate.

The feeling of reward we get when we satisfy some of our terminal values is what makes us think that they aren't based on a misunderstanding of reality. So it's probably a pretty good bet to keep wanting at least the things that have led to rewards in the past, even if we aren't as sure about the rest of them, like going back in time.

Peter de Blanc wrote a paper on this topic.

I think this post is asking a very important and valuable question. However, I think it's limiting the possible answers by making some unnecessary and unjustified assumptions. I agree that Bob, as described, is screwed, but I think we are sufficiently unlike Bob that that conclusion does not apply to us.

As TheOtherDave says here,

I don't understand why you say "I want to travel back in time and ride a dinosaur" is meaningless. Even granting that it's impossible (or, to say that more precisely, granting that greater understanding of reality tends to sharply reduce its probability), how does that make it meaningless? You seem to offer "By definition, someone in Bob's past riding a dinosaur is not a future evolution of the present Bob" as an answer to that question, but that just completely confuses me. By definition of what, and why are we using that definition, and why is that important?

That was my reaction as well. In particular, "I want to go back in time and ride a dinosaur" is, on close inspection actually a rather vague value. It has many possible concrete interpretations and realizations. One more specific version of it is, "I want a physically future evolution of myself to ride a dinosaur in my physical past." As the post points out, this is impossible. But why privilege this particular interpretation of the goal?

You say that Bob wants not just the subjective experience but also objective fact of riding a dinosaur. If that's all he wants, then you're right, he's shit out of luck. I suspect though that we are not like Bob, and that our actual values are of the more vague sort with many possible realizations. And some of these will turn out to be meaningful and realizable and some won't be.

If that's so then the solution is to figure out which are the realizable interpretations of our goals and work towards those. I'm hopeful that this is not the empty set.

So, I'm basically ignoring the "terminal" part of this, for reasons I've belabored elsewhere and won't repeat here.

I agree that there's a difference between wanting to do X and wanting the subjective experience of doing X. That said, frequently people say they want the former when they would in fact be perfectly satisfied by the latter, even knowing it was the latter. But let us assume Bob is not one of those people, he really does want to travel back in time and ride a dinosaur, not just experience doing so or having done so.

I don't understand why you say "I want to travel back in time and ride a dinosaur" is meaningless. Even granting that it's impossible (or, to say that more precisely, granting that greater understanding of reality tends to sharply reduce its probability), how does that make it meaningless? You seem to offer "By definition, someone in Bob's past riding a dinosaur is not a future evolution of the present Bob" as an answer to that question, but that just completely confuses me. By definition of what, and why are we using that definition, and why is that important?

That said, as far as I can tell its meaninglessness is irrelevant to your actual point. The key point here is that if Bob knew enough about the world, he would give up on devoting resources to realizing his desire to go back in time and ride a dinosaur... right? I'm fine with assuming that; there are lots of mechanisms that could make this true, even if the whole "meaningless" thing doesn't quite work for me.

how do we know that anything we value won't similarly dissolve under sufficiently thorough deconstruction?

We don't know that. Indeed, I expect most of our values are extremely fragile, not to mention mutually opposed. Anything resembling a process of "calculating my coherent extrapolated volition" will, I expect, produce a result that I am as likely to reject in horror, or stare at in bewilderment, as I am to embrace as valuable. (Add another seven billion minds' values to the mix and I expect this becomes somewhat more true, but probably not hugely more true.)

Let's suppose for a minute that all "human values" are dangling units [..] What is the rational course of action in such a situation?

The rational course of action for an agent is to optimally pursue its values.
That is, keep attempting to XXX anyway.
What privileges XXX over other alternatives is that the agent values XXX more than those alternatives.
There's no precommitment involved... if the agents values change such that it no longer values XXX but rather YYY, at that point it ought to stop pursuing XXX and pursue YYY instead. It may well regret having previously pursued XXX at that time. It may even predict its later regret of pursuing XXX, and its rational course of action is still to pursue XXX.
Of course, if it happens to value its future value-satisfaction, then YYY is part of XXX to the extent that the later value-shift is expected.

More generally: you seem to want your values to be justified in terms of something else.
Do you have any coherent notion of what that "something else" might be, or what properties it might have?

All our values are fallible, but doubt requires justification.

You're right that a meaningless goal cannot be pursued, but nor can you be said to even attempt to pursue it - i.e., the pursuit of a meaningless goal is itself a meaningless activity. Bob can't put any effort into his goal of time travel, he can only confusedly do things he mistakenly thinks of as "pursuing the goal of time travel", because pursuing the goal of time travel isn't a possible activity. What Bob has learned is that he wasn't pursuing the goal of time travel to begin with. He was altogether wrong about having a terminal value of travelling back in time and riding a dinosaur because there's no such thing.

When you said to suppose that "everything we want is [impossible]", did you mean that literally? Because normally if what you want is impossible, you should start wanting a different thing (or do that super-saiyan effort thing if it's that kind of impossible), but if everything is impossible, you couldn't do that either. If there is no possible action that produces a favorable outcome, I can think of no reason to act at all.

(Of course, if I found myself in that situation, I would assume I made a math error or something and start trying to do things that I want and that I think I messed up when I decided that the thing was impossible.)

If you didn't mean -everything-, then why not just start pursuing the thing which gives the most value which is possible to do?

Perhaps I misunderstood the question?

Another way to think about Dave's situation is that his utility function assigns the same value to all possible futures (i.e. zero) because the one future that would've been assigned a non-zero value turned out to be unrealizable. His real problem is that his utility function has very little structure: it is zero almost everywhere.

I suspect our/my/your utility function is structured in a way that even if broad swaths of possible futures turn out to be unrealizable, the remainder will still contain gradients and local maxima, so there will be some more desirable and some less desirable possibilities.

Of course this is not guaranteed, but most utility functions have gradients and local maxima over most sets. You need a very special utility function and a very special set of realizable futures in order for all futures to be assigned exactly the same value.

This comment by Wei Dai might be relevant, also see steven0461's answer.

No matter what the universe is, all you need for casual decision theory is that you live in a universe in which your actions have consequences, and you prefer some of the possible consequences over others. (you can adjust and alter this sentence for your preferred decision theory)

What if that doesn't happen? What if you didn't prefer any consequence over any other, and you were quite certain no action you took would make any difference to anything that mattered?

Well, it's not a trick question ... you'll just act in any arbitrary way. It won't matter. All actions would be equally rational.

The original Bob would not want to be this Bob.

That implies that the original Bob values ignorance/working towards goals/etc in addition to riding on dinosaurs in the past. Bob only needs one coherent value to have a reason to take certain actions.

The statement "The original Bob would not want to be this Bob." doesn't follow from the premise "Bob's terminal value is not only utterly impossible but meaningless". If that was really the only thing Bob valued, then the original Bob would be utterly neutral to becoming the enlightened Bob, since what happens to Bob and what Bob learns doesn't matter. All possible futures are equally preferred, since none of them are closer to bringing Bob travelling to the past and riding a dinosaur.

What is the rational course of action in such a situation?

Being able to cast off self-contradictions (A is equal to negation-of-A) is as close as I can offer to a knowable value that won't dissolve. But I may be wrong, depending on what you mean by sufficient deconstruction. If the deconstruction is sufficient, it is sufficient, and therefore sufficient, and you've answered your own question: we cannot know. Which leads to the self-contradiction that we know one thing and that is we cannot know any thing including that we cannot know anything.

Self-contradictions (and infinite regressions) suggest a problem with explanations, not what is explained. The rational course of action is to not confuse the world with a map of the world (in this case, rationality) even if the world does contain that map. The map is entirely inside the world, the world is not entirely inside the map.

More specifically, I spent a little time today answering this question and several hours interpreting a classroom for a deaf student. If you reverse those priorities, you get caught up in maps and don't do much to enjoy or improve this nice territory.

I live my life under the assumption that I do have achievable values. If I had no values that I could achieve and I was truly indifferent between all possible outcomes, then my decisions do not matter. I can ignore any such possible worlds in my decision theory.

My terminal value is my own happiness. I know that it exists because I have experienced it, and experience it regularly. I can't imagine a world in which someone convinces me that I don't experience something that I experience.

Lukeprog's metaethics posts went over this - so how about getting the right answer for him? :)

Terminal values are part of the map, not the territory.

Heh. It's even worse than that. The idea that Bob is a single agent with terminal values is likely wrong. There are several agents comprising Bob and their terminal values change constantly, depending on the weather.

An agent optimized to humanity's CEV would instantly recognize that trying to skip ahead would be incredibly harmful to our present psychology; without dreams—however irrational—we don't tend to develop well in terms of CEV. If all of our values break down over time, a superintelligent agent optimized for our CEV will plan for the day our dreams are broken, and may be able to give us a helping hand and a pat on the back to let us know that there are still reasons to live.

This sounds like the same manner of fallacy associated with determinism and the ignorance of the future being derived from the past though the present rather than by a timeless external "Determinator."