Objections to Coherent Extrapolated Volition

If you already knew a movie, you wouldn’t watch it.

This example, and a few others in your post that follow the general pattern of "If you already knew X, then you would have no volition to go and learn X", don't apply to CEV as I understand it. The hypothetical extrapolated better-faster-smarter future yous are not being asked "What do you want now?", they're being asked "What should the past primitive-slow-dumb-you want?"

I might well advise someone to see a movie that I enjoyed, even if I have no particular desire to watch the movie again myself.

You might conclude that what the hunter-gatherer really wants is to solve abstract mathematical problems, he just doesn’t know it. But there is no set of values that a person “really” wants. Humans are largely defined by the circumstances they reside in.

At which point, you should ask, what general principles do (should) they want to use in order to decide what to do depending on the circumstances. Dependence on a lot of parameters is not chaos, not absence of a fact, it's merely a more difficult problem where all the different cases should be addressed, rather than just a single configuration, a blind answer that doesn't distinguish.

Take for example an agent that is facing the Prisoner’s dilemma. Such an agent might originally tend to cooperate and only after learning about game theory decide to defect and gain a greater payoff. Was it rational for the agent to learn about game theory, in the sense that it helped the agent to achieve its goal or in the sense that it deleted one of its goals in exchange for a allegedly more “valuable” goal?

The agent's goals aren't changing due to increased rationality, but just because the agent confused him/herself. Even if this is a payment-in-utilons and no-secondary-consequences Dilemma, it can still be rational to cooperate if you expect the other agent will be spending the utilons in much the same way. If this is a more down-to-earth Prisoner's Dilemma, shooting for cooperate/cooperate to avoid dicking over the other agent is a perfectly rational solution that no amount of game theory can dissuade you from. Knowledge of game theory here can only change your mind if it shows you a better way to get what you already want, or if you confuse yourself reading it and think defecting is the 'rational' thing to do without entirely understanding why.

You describe a lot of goals as terminal that I would describe as instrumental, even in their limited context. While it's true that our ideals will be susceptible to culture up until (if ever) we can trace and order every evolutionary desire in an objective way, not many mathematicians would say "I want to determine if a sufficiently-large randomized Conway board would converge to an all-off state so I will have determined if a sufficiently-large randomized Conway board would converge to an all-off state". Perhaps they find it an interesting puzzle or want status from publishing it, but there's certainly a higher reason than 'because they feel it's the right thing to do'. No fundamental change in priorities needs occur between feeding one's tribe and solving abstract mathematical problems.

I won't extrapolate my arguments farther than this, since I really don't have the philosophical background it needs.

Instead of thinking about a hypothetical human hunter, I find it useful to think about the CEV for dogs, or for a single dog. (Obviously the CEV for a single dog is wildly different from the CEV for all dogs, but the same types of observations emerge from either.)

I think it would be pretty straightforward to devise a dog utopia. The general features of dog values are pretty apparent and seem very simple to humans. If our technology were a bit more advanced, a study of dog brains and dog behaviors would tell us enough to design a virtual universe of dog bliss.

We are so much smarter than dogs that we could even create ways of intervening into inter-dog conflicts in ways not obvious to the dogs. We could remove the whole mentality of a dominance hierarchy from the dog's psychology and make them totally egalitarian. Since these dogs would be immortal and imbued with greater intelligence, they could be taught to enjoy more complex pleasures and possibly even to generate art.

Of course none of what I just described is actually dog CEV. It is more like what a human thinks human CEV might look like, applied to dogs in a lazy fashion. It is not Coherent in the sense that it is ad hoc, nor is it Extrapolated, in the sense that it essentially disregards what dogs actually want - in this case, to be the alpha dog, and to be perpetually gorging on raw meat.

Still - STILL - the dogs probably wouldn't complain if we imposed our humanized-CEV onto them, after the fact. At least, they wouldn't complain unless we really messed it up. There is probably an infinite space of stable, valid Utopias that human beings would willingly choose and would be perpetually content with. The idea that human CEV is or should be one single convergent future does not seem obviously true to me. Maybe CEV should be a little bit lazy on purpose, as the above human design designed a lazy but effective dog utopia.

My main point here is that this singleton-attractor phenomenon really becomes a problem when the CEV subject and the CEV algorithm become too tightly coupled. It seems to be generally assumed that CEV will be very tightly coupled to human desires. Maybe there should be a bit of wiggle room.

If you already knew a movie, you wouldn’t watch it.

Indeed? May I suggest reading http://www.wired.com/wiredscience/2011/08/spoilers-dont-spoil-anything/ (PDF) ?

Much of our values and goals, what we want, are culturally induced or the result of our ignorance. Reduce our ignorance and you change our values. One trivial example is our intellectual curiosity. If we don’t need to figure out what we want on our own, our curiosity is impaired.

I don't follow this one. Is this just making the argument-by-definition that an omniscient being couldn't be curious? The universe seems to place hard limits on how much computation can be done and storage accessed, so there will always be things a FAI will not know. (I can also appeal to more general principles here: Godel, Turing, Hutter or Legge's no-elegant-predictor results, etc.)

Take for example an agent that is facing the Prisoner’s dilemma. Such an agent might originally tend to cooperate and only after learning about game theory decide to defect and gain a greater payoff. Was it rational for the agent to learn about game theory, in the sense that it helped the agent to achieve its goal or in the sense that it deleted one of its goals in exchange for a allegedly more “valuable” goal?

Er, what? If the agent isn't reaping greater payoffs then it was simply mistaken (that happens sometimes) and can go back to not cooperating. If it had defecting as an intrinsically good thing, then why did it ever start cooperating?

It seems to me that becoming more knowledgeable and smarter is gradually altering our utility functions.

If this is the basic point, you're missing a lot of more germane results than what you put down. Openness and parasite load (or psilocybin), IQ and cooperation and taking normative economics stances (besides the linked cites, Pinker had a ton of relevant stuff in the later chapters of Better Angels), etc.

My counter thought experiment to CEV is to consider our distant ancestors. I mean so far distant that we wouldn't call them human, maybe even as far back as some sort of fish-like creature. Suppose a super AI somehow offered this fish the chance to rapidly "advance", following its CEV and it showed it a vision of the future, us, and asked the fishy thing whether to go ahead. Do you think the fishy thing would say yes?
Similarly, if an AI offered to evolve humankind, in 50 years, into telepathic little green men that it assured us was the result of our CEV, would we not instantly shut it down in horror?
My personal preference, I like to call the GFP - Glorious Five-year Plan: You have the AI offer a range of options for 5 (or 50 but definitely no longer) years in the future, and we pick one. And in 5 years time we repeat the process. The bottom line is that humans do not want rapid change. Just we are happier with 2% inflation than 0% or 100%, we want a moderate rate of change. 
At its heart there is a "Ship of Theseus" problem. If the AI replaces every part of the ship overnight so that in the morning we find the QE2 at dock then it is not the ship of Theseus.

Couple minor nitpicks:

Consider the difference between a hunter-gatherer, who cares about his hunting success and to become the new tribal chief

Not too many foragers have chiefs, unless they're living in such opulence that they can settle down (see the very wikipedia article you linked and comments about social structures).

If you already knew a movie, you wouldn’t watch it.

So why do people re-watch movies? Or watch movies to which they already know the spoiler or premise?

Humans are largely defined by the circumstances they reside in.

This reminds me of an objection I once came up with about the idea that there's no reason to expect that your utility function for money should have an inflection point/some other special property at or near your current net worth. I think that's a not-so-unreasonable assumption indeed, if you are used to that level of wealth (i.e., unless you've recently gained or lost a huge lot of money): I think that someone who grew up in a wealthy family and stayed wealthy until recently would find it much harder to live on $1000 a month than someone who has been that poor all along, as the former would find it very hard to imagine life without [insert expensive pastime here], the latter would know more ways to cope with poverty (e.g. where to buy cheap clothes), etc.

An awareness of the context-sensitivity of our desires only leads me to value novelty (diversity on a personal-experience level). Not only do I expect some drift, I intend to enjoy it.

I haven't thought about how I'd like protection against black-holes of desire to be implemented. It's easy to imagine some hypothetical future-me resenting a growing novelty-itch I committed to.

Since I expect my preference for novelty to inform CEV, if you're making an argument against CEV that might persuade me, it must be that the optimization problem is too hard - that the result will either deny unexpected pleasures that could have otherwise been reached, or that the pull of expected pleasures will be stronger than expected and lead to a fixed state (violating our present belief in our preference for novelty).

It seems to me that becoming more knowledgeable and smarter is gradually altering our utility functions.

That's probably down to memes and memetic hijacking. Advanced civilisation results in a greater density of memes, and they have greater say in civilisation's values. These values usually include reduced reproduction for humans - because of the resource competition between memes and genes - thus the demographic transition.

But what is it that we are approaching if the extrapolation of our volition becomes a purpose in and of itself?

As I say, the actual effect is probably down to more memes. If that continues, we'll probably get this.

It reminds me of a story I've read, where someone from an early-21st-century capitalist culture goes back in time and tells a few Cro-Magnon hunters and gatherers what wonders the future will contain, and they (very convincingly) argue that they are no overall improvement at all. (Of course, there are many more people alive today than Earth could support if agriculture hadn't been invented, so a total utilitarian would disagree with them.)

If you already knew a movie, you wouldn’t watch it.

The usual framework seems to not apply here: IIRC there's some theorem showing that the value of information cannot be negative, but that seems obviously false if the information in question is the ending of a film you've already paid a ticket for.