Values Weren't Complex, Once.

It is not obvious that "value" is synonymous with "moral value", and it is no clear that divergence in values is a moral issue, or necessarily leads to conflict. Food preferences are the classic example of preferences that can be satisfied in a non-exclusionary way.

My favorite analogy is that of an automated control system, like a radio receiver. The main goal is to lock onto the station and maximize the signal fidelity, but once it's done, there is always the noise of the locking algorithm steering around the optimal point. The drive to optimize doesn't disappear, but ends up cranking up the gain on this residual optimization noise and trying to find something else to optimize. In psychology this is phenomenologically described as the Maslow's hierarchy. This (misapplied) drive to optimize creates this complexity, because it is trying to work on its own internal optimization algorithms, leading to a lot of non-linearity and spurious chaotic behavior. Hmm, wonder how hard it would be to model this mathematically or numerically.

Yeah, if you fulfill people's material desires, their remaining unfulfilled desires will be more conflicting. But the picture you paint - that "basic desires run out", and what remains is noise and irrelevance - doesn't look right to me.

Rather, what remains is mostly positional (status) desires. These are just as basic and universal as wanting food or comfort. And they're still relevant to the original goal, increasing the relative (zero-sum) frequency of your genes. That's why these desires are hard to fulfill for everyone, not because of noise or irrelevance. This old post by Phil Goetz [LW · GW] explains the idea and takes it further.

Have you read the Blue-Minimizing Robot [LW · GW]? Early Homo sapiens was in the simple environment where it seemed like they were "minimizing blue," i.e. maximizing genetic fitness. Now, you might say, it seems like our behavior indicates preferences for happiness, meaning, validation, etc, but really that's just an epiphenomenon no more meaningful than our previous apparent preference for genetic fitness.

However, there is an important difference between us and the blue-minimizing robot, which is that we have a much better model of the world, and within that model of the world we do a much better job than the robot at making plans. What kind of plans? The thing that motivates our plans is, from a purely functional perspective, our preferences. And this thing isn't all that different in modern humans versus hunter-gatherers. We know, we've talked to them. There have been some alterations due to biology and culture, but not as much as there could have been. Hunter-gatherers still like happiness, meaning, validation, etc.

What seems to have happened is that evolution stumbled upon a set of instincts that produced human planning, and that in the ancestral environment this correlated well with genetic fitness, but in the modern environment this diverges even though the planning process itself hasn't changed all that much. There are certain futuristic scenarios that could seriously disrupt the picture of human values I've given, but I don't think it's the default, particularly if there aren't any optimization processes much stronger than humans running around.