Hedonic asymmetries

post by paulfchristiano · 2020-01-26T02:10:01.323Z · score: 84 (30 votes) · LW · GW · 17 comments

Contents

  First: some other explanations
  Main argument: reward errors are asymmetric
  Implications
None
17 comments

Automatically crossposted

Creating really good outcomes for humanity seems hard. We get bored. If we don’t get bored, we still don’t like the idea of joy without variety. And joyful experiences only seems good if they are real and meaningful (in some sense we can’t easily pin down). And so on.

On the flip side, creating really bad outcomes seems much easier, running into none of the symmetric “problems.” So what gives?

I’ll argue that nature is basically out to get us, and it’s not a coincidence that making things good is so much harder than making them bad.

First: some other explanations

Two common answers (e.g. see here and comments):

I think both of these are real, but that the consideration in this post is at least as important.

Main argument: reward errors are asymmetric

Suppose that I’m building an RL agent who I want to achieve some goal in the world. I can imagine different kinds of errors:

Pessimistic errors are no big deal. The agent will randomly avoid behaviors that get penalized, but as long as those behaviors are reasonably rare (and aren’t the only way to get a good outcome) then that’s not too costly.

But optimistic errors are catastrophic. The agent will systematically seek out the behaviors that receive the high reward, and will use loopholes to avoid penalties when something actually bad happens. So even if these errors are extremely rare initially, they can totally mess up my agent.

When we try to create suffering by going off distribution, evolution doesn’t really care. It didn’t build the machinery to be robust.

But when we try to create incredibly good stable outcomes, we are fighting an adversarial game against evolution. Every animal forever has been playing that game using all the tricks it could learn, and evolution has patched every hole that they found.

In order to win this game, evolution can implement general strategies like boredom, or an aversion to meaningless pleasures. Each of these measures makes it harder for us to inadvertently find a loophole that gets us high reward.

Implications

Overall I think this is a relatively optimistic view: some of our asymmetrical intuitions about pleasure and pain may be miscalibrated for a world where we are able to outsmart evolution. I think evolution’s tricks just mean that creating good worlds is difficult rather than impossible, and that we will be able to create an incredibly good world as we become wiser.

It’s possible that evolution solved the overoptimism problem in a way that is actually universal—such that it is in fact impossible to create outcomes as good as the worst outcomes are bad. But I think that’s unlikely. Evolution’s solution only needed to be good enough to stop our ancestors from finding loopholes, and we are a much more challenging adversary.

17 comments

Comments sorted by top scores.

comment by DanArmak · 2020-01-26T17:40:25.793Z · score: 10 (6 votes) · LW(p) · GW(p)
Pessimistic errors are no big deal. The agent will randomly avoid behaviors that get penalized, but as long as those behaviors are reasonably rare (and aren’t the only way to get a good outcome) then that’s not too costly.

Also, if an outcome really is very bad, evolution has no reason to limit the amount of suffering experienced.

Getting burned is bad for you. Evolution makes it painful so you know to avoid it. But if strong and extreme pain result in the same amount of avoidance, evolution has no reason to choose "strong" over "extreme". In fact, it might prefer "extreme" to get a more robust outcome.

And that's how we get the ability to inflict the kind of torture which is 'worse than death', and to imagine a Hell (and simulations thereof) with infinite negative utility, even though evolution doesn't have a concept of a fate being "worse than death" - certainly not worse than the death of your extended family.

comment by G Gordon Worley III (gworley) · 2020-01-28T02:03:21.887Z · score: 5 (4 votes) · LW(p) · GW(p)
We get bored. If we don’t get bored, we still don’t like the idea of joy without variety.

For what it's worth I think this popular sentiment is misplaced. Joy without variety is just as good as joy with variety, but most people either never experience it or fail to realize they are experiencing it and so don't learn that joy without variety is equally good to joy with variety. Instead I think most people experience rare moments of peak joy and since they don't understand how they happened they come to believe that joy comes from variety itself rather than anywhere else.

comment by Pattern · 2020-01-28T18:40:14.723Z · score: 1 (1 votes) · LW(p) · GW(p)

Perhaps hobbies are areas where people understand this about themselves, albeit narrowly.

comment by Dagon · 2020-01-26T18:19:07.316Z · score: 4 (3 votes) · LW(p) · GW(p)

Hedonic adaptation (feeling reward/penalty for the relative change, much more than the absolute situation) may be a key strategy for this. It adjusts both upward and downward, to avoid either mistake for very long.

comment by masha · 2020-01-31T19:48:37.811Z · score: 3 (2 votes) · LW(p) · GW(p)
If we don’t get bored, we still don’t like the idea of joy without variety. And joyful experiences only seems good if they are real and meaningful

I don't understand what you mean about not liking the idea of joy without variety. Do you mean that people don't want to constantly feel joy, that they would rather feel a range of different emotions, including the unpleasant ones? This is not true for me personally.

Also, why do joyful experiences need to be real or meaningful? I think there is meaning in pleasure itself. Perhaps the most joyful experiences you have had were experiences that were "real and meaningful", so you have come to see joy as being inextricably connected to meaningfulness. Throughout evolution, this was probably true. But with modern technology, and the ability to biohack our neurochemistry, this association is no longer a given.

comment by edoarad · 2020-01-27T13:20:41.432Z · score: 3 (3 votes) · LW(p) · GW(p)
Pessimistic errors are no big deal. The agent will randomly avoid behaviors that get penalized, but as long as those behaviors are reasonably rare (and aren’t the only way to get a good outcome) then that’s not too costly.

But optimistic errors are catastrophic. The agent will systematically seek out the behaviors that receive the high reward, and will use loopholes to avoid penalties when something actually bad happens. So even if these errors are extremely rare initially, they can totally mess up my agent.

I'd love to see someone analyze this thoroughly (or I'll do it if there will be an interest). I don't think it's that simple, and it seems like this is the main analytical argument.

For example, if the world is symmetric in the appropriate sense in terms of what actions get you rewarded or penalized, and you maximize expected utility instead of satisficing in some way, then the argument is wrong. I'm sure there is good literature on how to model evolution as a player, and the modeling of the environment shouldn't be difficult.

comment by Dagon · 2020-01-27T18:13:35.595Z · score: 3 (2 votes) · LW(p) · GW(p)

I suspect it all comes down to modeling of outcome distributions. If there's a narrow path to success, then both biases are harmful. If there are a lot of ways to win, and a few disasters, then optimism bias is very harmful, as it makes the agent not loss-averse enough. If there are a lot of ways to win a little, and few ways to win a lot, then pessimism bias is likely to miss the big wins, as it's trying to avoid minor losses.

I'd really enjoy an analysis focused on your conditions (maximize vs satisfice, world symmetry) - especially what kinds of worlds and biased predictors lead satisficing to get better outcomes than optimizing.

comment by paulfchristiano · 2020-01-27T16:34:14.453Z · score: 3 (2 votes) · LW(p) · GW(p)
For example, if the world is symmetric in the appropriate sense in terms of what actions get you rewarded or penalized, and you maximize expected utility instead of satisficing in some way, then the argument is wrong. I'm sure there is good literature on how to model evolution as a player, and the modeling of the environment shouldn't be difficult.

I would think it would hold even in that case, why is it clearly wrong?

comment by edoarad · 2020-01-28T16:59:36.242Z · score: 3 (2 votes) · LW(p) · GW(p)

I may be mistaken. I tried reversing your argument, and I bold the part that doesn't feel right.

Optimistic errors are no big deal. The agent will randomly seek behaviours that get rewarded, but as long as these behaviours are reasonably rare (and are not that bad) then that’s not too costly.
But pessimistic errors are catastrophic. The agent will systematically make sure not to fall into behaviors that avoid high punishment, and will use loopholes to avoid penalties even if that results in the loss of something really good. So even if these errors are extremely rare initially, they can totally mess up my agent.

So I think that maybe there is inherently an asymmetry between reward and punishment when dealing with maximizers.

But my intuition comes from somewhere else. If the difference between pessimism and optimism is given by a shift by a constant then it ought not matter for a utility maximizer. But your definition goes at errors conditional on the actual outcome, which should perhaps behave differently.

comment by paulfchristiano · 2020-01-28T18:10:47.249Z · score: 3 (2 votes) · LW(p) · GW(p)

I think this part of the reversed argument is wrong:

The agent will randomly seek behaviours that get rewarded, but as long as these behaviours are reasonably rare (and are not that bad) then that’s not too costly

Even if the behaviors are very rare, and have a "normal" reward, then the agent will seek them out and so miss out on actually good states.

comment by Pattern · 2020-01-28T18:38:59.280Z · score: 1 (1 votes) · LW(p) · GW(p)

But there are behaviors we always seek out. Trivially, eating, and sleeping.

comment by edoarad · 2020-01-31T08:58:00.612Z · score: 1 (1 votes) · LW(p) · GW(p)

I think that the intuition for this argument comes from something like a gradient ascent under an approximate utility function. The agent will spend most of it's time near what it perceives to be a local(ish) maximum.

So I suspect the argument here is that Optimistic Errors have a better chance of locking into a single local maximum or strategy, which get's reinforced enough (or not punished enough), even though it is bad in total.

Pessimistic Errors are ones in which the agent strategically avoids locking into maxima, perhaps by Hedonic Adaptation as Dagon suggested. This may miss big opportunities if there are actual, territorial, big maxima, but that may not be as bad (from a satisficer point of view at least).

comment by edoarad · 2020-01-31T09:03:25.392Z · score: 1 (1 votes) · LW(p) · GW(p)

If this is the case, this seems more like a difference in exploration/exploitation strategies.

We do have positively valenced heuristics for exploration - say curiosity and excitement

comment by DanArmak · 2020-01-26T17:17:48.833Z · score: 3 (2 votes) · LW(p) · GW(p)
But optimistic errors are catastrophic. The agent will systematically seek out the behaviors that receive the high reward, and will use loopholes to avoid penalties when something actually bad happens. So even if these errors are extremely rare initially, they can totally mess up my agent.

In other words, make sure your agent can't wirehead! And from evolution's point of view, not spending your time promoting your inclusive genetic fitness is wireheading.

We didn't have the technology to literally wirehead until maybe very recently, so we went about it the long way round. But we still spend a lot of time and resources on e.g. consuming art, even absent signalling benefits, like watching TV alone at home. And evolution doesn't seem to be likely to "fix" that given some more time.

We don't behave in a "Malthusian" way, investing all extra resources in increasing the number or relative proportion of our descendants in the next generation. Even though we definitely could, since population grows geometrically. It's hard to have more than 10 children, but if every descendant of yours has 10 children as well, you can spend even the world's biggest fortune. And yet such clannish behavior is not a common theme of any history I've read; people prefer to get (almost unboundedly) richer instead, and spend those riches on luxuries, not children.

comment by maximkazhenkov · 2020-01-26T17:58:11.307Z · score: 2 (2 votes) · LW(p) · GW(p)
And evolution doesn't seem to be likely to "fix" that given some more time.

Why would you suppose that?

We don't behave in a "Malthusian" way, investing all extra resources in increasing the number or relative proportion of our descendants in the next generation. Even though we definitely could, since population grows geometrically. It's hard to have more than 10 children, but if every descendant of yours has 10 children as well, you can spend even the world's biggest fortune. And yet such clannish behavior is not a common theme of any history I've read; people prefer to get (almost unboundedly) richer instead, and spend those riches on luxuries, not children.

Isn't that just due to the rapid advance of technology creating a world in disequilibrium? In the ancestral environment of pre-agricultural societies these behaviors you describe line up with maximizing inclusive genetic fitness pretty well; any recorded history you can read is too new and short to reflect what evolution intended to select for.

comment by DanArmak · 2020-01-26T18:50:01.603Z · score: 4 (3 votes) · LW(p) · GW(p)
Why would you suppose that?

Exactly for the reason you give yourself - we now change our behavior and our environment on much shorter timescales than evolution operates on, due in large part to modern technology. We have a goal of circumventing evolution (see: this post) and we modify our goals to suit ourselves. Evolution is no longer fast enough to be the main determinant of prevailing behavior.

In the ancestral environment of pre-agricultural societies these behaviors you describe line up with maximizing inclusive genetic fitness pretty well

We don't know almost anything about most relevant human behavior from before the invention of writing. Did they e.g. consume a lot of art (music, storytelling, theater, dancing)? How much did such consumption correlate with status or other fitness benefits, e.g. by conspicuous consumption or advertising wealth? We really don't know.

comment by maximkazhenkov · 2020-01-26T19:13:41.070Z · score: 3 (3 votes) · LW(p) · GW(p)

Ok I see, I was just confused by the wording "given some more time". I've become less optimistic over time about how long this disequilibrium will last given how quickly certain religious communities are growing with the explicit goal of outbreeding the rest of us.