Emotional valence vs RL reward: a video game analogy

post by Steven Byrnes (steve2152) · 2020-09-03T15:28:08.013Z · LW · GW · 6 comments

Contents

  Definition of "the valence of an emotional state" (at least as I'm using the term)
  How is emotional valence implemented computationally? A video game analogy
  Worked examples
None
6 comments

(Update Sept 2021: I no longer believe that we make decisions that maximize the expected sum of future rewards—see discussion of TD learning here [LW · GW]. It's more like "maximizing expected reward next step" (but "next step" can entail making a long-term plan). So this post isn't quite right in some of its specifics. That said, I don't think it's wildly wrong and I think it would just need a couple tweaks. Anyway, it's just a brainstorming post, don't take it too literally.)

I recently read a book about emotions and neuroscience (brief review here [LW · GW]) that talked about "valence and arousal" as two key ingredients of our interoception. Of these, arousal seems pretty comprehensible—the brain senses the body's cortisol level, heart rate, etc. But the valence of an emotion—what is that? What does it correspond to in the brain and body? My brief literature search didn't turn up anything that made sense to me, but after thinking about it a bit, here is what I came up with (with the usual caveat that it may be wrong or obvious). But first,

Definition of "the valence of an emotional state" (at least as I'm using the term)

Here's how I want to define the valence of an emotional state:

There's a chance that I'm misusing the term; the psychological literature itself seems all over the place. For example, some people say anger is negative valence, but when I feel righteous anger, I like having that feeling, and I want that feeling to continue. (I don't want to want that feeling to continue, but I do want that feeling to continue!) So by my definition, righteous anger is positive valence!

There are some seemingly-paradoxical aspects of how valence does or doesn't drive behavior:

(See also: Scott Alexander on Wanting vs Liking vs Approving [LW · GW])

How is emotional valence implemented computationally? A video game analogy

In Doom II (1994 ... I guess I'm showing my age), you could lose a bunch of health points all at once by getting hit by an enemy (left), or you could go running on lava and you'll lose a few health points every second until you get off the lava (right). By analogy, when I eat junk food, I get a big transient positive reward (a.k.a. "dopamine hit"); when I feel positive-valence emotions (happy, proud, righteous indignation, etc.) I claim that I'm getting a constant stream of positive reward as long as I'm in that state; and conversely when I feel negative-valence emotions (guilt, suffering, etc.), I claim that I'm getting a constant stream of negative reward as long as I'm in that state.  

Here's a simple picture I kinda like, based on an analogy to action-type video games. (Ha, I knew it, playing all those video games in middle school wasn't a waste of time after all!)

In many video games you control a character with a "health" level. It starts at 100 (or whatever), and if it ever gets to 0, you die. There are two ways to gain or lose health:

In the brain, I've come around to the reinforcement-learning-type view that the neocortex tries to maximize a "reward" signal (among other things) [LW · GW]. So in the above, replace "gain health points" with "get positive reward", replace "lose health points" with "get negative reward", then the "state-based" situation corresponds to my current working theory of what valence is. Pretty simple, right?

To be explicit:

Worked examples

6 comments

Comments sorted by top scores.

comment by Gordon Seidoh Worley (gworley) · 2020-09-05T22:27:30.167Z · LW(p) · GW(p)

There's this from QRI that I think also points to a similar interpretation of valence and arousal as the one you use here.

Replies from: steve2152
comment by Steven Byrnes (steve2152) · 2020-09-11T22:45:50.219Z · LW(p) · GW(p)

Thanks but I don't see the connection between what I wrote and what they wrote ...

Update: Maybe you meant that they understand the term "valence" in the same way I do. That seems plausible. Their explanation of valence is wildly different than mine, but we are both talking about the same thing, I think.

comment by Kaj_Sotala · 2020-09-04T14:19:14.594Z · LW(p) · GW(p)
What does it correspond to in the brain

lukeprog talks a bit about this in "The Neuroscience of Pleasure [LW · GW]".

comment by noggin-scratcher · 2020-09-03T20:32:45.000Z · LW(p) · GW(p)

Well, if the health gain from the health kit is large enough to outweigh the health loss from needing to run through lava afterwards, then OK, maybe that's worth doing.

Also even if it's not actually enough, and you're going to come out at a small loss overall, sometimes by the magic of time discounting it still feels like a net positive to your present self - because the cost is further away into the future than the gain.

Replies from: steve2152
comment by Steven Byrnes (steve2152) · 2020-09-03T23:35:24.326Z · LW(p) · GW(p)

Yes and this is one of many ways that humans don't maximize the time-integral of reward. Sorry for an imperfect analogy. :)

Replies from: noggin-scratcher
comment by noggin-scratcher · 2020-09-05T10:22:23.447Z · LW(p) · GW(p)

I thought it was still quite an apt analogy, because we do essentially the same time-biased thing in all kinds of other contexts