The Human's Hidden Utility Function (Maybe)

This is also a repeat of the lesson that "you can often out-pace most philosophers simply by reading what today's leading scientists have to say about a given topic instead of reading what philosophers say about it."

On the other hand, rationality can be faster than science. And I'm feeling pretty good about positing three different forms of motivation, divided between model-free tendencies based on conditioning, and model-based goals, then saying we could use transhumanism to focus on the higher-level rational ones, without having read the particular neuroscience you're citing...

...actually, wait. I read as much of the linked paper as I could (Google Books hides quite a few pages) and I didn't really see any strong neuroscientific evidence. It looked like they were inferring the existence of the three systems from psychology and human behavior, and then throwing in a bit of neuroscience by mentioning some standard results like the cells that represent error in reinforcement learning. What I didn't see was a description of how three separate systems naturally fall out of brain studies. But I missed a lot of the paper - is there anything like that in there?

Um, objection, I didn't actually say that and I would count the difference as pretty significant here. I said, "I would be suspicious of that for the inverse reason my brain wants to say 'but there has to be a different way to stop the train' in the trolley problem - it sounds like the correct answer ought to be to just keep the part with the coherent utility function in CEV which would make it way easier, but then someone's going to jump up and say: 'Ha ha! Love and friendship were actually in the other two!'"

What's the evidence that this is the "leading theory of choice in the human brain"? (I am not saying I have evidence that it isn't, but it's important for this post that some large relevant section of the scientific community thinks this theory is awesome.)

Congratulations on continuing this line of inquiry!

One thing that worries me is that it seems to focus on the "wanting" part to the exclusion of the "liking" part, so we may end up in a world we desire today but won't enjoy tomorrow. In particular, I suspect that a world built according to our publicly stated preferences (which is what many people seem to think when they hear "reflective equilibrium") won't be very fun to live in. That might happen if we get much of our fun from instinctive and Pavlovian actions rather than planned actions, which seems likely to be true for at least some people. What do you think about that?

Suppose it turned out that humans violate the axioms of VNM rationality (and therefore don't act like they have utility functions) because there are three valuation systems in the brain that make conflicting valuations

Humans violate any given set of axioms simply because they are not formally flawless, so such explanations only start being relevant when discussing an idealization, in this case a descriptive one. But properties of descriptive idealizations don't easily translate into properties of normative idealizations.

The quoted summaries of each of the three systems are confusing and I don't feel like I have an understanding of them, except insofar as the word "Pavlovian" gives a hint. Can you translate more clearly, please?

I'm not sure I understand the difference between 2 and 3. The term pavlovian is being applied to the third system, but 2 sounds more like the archtypal pavlovian learned response (dog learns that bell results in food). Does 3 refer exclusively to pre-encoded pleasant/unpleasant responses rather than learned ones? Or is there maybe a distinction between a value and an action response that I'm missing?

Where do the model-based system's terminal goals come from?

If anyone reads this comment... 
Do you know if this claims are have held up? Does this post still agree with current neuroscience, or have there been some major updates?

I'm skeptical of any clear divide between the systems. Of course, there are more abstract and more primitive information paths, but they talk to each other, and I don’t buy that they can be cleanly separated.

Plans can be more or less complicated, and can involve “I don’t know how this part works, but it worked last time, so lets do this” and what worked last time can be very pleasurable and rewarding - so it doesn’t seem to break down cleanly into any one category.

I’d also argue that, to the extent that abstract planning is successful, it is because it propagates top down and affects the lower pavlovian systems. If your thoughts about your project aren’t associated with motivation and wanting to actually do something, then your abstract plans aren’t of much use. It just isn’t salient that this happening unless the process is disrupted and you find yourself not doing what you “want” to do.

Another point that is worth stating explicitly is that algorithms for maximizing utility are not utility functions. In theory, you could have 3 different optimizers that all maximize the same utility function, or 3 different utility functions that all use the same optimizer - or any other combination.

I don’t think this is a purely academic distinction either - I think that we have conflicts at the same level all the time (multiple personality disorder being an extreme case). Conflicts between systems with no talk at between levels look like someone saying they want something, and then doing another without looking bothered at all. When someone is obviously pained by the conflict, then they are clearly both operating on an emotional level, even if the signal originated at different places. Or I could create a pavlovian conflict in my dog by throwing a steak on the wet tile, and watching as his conditioned fear of wet tile fights the conditioned desire of steak.

'Ha ha! Love and friendship were actually in the other two!'"

This concern is not-abstract and very personal for me. As I've said around here before, I often find myself exhibiting borderline-sociopathic thinking in many situations, but the arrangement of empathy and ethical inhibitions in my brain, though off-kilter in many ways*, drives me to take even abstract ethical problems (LW examples: Three Worlds Collide, dust specks, infanticide, recently Moldbug's proposal of abolishing civil rights for the greater good) very personally, generates all kinds of strong emotions about them - yet it has kept me from doing anything ugly so far.

(The most illegal thing I've done in my life during the moments when I 'let myself go' was some petty and outwardly irrational shoplifting in my teenage years; reflecting back upon that, I did it not solely to get an adrenaline rush but also to push my internal equilibrium into a place where this "superego" thing would recieve an alarm and come back online)

What if this internal safety net of mine is founded solely upon #2 and #3?

( As I've mentioned in some personal anecdotes, - and hell, I don't wish to drone on and on about this, just feeling it's relevant - this part of me has been either very weak or dormant until I watched Evangelion when I was 18. The weird, lingering cathartic sensation and the feeling of psychological change, which felt a little like growing up several years in a week, was the most interesting direct experience in my life so far. However, I've mostly been flinching from consciously* trying to push myself towards the admirable ethics of interpersonal relations that I view as the director's key teaching. It's painful enough when it's happening without conscious effort on your part!)

This might be a silly question, but still:

Are the three models actually running on three different sets of wetware within the brain, or are they merely a convenient abstraction of human behavior ?

It seems to me that the actual situation is that upon reflection we would clearly reject (most of) the outputs of all three systems. What human brain actually computes, in any of its modules or in all of them together, is not easily converted into considerations about how the decisions should be made.

In other words, the valuations made by human valuation systems are irrelevant, even though the only plausible solution involves valuations based on human valuation systems. And converting brains into definitions of value will likely break any other abstractions about the brains that theorize them as consisting of various modules with various purposes.

If I understand this correctly, then the model-based system and the model-free system sound like inside and outside views.

Suppose it turned out that humans violate the axioms of VNM rationality (and therefore don't act like they have utility functions) because there are three valuation systems in the brain that make conflicting valuations, and

A question, probably silly: Suppose you calculate what a person would do given every possible configuration of sensory inputs, and then construct a utility function that returns one if that thing is done and zero otherwise. Can't we then say that any deterministic action-taking thing acts according to some utility function?

Or, even more trivially, just let the utility be constant. Then any action maximizes utility.

Edit: If you're using utility functions to predict actions, then the constant utility function is like a maximum entropy prior, and the "every possible configuration" thing is like a hypothesis that simply lists all observations without positing some underlying pattern, so it would eventually get killed off by being more complicated than hypotheses that actually "compress" the evidence.

As a first reaction (and without being read up on the details), I'm very skeptical. Assuming these three systems are actually in place, I don't see any convincing reason why any one of them should be trusted in isolation. Natural selection has only ever been able to work on their compound output, oblivious to the role played by each one individually and how they interact.

Maybe the "smart" system has been trained to assign some particular outcome a value of 5 utilons, whereas we would all agree that it's surely and under all circumstances worth more than 20, because as it happens throughout evolution one of the other "dumb" systems has always kicked in and provided the equivalent of at least 15 utilons. If you then extract the first system bare and naked, it might deliver some awful outputs.

Just some initial thoughts,

I do understand that these statements are broad generalisations for what really does occur though the premise is that a successful choice would be made from wieighting options provided from the scenarios.

As with genetics and other systems the beneficial error scenario which can be described in situations such as a miskeyed note on a keyboard leading to a variation of the sequence that is favourable seems excluded from these scenarios.

Improvisation based on self introduced errors may also be a core to these utilities being able to evolve reason.

Model-based system: Figure out what's going on, and what actions maximize returns, and do them.

Model-free system: Do the thingy that worked before again!

Pavlovian system: Avoid the unpleasant thing and go to the pleasant thing. Repeat as necessary.

I think that you can keep up the utility function a bit longer if you add the costs of thinking to it - required time and energy, and maybe aversion of thinking about it. "I could compare these two items in the supermarket for 30 minutes and finally find out which product is better - or I could just ignore the new option and take the same thing as last time". It can be the perfectly rational option to just stick with something which worked before.

It is also rational to decide how much time you invest to decide something (and if there is a lot of money involved, this is usually done). If the time for a decision is not enough to build and use a model, you fall back to more "primitive" methods. In fact, most of the everyday decisions have to be done like that. Each second, you have several options available, and no possibility to re-think about all of them every time.

We need all 3 systems for our life. The interesting thing is just to decide which system is useful for which decision and which time it should get. Look at it from a higher perspective, and you can get a well-defined utility function for a brain which has access to these systems to evaluate things.

Okay, which system decides which way the rat should turn when rat is navigating a maze? A cat doing actual path-finding on complex landscape? (which is surprisingly hard to do if you are coding a cat AI. Path finding, well, it is rather 'rational' in the sense that animals don't walk into the walls and the like) A human navigating a maze with a map to get food? A cat doing path finding avoiding a place where the cat had negative experience? ("conditioning").

It seems to me that those 3 'systems', if there are such 3 systems, aren't interacting in the way that article speaks of.

At a glance, it seems that upon reflection I might embrace an extrapolation of the model-based system's preferences as representing "my values," and I would reject the outputs of the model-free and Pavlovian systems as the outputs of dumb systems that evolved for their computational simplicity, and can be seen as ways of trying to approximate the full power of a model-based system responsible for goal-directed behavior.

At a glance, I might be more comfortable embracing an extrapolation of the combination of the model-based system's preferences and the Pavlovian system's preferences.

Admittedly, a first step in extrapolating the Pavlovian system's preferences might be to represent its various targets as goals in a model, thereby leaving the extrapolator with a single system to extrapolate, but given that 99% of the work takes place after this point I'm not sure how much I care. Much more important is to not lose track of that stuff accidentally.

Suppose it turned out that humans violate the axioms of VNM rationality (and therefore don't act like they have utility functions) because there are three valuation systems in the brain that make conflicting valuations, and all three systems contribute to choice.

Er, I don't think so. To quote from here:

Utility maximisation is a general framework which is powerful enough to model the actions of any computable agent. The actions of any computable agent - including humans - can be expressed using a utility function. This was spelled out by Dewey in a 2011 paper titled: "Learning What to Value" - in his section about "O-Maximisers".

Some argue that humans have no utility function. However, this makes little sense: all computable agents have utility functions. The human utility function may not be easy to write down - but that doesn't mean that it doesn't exist.