Why the beliefs/values dichotomy?

Just to distance this very interesting question from expected utility maximization: "Beliefs" sound like they are about couldness, and values about shouldness. Couldness is about behavior of the environment outside the agent, and shouldness is about behavior of the agent. Of course, the two only really exist in interaction, but as systems they can be conceptualized separately. When an agent asks what it could do, the question is really about what effects in environment could be achieved (some Tarskian hypocrisy here: using "could" to explain "couldness"). Beliefs is what's assumed, and values is what's asserted. In a decision tree, beliefs are associated with knowledge about other agent's possible actions, and values with the choice of the present agent's action. Both are aspects of the system, but playing different roles in the interaction: making a choice versus accepting a choice. Naturally, there is a duality here, when the sides are exchanged: my values become your beliefs, and my beliefs become your values. Choice of representation is not that interesting, as it's all interpretation: nothing changes in behavior.

It seems clear that our preferences do satisfy Independence, at least approximately.

How big of a problem does this simple example signify?

• A = I acquire a Nintendo
• B = I acquire a Playstation
• C = I acquire a game for the Nintendo
• D = I acquire a game for the Playstation
• A&C > B&C but A&D < B&D

I have a tentative answer for the second question of "Why this representation?". Given that a set of preferences can be represented as a probability function and a utility function, that seems computationally more convenient than using two probability functions, since then you only have to do half of the Bayesian updating.

Another part of this question is that such a set of preferences can usually be decomposed many different ways into probability and utility, so what explains the particular decomposition that we have? I think there should have been a selection pressure for humans to have a common prior, to the extent possible, and move as much as possible of the differences in preferences into the utility function, since that would facilitate communication and sharing of information. It seems that if we had common priors, and I have a lot of information about something (and you trust me), I can just tell you my posterior beliefs, instead of having to give you all of the raw information and let you recompute your own posterior beliefs.

"Of all the axioms, independence is the most often discarded. A variety of generalized expected utility theories have arisen, most of which drop or relax the independence axiom."

• http://en.wikipedia.org/wiki/Utility

Paul Churchland calls the belief/values (he says belief/desires) model "folk psychology" and assigns a low probability to it "being smoothly reduced by neuroscience" rather than being completely disregarded like, say, the phlogiston theory of combustion. The paper is called Eliminative Materialism and the Propositional Attitudes and was printed in The Journal of Philosophy. I didn't find the paper all that convincing, but your mileage may vary.

This paper was cited along with another by someone (can't remember who) arguing that the belief/values theory of behavior (i.e. expected utility theory) doesn't capture how humans behave. The second paper I think argues that much of what we do can be explained by control theory without reference to beliefs or values, but I haven't read it yet.

The papers are:

Churchland, Paul. Eliminative Materialism and the Propositional Attitudes, The Journal of Philosophy.

van Gelder, Tim. What Might Cognition be, if not Computation?, The Journal of Philosophy.

For those of you who don't have the benefit of a university subscription to J-stor or something similar, I have pdfs of both papers. Just shoot me an email at : themattsimpson AT DOT company

This comment is directly about the question of probability and utility. The division is not so much about considering the two things separately, as it is about extracting tractable understanding of the whole human preference (prior+utility) into a well-defined mathematical object (prior), while leaving all the hard issues with elicitation of preference in the utility part. In practice it works like this: a human conceptualizes a problem so that a prior (that is described completely) can be fed to an automatic tool, then tool's conclusion about the aspect specified as probability is interpreted by a human again. People fill in the utility part by using their preference, even though they can't represent it as the remaining utility part. Economists, having to create autonomous models of decision-making (as distinct from autonomous decision-making systems), have to introduce the whole preference, but it's so approximate that it's of no use in most other contexts.

Because of the utility-prior divide of human preference in practice of human decision-making, with only prior in the domain of things that are technically understood, there is a strong association of prior with "knowledge" (hence "belief", but being people of science we expel feeling-associated connotations from the concept), while utility remains vague, but is a necessary part that completes the picture to the expression of whole preference, hence introduction of utility to a problem is strongly associated with values.

It's not the result of an "accidental" product of evolution that organisms are goal-directed and have values. Evolution made creatures that way for a reason - organisms that pursue their biological goals (without "updating" them) typically have more offspring and leave more descendants.

Mixing up your beliefs and values would be an enormous mistake - in the eyes of evolution. You might then "update" your values - trashing them in the process - a monumental disaster for your immortal coils.

This assumption is central to establishing the mathematical structure of expected utility maximization, where you value each possible world separately using the utility function, then take their weighted average. If your preferences were such that A&C > B&C but A&D < B&D, then you wouldn’t be able to do this.

I can imagine having preferences that don't value each possible world separately. I can also imagine doing other things to my utility function than maximising expectation. For example, if I maximised the top quartile of expected values then I may choose to engage in practices analogous to quantum suicide. That I prefer, in principle, to maximise expected utility is itself a value. It is a value I that I expect to see in most successful agents, for fundamental reasons.

Here, have a mathematical perspective that conflates beliefs and values:

Suppose that some agent is given a choice between A and B. A is an apple. B is an N chance of a banana, otherwise nothing. The important thing here is the ambivalence equation: iff U(apple) = N*U(banana), the agent is ambivalent between the apple and the banana. Further suppose that N is 50%, and the agent likes bananas twice as much as it likes apples. In this case, at least, the agent might as well modify itself to believe that N is 20% and to like bananas five times as much as apples.

Now, doing this might result in inconsistencies elsewhere, but I'm guessing that a rational agent will be able to apply transformations to its beliefs and values--but only both simultaneously--so as to preserve expected utility given actions.

I think a more concrete example of the beliefs-values meld is anthropic reasoning. Take the presumptuous friend: you, the presumptuous philosopher's presumptuous friend, have just been split into 1,000,001 branches, and each of those branches has been placed in a hotel room, 1,000,000 rooms being in one hotel, and the other room being in another hotel. Is the probability that you're in the small hotel 50%, negligible, or something in between? Well, that depends: if you care about each of your branches equally, it's negligible; if you care about each hotel equally, it's 50%.

I think values (in a finite agent), also need to have some role in what beliefs "should" be stored/updated/remembered. Of course in theories which don't constrain the agents computational ability this isn't needed.

I dispute your premise: what makes you so sure people do decompose their thoughts into beliefs and values, and find these to be natural, distinct categories? Consider the politics as mind-killer phenomenon. That can be expressed as, "People put your words into a broader context of whether they threaten their interests, and argue for or against your statements on that basis."

For example, consider the difficulty you will have communicating your position if you believe both a) global warming is unlikely to cause any significant problems in the business-as-usual scenario, b) high taxes on CO2 emissions should be levied. (e.g., you believe it's a good idea as an insurance policy and can be done in a way that blocks most of the economic damage)

(Yes, I had to use a present example to make the reactions easier to imagine.)

The "ought" is so tightly coupled to the "is", that in any case where the "ought" actually matters, the "is" comes along for the ride.

Note: this is related to the problem I had with the exposition of could/would/should agents: if you say humans are CSAs, what's an example of an intelligent agent that isn't?

Maybe these are to do with differences across individuals. My beliefs/values may be mashed togather and impossible to seperate, but I expect other people's beliefs to mirror my own more closely than their values do.

Because it's much easier to use beliefs shorn of values as building blocks in a machine that does induction, inference, counterfactual reasoning, planning etc compared to belief-values that are all tied up together.

Sea slugs and Roombas don't have the beliefs/values separation it because the extra complexity isn't worth it. Humans have it to some degree and rule the planet. AIs might have even more success.

Some of this is going over my head, but...

I think you need to specify if you're talking about terminal values or instrumental values.

There's obviously a big difference between beliefs and terminal values. Beliefs are inputs to our decision-making processes, and terminal values may be as well. However, while beliefs are outputs of the belief-making processes whose inputs are our perceptions, terminal values aren't the output of any cognitive process, or they wouldn't be terminal.

As for instrumental values, well, yes, they are beliefs about the best values that one should have in order to reach the terminal goals, so they would be outputs of the same or similar processes that create beliefs. Or at least, they start out that way; some or all of them end up becoming terminal values.

I think I tried to solve a similar problem before: that of looking at the simplest possible stable control system and seeing how I can extract the system's "beliefs" and "values" that result in it remaining stable. Then, see if I can find a continuous change between the structure of that system, and a more complex system, like a human.

For example, consider the simple spring-mass-damper system. If you move it from its equlibrium position xe, it will return. What do the concepts of "belief" and "value" map onto here? For beliefs, I used the concept of mutual information: what about the system could you look at to learn whether the mass is not at xe? How does the system know it's not at xe?

The information is contained in the force the spring exerts. However, this is also the determinant of which direction it moves the spring, it's "value". So it looks like the beliefs and values are fully-mixed: the same thing that tells you what it believes, tells you what it does. In that case, at what point, in the structural transition from the spring to intelligent agents, does the distinction between values and beliefs start to form, if at all?

Incidently, I only just now read Vladimir_Nesov's post because previously I hadn't bothered to make the equations readable, since they don't render properly in the browser I used. "Beware trivial inconveniences", indeed!

(Separate post because of the different issues and to avoid tl;dr.)

Is it possible that the dichotomy between beliefs and values is just an accidental byproduct of our evolution, perhaps a consequence of the specific environment that we’re adapted to, instead of a common feature of all rational minds?

In the normal usage, "mind" implies the existence of a distinction between beliefs and values. In the LW/OB usage, it implies that the mind is connected to some actuators and sensors which connect to an environment and is actually doing some optimization toward those values. Certainly "rational mind" entails a beliefs/values separation.

But suppose we abandon the beliefs/values separation: what properties do we have left? Is the concept "mind without a beliefs/values separation" Simply the concept "thing"?