Why the beliefs/values dichotomy?

post by Wei Dai (Wei_Dai) · 2009-10-20T16:35:52.427Z · LW · GW · Legacy · 156 comments

Contents

  Why this mathematical structure?
  Why this representation?
None
156 comments

I'd like to suggest that the fact that human preferences can be decomposed into beliefs and values is one that deserves greater scrutiny and explanation. It seems intuitively obvious to us that rational preferences must decompose like that (even if not exactly into a probability distribution and a utility function), but it’s less obvious why.

The importance of this question comes from our tendency to see beliefs as being more objective than values. We think that beliefs, but not values, can be right or wrong, or at least that the notion of right and wrong applies to a greater degree to beliefs than to values. One dramatic illustration of this is in Eliezer Yudkowsky’s proposal of Coherent Extrapolated Volition, where an AI extrapolates the preferences of an ideal humanity, in part by replacing their "wrong” beliefs with “right” ones. On the other hand, the AI treats their values with much more respect.

Since beliefs and values seem to correspond roughly to the probability distribution and the utility function in expected utility theory, and expected utility theory is convenient to work with due to its mathematical simplicity and the fact that it’s been the subject of extensive studies, it seems useful as a first step to transform the question into “why can human decision making be approximated as expected utility maximization?”

I can see at least two parts to this question:

Not knowing how to answer these questions yet, I’ll just write a bit more about why I find them puzzling.

Why this mathematical structure?

It’s well know that expected utility maximization can be derived from a number of different sets of assumptions (the so called axioms of rationality) but they all include the assumption of Independence in some form. Informally, Independence says that what you prefer to happen in one possible world doesn’t depend on what you think happens in other possible worlds. In other words, if you prefer A&C to B&C, then you must prefer A&D to B&D, where A and B are what happens in one possible world, and C and D are what happens in another.

This assumption is central to establishing the mathematical structure of expected utility maximization, where you value each possible world separately using the utility function, then take their weighted average. If your preferences were such that A&C > B&C but A&D < B&D, then you wouldn’t be able to do this.

It seems clear that our preferences do satisfy Independence, at least approximately. But why? (In this post I exclude indexical uncertainty from the discussion, because in that case I think Independence definitely doesn't apply.) One argument that Eliezer has made (in a somewhat different context) is that if our preferences didn’t satisfy Independence, then we would become money pumps. But that argument seems to assume agents who violate Independence, but try to use expected utility maximization anyway, in which case it wouldn’t be surprising that they behave inconsistently. In general, I think being a money pump requires having circular (i.e., intransitive) preferences, and it's quite possible to have transitive preferences that don't satisfy Independence (which is why Transitivity and Independence are listed as separate axioms in the axioms of rationality).

Why this representation?

Vladimir Nesov has pointed out that if a set of preferences can be represented by a probability function and a utility function, then it can also be represented by two probability functions. And furthermore we can “mix” these two probability functions together so that it’s no longer clear which one can be considered “beliefs” and which one “values”. So why do we have the particular representation of preferences that we do?

Is it possible that the dichotomy between beliefs and values is just an accidental byproduct of our evolution, perhaps a consequence of the specific environment that we’re adapted to, instead of a common feature of all rational minds? Unlike the case with anticipation, I don’t claim that this is true or even likely here, but it seems to me that we don’t understand things well enough yet to say that it’s definitely false and why that's so.

156 comments

Comments sorted by top scores.

comment by Vladimir_Nesov · 2009-10-20T18:26:41.340Z · LW(p) · GW(p)

Just to distance this very interesting question from expected utility maximization: "Beliefs" sound like they are about couldness, and values about shouldness. Couldness is about behavior of the environment outside the agent, and shouldness is about behavior of the agent. Of course, the two only really exist in interaction, but as systems they can be conceptualized separately. When an agent asks what it could do, the question is really about what effects in environment could be achieved (some Tarskian hypocrisy here: using "could" to explain "couldness"). Beliefs is what's assumed, and values is what's asserted. In a decision tree, beliefs are associated with knowledge about other agent's possible actions, and values with the choice of the present agent's action. Both are aspects of the system, but playing different roles in the interaction: making a choice versus accepting a choice. Naturally, there is a duality here, when the sides are exchanged: my values become your beliefs, and my beliefs become your values. Choice of representation is not that interesting, as it's all interpretation: nothing changes in behavior.

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-10-20T22:36:38.084Z · LW(p) · GW(p)

I gave an example where choice of representation is important: Eliezer's CEV. If the choice of representation shouldn't to be important, then that seems to be argument against CEV.

Replies from: SilasBarta, Vladimir_Nesov
comment by SilasBarta · 2009-10-20T22:54:59.822Z · LW(p) · GW(p)

Bullet acknowledged and bitten. A Friendly AI attempting to identify humanity's supposed CEV will also have to be a politician and have enough support so that they don't shut it down. As a politician, it will have to appeal to people with the standard biases. So it's not enough for it to say, "okay, here's something all of you should agree on as a value, and benefit from me moving humanity to that state".

And in figuring out what would appeal to humans, it will have to model the same biases that blur the distinction.

comment by Vladimir_Nesov · 2009-10-20T22:54:08.875Z · LW(p) · GW(p)

I was referring to you referring to my post on playing with utility/prior representations.

comment by PhilGoetz · 2009-10-22T21:39:01.722Z · LW(p) · GW(p)

It seems clear that our preferences do satisfy Independence, at least approximately.

How big of a problem does this simple example signify?

  • A = I acquire a Nintendo
  • B = I acquire a Playstation
  • C = I acquire a game for the Nintendo
  • D = I acquire a game for the Playstation
  • A&C > B&C but A&D < B&D
Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-10-24T11:29:16.257Z · LW(p) · GW(p)

Your example shows that we can't assign utilities to events within a single world, like acquiring game systems and games, and then add them up into a utility for that world, but it's not a counterexample to Independence, because of this part:

A and B are what happens in one possible world, and C and D are what happens in another.

Independence is necessary to assign utilities to possible world histories and aggregate those utilities linearly into expected utility. Consider the apples/oranges example again. There,

  • A = I get an apple in the world where coin is heads
  • B = I get an orange in the world where coin is heads
  • C = I get an apple in the world where coin is tails
  • D = I get an orange in the world where coin is tails

Then, according to Independence, my preferences must be either

  1. A&C > B&C and A&D > B&D, or
  2. A&C < B&C and A&D < B&D

If case 1, I should pick the transparent box with the apple, and if case 2, I should pick the transparent box with the orange.

(I just realized that technically, my example is wrong, because in case 1, it's possible that A&D > A&C and B&D > B&C. Then, I should most prefer an opaque box that contains an apple if the coin is heads and an orange if the coin is tails, since that gives me outcome A&D, and least prefer an opaque box that contains the opposite (gives me B&C). So unless I introduce other assumptions, I can only derive that I shouldn't simultaneously prefer both kinds of opaque boxes to transparent boxes.)

comment by Wei Dai (Wei_Dai) · 2009-10-21T12:02:44.065Z · LW(p) · GW(p)

I have a tentative answer for the second question of "Why this representation?". Given that a set of preferences can be represented as a probability function and a utility function, that seems computationally more convenient than using two probability functions, since then you only have to do half of the Bayesian updating.

Another part of this question is that such a set of preferences can usually be decomposed many different ways into probability and utility, so what explains the particular decomposition that we have? I think there should have been a selection pressure for humans to have a common prior, to the extent possible, and move as much as possible of the differences in preferences into the utility function, since that would facilitate communication and sharing of information. It seems that if we had common priors, and I have a lot of information about something (and you trust me), I can just tell you my posterior beliefs, instead of having to give you all of the raw information and let you recompute your own posterior beliefs.

comment by timtyler · 2009-10-20T20:32:07.095Z · LW(p) · GW(p)

"Of all the axioms, independence is the most often discarded. A variety of generalized expected utility theories have arisen, most of which drop or relax the independence axiom."

Replies from: Douglas_Knight, ShardPhoenix
comment by Douglas_Knight · 2009-10-24T19:22:16.251Z · LW(p) · GW(p)

"Of all the axioms, independence is the most often discarded. A variety of generalized expected utility theories have arisen, most of which drop or relax the independence axiom."

The examples in the generalized expected utility link are descriptive theories of how humans are irrational money pumps. (The two bullet points after the sentence in wikipedia are examples of conventional utility functions; in that context the sentence is false.)

Replies from: timtyler
comment by timtyler · 2009-10-25T18:22:11.275Z · LW(p) · GW(p)

I'm not sure what the bullet points are doing there either - but I don't really see how they impact the original statement.

comment by ShardPhoenix · 2009-10-22T10:20:13.373Z · LW(p) · GW(p)

Reminds me of the parallel postulate - non-Euclidean utility?

comment by Matt_Simpson · 2009-10-22T06:47:42.640Z · LW(p) · GW(p)

Paul Churchland calls the belief/values (he says belief/desires) model "folk psychology" and assigns a low probability to it "being smoothly reduced by neuroscience" rather than being completely disregarded like, say, the phlogiston theory of combustion. The paper is called Eliminative Materialism and the Propositional Attitudes and was printed in The Journal of Philosophy. I didn't find the paper all that convincing, but your mileage may vary.

This paper was cited along with another by someone (can't remember who) arguing that the belief/values theory of behavior (i.e. expected utility theory) doesn't capture how humans behave. The second paper I think argues that much of what we do can be explained by control theory without reference to beliefs or values, but I haven't read it yet.

The papers are:

Churchland, Paul. Eliminative Materialism and the Propositional Attitudes, The Journal of Philosophy.

van Gelder, Tim. What Might Cognition be, if not Computation?, The Journal of Philosophy.

For those of you who don't have the benefit of a university subscription to J-stor or something similar, I have pdfs of both papers. Just shoot me an email at : themattsimpson AT DOT company

comment by Vladimir_Nesov · 2009-10-20T20:26:37.588Z · LW(p) · GW(p)

This comment is directly about the question of probability and utility. The division is not so much about considering the two things separately, as it is about extracting tractable understanding of the whole human preference (prior+utility) into a well-defined mathematical object (prior), while leaving all the hard issues with elicitation of preference in the utility part. In practice it works like this: a human conceptualizes a problem so that a prior (that is described completely) can be fed to an automatic tool, then tool's conclusion about the aspect specified as probability is interpreted by a human again. People fill in the utility part by using their preference, even though they can't represent it as the remaining utility part. Economists, having to create autonomous models of decision-making (as distinct from autonomous decision-making systems), have to introduce the whole preference, but it's so approximate that it's of no use in most other contexts.

Because of the utility-prior divide of human preference in practice of human decision-making, with only prior in the domain of things that are technically understood, there is a strong association of prior with "knowledge" (hence "belief", but being people of science we expel feeling-associated connotations from the concept), while utility remains vague, but is a necessary part that completes the picture to the expression of whole preference, hence introduction of utility to a problem is strongly associated with values.

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-10-20T22:32:18.365Z · LW(p) · GW(p)

But why do human preferences exhibit the (approximate) independence which allows the extraction to take place?

Replies from: SilasBarta, Vladimir_Nesov
comment by SilasBarta · 2009-10-20T22:49:08.282Z · LW(p) · GW(p)

Simple. They don't.

Maybe it's just me, but this looks like another case of overextrapolation from a community of rationalists to all of humanity. You think about all the conversations you've had distinguishing beliefs from values, and you figure everyone else must think that way.

In reality, people don't normally make such a precise division. But don't take my word for it. Go up to your random mouthbreather and try to find out how well they adhere to a value/belief distinction. Ask them whether the utility assigned to an outcome, or its probability was a bigger factor.

No one actually does those calculations consciously; if anything like it is done non-consciously, it's extremely economical in computation.

comment by Vladimir_Nesov · 2009-10-20T22:51:17.290Z · LW(p) · GW(p)

Simple: the extraction cuts across preexisting independencies. (I don't quite see what you refer to by "extraction", but my answer seems general enough to cover most possibilities.)

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-10-20T23:01:31.254Z · LW(p) · GW(p)

I'm referring to the extraction that you were talking about: extracting human preference into prior and utility. Again, the question is why the necessary independence for this exists in the first place.

Replies from: Vladimir_Nesov
comment by Vladimir_Nesov · 2009-10-20T23:06:03.017Z · LW(p) · GW(p)

I was talking about extraction of prior about a narrow situation as the simple extractable aspect of preference, period. Utility is just the rest, what remains unextractable in preference.

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-10-20T23:23:44.670Z · LW(p) · GW(p)

Ok, I see. In that case, do you think there is still a puzzle to be solved, about why human preferences seem to have a large amount of independence (compared to, say, a set of randomly chosen transitive preferences), or not?

Replies from: Vladimir_Nesov
comment by Vladimir_Nesov · 2009-10-20T23:41:42.555Z · LW(p) · GW(p)

That's just a different puzzle. You are asking a question about properties of human preference now, not of prior/utility separation. I don't expect strict independence anywhere.

Independence is indifference, due to inability to see and precisely evaluate all consequences, made strict in form of probability, by decree of maximum entropy. If you know your preference about an event, but no preference/understanding on the uniform elements it consists of, you are indifferent to these elements -- hence maximum entropy rule, air molecules in the room. Multiple events for which you only care in themselves, but not in the way they interact, are modeled as independent.

[W]hy human preferences seem to have a large amount of independence (compared to, say, a set of randomly chosen transitive preferences)[?]

Randomness is info, so of course the result will be more complex. Where you are indifferent, random choice will fill in the blanks.

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-10-21T00:17:07.737Z · LW(p) · GW(p)

It sounds like what you're saying is that independence is a necessary consequence of our preferences having limited information. I had considered this possibility and don't think it's right, because I can give a set of preferences with little independence and also little information, just by choosing the preferences using a pseudorandom number generator.

I think there is still a puzzle here, why our preferences show a very specific kind of structure (non-randomness).

Replies from: Vladimir_Nesov
comment by Vladimir_Nesov · 2009-10-21T01:02:10.235Z · LW(p) · GW(p)

That new preference of yours still can't distinguish the states of air molecules in the room, even if some of these states are made logically impossible by what's known about macro-objects. This shows both the source of dependence in precise preference and of independence in real-world approximations of preference. Independence remains where there's no computed info that allows to bring preference in contact with facts. Preference is defined procedurally in the mind, and its expression is limited by what can be procedurally figured out.

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-10-21T11:38:05.491Z · LW(p) · GW(p)

I don't really understand what you mean at this point. Take my apples/oranges example, which seems to have nothing to do with macro vs. micro. The Axiom of Independence says I shouldn't choose the 3rd box. Can you tell me whether you think that's right, or wrong (meaning I can rationally choose the 3rd box), and why?

To make that example clearer, let's say that the universe ends right after I eat the apple or orange, so there are no further consequences beyond that.

Replies from: timtyler
comment by timtyler · 2009-10-21T15:53:18.392Z · LW(p) · GW(p)

To make the example clearer, surely you would need to explain what the "" notation was supposed to mean.

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-10-22T00:47:56.556Z · LW(p) · GW(p)

It's from this paragraph of http://lesswrong.com/lw/15m/towards_a_new_decision_theory/ :

What if you have some uncertainty about which program our universe corresponds to? In that case, we have to specify preferences for the entire set of programs that our universe may correspond to. If your preferences for what happens in one such program is independent of what happens in another, then we can represent them by a probability distribution on the set of programs plus a utility function on the execution of each individual program. More generally, we can always represent your preferences as a utility function on vectors of the form where E1 is an execution history of P1, E2 is an execution history of P2, and so on.

In this case I'm assuming preferences for program executions that aren't independent of each other, so it falls into the "more generally" category.

Replies from: timtyler
comment by timtyler · 2009-10-22T06:21:33.055Z · LW(p) · GW(p)

Got an example?

You originally seemed to suggest that represented some set of preferences.

Now you seem to be saying that it is a bunch of vectors representing possible universes on which some unspecified utility function might operate.

comment by timtyler · 2009-10-20T17:01:10.240Z · LW(p) · GW(p)

It's not the result of an "accidental" product of evolution that organisms are goal-directed and have values. Evolution made creatures that way for a reason - organisms that pursue their biological goals (without "updating" them) typically have more offspring and leave more descendants.

Mixing up your beliefs and values would be an enormous mistake - in the eyes of evolution. You might then "update" your values - trashing them in the process - a monumental disaster for your immortal coils.

Replies from: None
comment by [deleted] · 2009-10-21T03:21:41.122Z · LW(p) · GW(p)

Since I'm often annoyed when my posts are downvoted without explanation, and I saw that this post was downvoted, I'll try to explain the downvotes.

Updating of values happens all the time; it's called operant conditioning. If my dog barks and immediately is poked with a hot poker, its value of barking is updated. This is a useful adaptation, as being poked with a hot poker decreases fitness. If my dog tries to mate and immediately receives an electric shock, its value of making is decreased. This is a harmful adaptation, as mating is a more fundamental fitness factor than electric shocks.

So, you seem to be explaining an observation that is not observed using a fact that is not true.

Replies from: timtyler, SilasBarta, AdeleneDawner, timtyler
comment by timtyler · 2009-10-21T07:56:11.116Z · LW(p) · GW(p)

Your disagreement apparently arises though using the term "value" in a different sense from me. If it helps you to understand, I am talking about what are sometimes called "ultimate values".

Most organisms don't update their values. They value the things evolution built into them - food, sex, warmth, freedom from pain, etc. Their values typically remain unchanged throughout their lives.

From my perspective, the dog's values aren't changed in your example. The dog merely associates barking with pain. The belief that a bark is likely to be followed by a poker prod is a belief, not a value. The dog still values pain-avoidance - just as it always did.

We actually have some theory that indicates that true values should change rarely. Organisms should protect their values - since changes to their values are seen as being very "bad" - in the context of the current values. Also, evolution wires in fitness-promoting values. These ideas help to explain why fixed values are actually extremely common.

Replies from: SilasBarta
comment by SilasBarta · 2009-10-21T14:47:33.980Z · LW(p) · GW(p)

Those are good points, but I still find your argument problematic.

First, do you know that dogs are capable of the abstract thought necessary to represent causality? You're saying that the dog has added the belief "bark causes pain", which combines with "pain bad".

That may be how a programmer would try to represent it, since you can rely on the computational power necessary to sweep through the search space quickly and find the "pain bad" module every time a "reason to bark" comes up. But is it good as a biological model? It requires the dog to indefinitely keep a concept of a prod in memory.

A simpler biological mechanism, consistent with the rest of neurobiology, would be to just lower the connection strengths that lead to the "barking" neuron so that it requires more activation of other "barking causes" to make it fire (and thus make the dog bark). I think that's a more reasonable model of how operant conditioning works in this context.

This mechanism, in turn, is better described as lowering the "shouldness" of barking, which is ambiguous with respect to whether it's a value or belief.

Replies from: timtyler, timtyler
comment by timtyler · 2009-10-21T15:24:13.197Z · LW(p) · GW(p)

It seems to be a common criticism of utility-based models that they no not map directly onto underlying biological hardware.

That is true - but it is not what such models are for in the first place. Nobody thinks that if you slice open an animal you will find a utility function, and some representation of utility inside.

The idea is more that you could build a functionally equivalent model which exhibited such an architecture - and then gain insight into the behaviour of the model by examining its utility function.

Replies from: SilasBarta
comment by SilasBarta · 2009-10-21T17:23:58.307Z · LW(p) · GW(p)

I'm concerned with the weaker constraint that the model must conceptually map to the biological hardware, and in this respect the utility-based model you gave doesn't work. There is no distinction, even conceptual, between values and beliefs: just synaptic weights from the causes-of-barking nodes, to the bark node.

Furthermore, the utility-based model does not give insight, because the "shortcuts" resulting from the neural hardware are fundamental to its operation. For example, the fact that it comes up with a quick, simple calculation affects how many options can be considered and therefore whether e.g. value transitivity will break down.

So the utility-based model is more complex than a neural network, and with worse predictive power, so it doesn't let you claim that its change in behavior resulted from beliefs rather than values.

Replies from: timtyler
comment by timtyler · 2009-10-21T19:13:23.406Z · LW(p) · GW(p)

Values are fixed, while many beliefs vary in response to sensory input.

You don't seem to appreciate the value of a utility based analysis.

Knowing that an animal likes food and sex, and doesn't like being hit provides all kinds of insights into its behaviour.

Such an analysis is much simpler than a neural network is, and it has the advantage that we can actually build and use the model - rather than merely dream about doing so in the far future, when computers are big enough to handle it, and neuroscience has advanced sufficiently.

Replies from: SilasBarta, RobinZ
comment by SilasBarta · 2009-10-21T23:36:46.783Z · LW(p) · GW(p)

That's not a very fair comparison! You're looking at the most detailed version of a neural network (which I would reject as a model anyway for the very reason that it needs much more resources than real brains to work) and comparing it to a simple utility-based model, and then sneaking in your intuitions for the UBM, but not the neural network (as RobinZ noted).

I could just as easily turn the tables and compare the second neural network here to a UDT-like utility-based model, where you have to compute your action in every possible scenario, no matter how improbable.

Anyway, I was criticizing utility-based models, in which you weight the possible outcomes by their probability. That involves a lot more than the vague notion that an animal "likes food and sex".

Of course, as you note, even knowing that it likes food and sex gives some insight. But it clearly breaks down here: the dog's decision to bark is made very quickly, and having to do an actual human-insight-free, algorithmic computation of expected utilities, involving estimates of their probabilities, takes way too long to be a realistic model. The shortcuts used in a neural network skew the dog's actions is predictable ways, showing them to be a better model, and showing the value/belief distinction to break down.

Replies from: timtyler
comment by timtyler · 2009-10-22T06:14:39.987Z · LW(p) · GW(p)

I am still not very sympathetic to the idea that neural network models are simple. They include the utility function and all the creature's beliefs.

A utility based model is useful - in part - since it abstracts those beliefs away.

Plus neural network models are renowned for being opaque and incomprehensible.

You seem to have some strange beliefs in this area. AFAICS, you can't make blanket statements like: neural-net models are more accurate. Both types of model can represent observed behaviour to any desired degree of precision.

Replies from: SilasBarta
comment by SilasBarta · 2009-10-22T13:54:30.769Z · LW(p) · GW(p)

You're using a narrower definition of neural network than I am. Again, refer to the last link I gave for an example of a simple neural network, which is equal to or less than the complexity of typical expected utility models. That NN is far from being opaque and incomprehensible, wouldn't you agree?

I am still not very sympathetic to the idea that neural network models are simple. They include the utility function and all the creature's beliefs.

No, they just have activation weights, which don't (afaict) distinguish between beliefs and values, or at least, don't distinguish between "barking causes a prod which is bad" and "barking isn't as good (or perhaps, as 'shouldish')".

A utility based model is useful - in part - since it abstracts those beliefs away.

The UBMs discussed in this context (see TL post) necessarily include probability weightings, which are used to compute expected utility, which factors in the tradeoffs between probability of an event and its utility. So it's certainly not abstracting those beliefs away.

Plus, you've spent the whole conversation explaining why your UBM of the dog allows you to classify the operant conditioning (of prodding the dog when it barks) as changing it's beliefs and NOT its values. Do you remember that?

comment by RobinZ · 2009-10-21T20:30:21.581Z · LW(p) · GW(p)

Correct me if I'm wrong, but it's only simpler if you already have a general-purpose optimizer ready to hand - in this case, you.

Replies from: timtyler
comment by timtyler · 2009-10-21T20:51:29.893Z · LW(p) · GW(p)

You have to have complicated scientists around to construct any scientific model - be it utility-based or ANN.

Since we have plenty of scientists around, I don't see much point in hypothesizing that there aren't any.

You seem to be implying that the complexity of utility based models lies in those who invent or use them. That seems to be mostly wrong to me: it doesn't matter who invented them, and fairly simple computer programs can still use them.

Replies from: RobinZ
comment by RobinZ · 2009-10-21T23:31:55.182Z · LW(p) · GW(p)

If you've seen it work, I'll take your word for it.

comment by timtyler · 2009-10-21T15:57:26.186Z · LW(p) · GW(p)

Incidentally, I did not claim that dogs can perform abstract thinking - I'm not clear on where you are getting that idea from.

Replies from: SilasBarta
comment by SilasBarta · 2009-10-21T17:25:37.601Z · LW(p) · GW(p)

You said that the dog had a belief that a bark is always followed by a poker prod. This posits separate entities and a way that they interact, which looks to me like abstract thought.

Replies from: timtyler
comment by timtyler · 2009-10-21T19:24:37.936Z · LW(p) · GW(p)

The definition of "abstract thought" seems like a can of worms to me.

I don't really see why I should go there.

comment by SilasBarta · 2009-10-21T03:32:27.599Z · LW(p) · GW(p)

Hm, I never before realized that operant conditioning is a blurring of the beliefs and values -- the new frequency of barking can be explained either by a change of the utility of barking, or by a change in the belief about what will result from the barking.

Replies from: timtyler
comment by timtyler · 2009-10-21T08:02:27.806Z · LW(p) · GW(p)

IMO, "a blurring of beliefs and values" is an unhelpful way of looking at what happens. It is best to consider an agent as valuing freedom from pain, and the association between barking and poker prods to be one of its beliefs.

If you have separated out values from beliefs in a way that leads to frequently updated values, all that means is that you have performed the abstraction incorrectly.

comment by AdeleneDawner · 2009-10-21T03:25:22.687Z · LW(p) · GW(p)

Or the dog values not being in pain more than it values barking or mating...

comment by timtyler · 2009-10-21T08:54:23.139Z · LW(p) · GW(p)

Because a comment is down-voted, that doesn't mean it is incorrect.

This particular comment implicitly linked people's values to their reproductive success. People don't like to hear that they are robot vehicles built to propagate their genes. It offends their sense of self-worth. Their mental marketing department spends all day telling everyone what an altruistic and nice person they are - and they repeat it so many times that they come to believe it themselves. That way their message comes across with sincerity. So: the possibility of biology underlying their motives is a truth that they often want to bury - and place as far out of sight as possible.

Replies from: MichaelBishop
comment by Mike Bishop (MichaelBishop) · 2009-10-24T15:41:22.393Z · LW(p) · GW(p)

While we can never escape our biology entirely, I dispute any suggestion that the selfish gene is always the best level of abstraction, or best model, for human behavior. I assume you agree even though that did not come across in this paragraph.

Replies from: timtyler
comment by timtyler · 2009-10-25T16:49:09.614Z · LW(p) · GW(p)

Humans behaviour is often illuminated by the concept of memes. Humans are also influenced by the genes of their pathogens (or other manipulators). If you cough or sneeze, that behaviour is probably not occurring since it benefits you.

Similarly with cancer or back pain - not everything is an adaptation.

comment by wedrifid · 2009-10-23T04:20:18.787Z · LW(p) · GW(p)

This assumption is central to establishing the mathematical structure of expected utility maximization, where you value each possible world separately using the utility function, then take their weighted average. If your preferences were such that A&C > B&C but A&D < B&D, then you wouldn’t be able to do this.

I can imagine having preferences that don't value each possible world separately. I can also imagine doing other things to my utility function than maximising expectation. For example, if I maximised the top quartile of expected values then I may choose to engage in practices analogous to quantum suicide. That I prefer, in principle, to maximise expected utility is itself a value. It is a value I that I expect to see in most successful agents, for fundamental reasons.

comment by [deleted] · 2009-10-21T17:39:24.704Z · LW(p) · GW(p)

Here, have a mathematical perspective that conflates beliefs and values:

Suppose that some agent is given a choice between A and B. A is an apple. B is an N chance of a banana, otherwise nothing. The important thing here is the ambivalence equation: iff U(apple) = N*U(banana), the agent is ambivalent between the apple and the banana. Further suppose that N is 50%, and the agent likes bananas twice as much as it likes apples. In this case, at least, the agent might as well modify itself to believe that N is 20% and to like bananas five times as much as apples.

Now, doing this might result in inconsistencies elsewhere, but I'm guessing that a rational agent will be able to apply transformations to its beliefs and values--but only both simultaneously--so as to preserve expected utility given actions.

I think a more concrete example of the beliefs-values meld is anthropic reasoning. Take the presumptuous friend: you, the presumptuous philosopher's presumptuous friend, have just been split into 1,000,001 branches, and each of those branches has been placed in a hotel room, 1,000,000 rooms being in one hotel, and the other room being in another hotel. Is the probability that you're in the small hotel 50%, negligible, or something in between? Well, that depends: if you care about each of your branches equally, it's negligible; if you care about each hotel equally, it's 50%.

comment by whpearson · 2009-10-20T22:30:05.109Z · LW(p) · GW(p)

I think values (in a finite agent), also need to have some role in what beliefs "should" be stored/updated/remembered. Of course in theories which don't constrain the agents computational ability this isn't needed.

comment by SilasBarta · 2009-10-20T18:02:34.906Z · LW(p) · GW(p)

I dispute your premise: what makes you so sure people do decompose their thoughts into beliefs and values, and find these to be natural, distinct categories? Consider the politics as mind-killer phenomenon. That can be expressed as, "People put your words into a broader context of whether they threaten their interests, and argue for or against your statements on that basis."

For example, consider the difficulty you will have communicating your position if you believe both a) global warming is unlikely to cause any significant problems in the business-as-usual scenario, b) high taxes on CO2 emissions should be levied. (e.g., you believe it's a good idea as an insurance policy and can be done in a way that blocks most of the economic damage)

(Yes, I had to use a present example to make the reactions easier to imagine.)

The "ought" is so tightly coupled to the "is", that in any case where the "ought" actually matters, the "is" comes along for the ride.

Note: this is related to the problem I had with the exposition of could/would/should agents: if you say humans are CSAs, what's an example of an intelligent agent that isn't?

Replies from: thomblake, DanArmak
comment by thomblake · 2009-10-20T20:43:49.695Z · LW(p) · GW(p)

I'm confused about this. Consider these statements:

A. "I believe that my shirt is red."
B. "I value cheese."

Are you claiming that:

  1. People don't actually make statements like A
  2. People don't actually make statements like B
  3. A is expressing the same sort of fact about the world as B
  4. Statements like A and B aren't completely separate; that is, they can have something to do with one another.

If you strictly mean 1 or 2, I can construct a counterexample. 3 is indeed counterintuitive to me. 4 seems uncontroversial (the putative is/ought problem aside)

Replies from: SilasBarta
comment by SilasBarta · 2009-10-20T22:01:45.008Z · LW(p) · GW(p)

If I had to say, it would be a strong version of 4: in conceptspace, people naturally make groupings that put is- and ought-statements together. But looking back at the post, I definitely have quite a bit to clarify.

When I refer to what humans do, I'm trying to look at the general case. Obviously, if you direct someone's attention to the issue of is/ought, then they can break down thoughts into values and beliefs without much training. However, in the absence of such a deliberate step, I do not think people normally make a distinction.

I'm reminded of the explanation in pjeby's earlier piece: people instinctively put xml-tags of "good" or "bad" onto different things, blurring the distinction between "X is good" and "Y is a reason to deem X good". That is why we have to worry about the halo effect, where you disbelieve everything negative about something you value, even if such negatives are woefully insufficient to justify not valuing it.

From the computational perspective, this can be viewed as a shortcut to having to methodically analyze all the positives and negatives of any course of action, and getting stuck thinking instead of acting. But if this is how the mind really works, it's not really reducible to a CSA, without severe stretching of the meaning.

comment by DanArmak · 2009-10-20T20:35:29.582Z · LW(p) · GW(p)

Seconded. Sometimes I don't even feel I have fully separate beliefs and values. For instance, I'm often willing to change my beliefs to achieve my values (e.g., by believing something I have no evidence for, to become friends with other people who believe it - and yes, ungrounded beliefs can be adopted voluntarily to an extent.)

Replies from: SforSingularity
comment by SforSingularity · 2009-10-26T07:53:07.542Z · LW(p) · GW(p)

ungrounded beliefs can be adopted voluntarily to an extent.

I cannot do this, and I don't understand anyone who can. If you consciously say "OK, it would be really nice to believe X, now I am going to try really hard to start believing it despite the evidence against it", then you already disbelieve X.

Replies from: DanArmak
comment by DanArmak · 2009-10-26T20:39:19.206Z · LW(p) · GW(p)

I already disbelieve X, true, but I can change that. Of course it doesn't happen in a moment :-)

Yes, you can't create that feeling of rational knowledge about X from nothing. But if you can retreat from rationality - to where most people live their lives - and if you repeat X often enough, and you have no strongly emotional reason not to believe X, and your family and peers and role models all profess X, and X behaves like a good in-group distinguishing mark - then I think you have a good chance of coming to believe X. The kind of belief associated with faith and sports team fandom.

It's a little like the recent thread where someone, I forget who, described an (edit: hypothetical) religious guy who when drunk confessed that he didn't really believe in god and was only acting religious for the social benefits. Then people argued that no "really" religious person would honestly say that, and other people argued that even if he said that what does it mean if he honestly denies it whenever he's sober?

In the end I subscribe to the "PR consciousness" theory that says consciousness functions to create and project a self-image that we want others to believe in. We consciously believe many things about ourselves that are completely at odds with how we actually behave and the goals we actually seek. So it would be surprising if we couldn't invoke these mechanisms in at least some circumstances.

Replies from: Douglas_Knight
comment by Douglas_Knight · 2009-10-27T00:59:31.076Z · LW(p) · GW(p)

someone, I forget who, described a religious guy who when drunk confessed that he didn't really believe in god and was only acting religious for the social benefits.

generalizing from fictional evidence

Replies from: DanArmak, RobinZ, SilasBarta
comment by DanArmak · 2009-10-27T09:12:44.202Z · LW(p) · GW(p)

When I wrote that I was aware that it was a fictional account deliberately made up to illustrate a point. I didn't mention that, though, so I created fictional evidence. Thanks for flagging this, and I should be more careful!

comment by RobinZ · 2009-10-27T01:07:19.355Z · LW(p) · GW(p)

Worse: fictional evidence flagged as nonfictional -- like Alicorn's fictional MIT classmates that time.

Replies from: Alicorn
comment by Alicorn · 2009-10-27T01:10:35.822Z · LW(p) · GW(p)

My what now? I think that was someone else. I don't think I've been associated with MIT till now.

MIT not only didn't accept me when I applied, they didn't even reject me. I never heard back from them yea or nay at all.

Replies from: Yvain, RobinZ
comment by Scott Alexander (Yvain) · 2009-10-27T01:15:40.815Z · LW(p) · GW(p)

That was me.

Of course, irony being what it is, people will now flag the Alicorn - MIT reference as nonfictional, and be referring to Alicorn's MIT example for the rest of LW history :)

Replies from: RobinZ
comment by RobinZ · 2009-10-27T01:31:34.337Z · LW(p) · GW(p)

Attempting to analyze my own stupidity, I suspect my confusion came from (1) both Alicorn and Yvain being both high-karma contributors and (2) Alicorn's handle coming more readily to mind, both because (a) I interacted more with her and (b) the pronunciation of "Alicorn" being more obvious than that of "Yvain".

In other words, I have no evidence that this was anything other than an ordinary mistake.

Replies from: Alicorn
comment by Alicorn · 2009-10-27T01:35:48.640Z · LW(p) · GW(p)

I've been imagining "Yvain" to be pronounced "ee-vane". I'd be interested in hearing a correction straight from the ee-vane's mouth if this is not right, though ;) I've heard people mispronounce "Alicorn" on multiple occasions.

Replies from: wedrifid, RobinZ
comment by wedrifid · 2009-10-27T02:27:49.829Z · LW(p) · GW(p)

I've heard people mispronounce "Alicorn" on multiple occasions.

You mean Alicorn is a real name? I had assumed a combination of Alison and Unicorn, with symbolic implications beyond my ken.

I've been imagining "Yvain" to be pronounced "ee-vane".

"Ye-vane" here, with the caveat that I was quite confident that it was way off.

Replies from: Alicorn
comment by Alicorn · 2009-10-27T02:34:19.419Z · LW(p) · GW(p)

No, it's not a real name (as far as I know). It's a real word. It means a unicorn's horn, although there are some modern misuses mostly spearheaded by Piers Anthony (gag hack cough).

Replies from: wedrifid
comment by wedrifid · 2009-10-27T02:53:50.906Z · LW(p) · GW(p)

Ahh. And I've been going about calling them well, unicorn horns all these years!

comment by RobinZ · 2009-10-27T01:44:11.501Z · LW(p) · GW(p)

I've been saying "al-eh-corn" in my mental consciousness. Also "ee-vane", which suggests my problem being less "Yvain is hard to pronounce" than "Yvain doesn't look like the English I grew up speaking".

Incidentally, I can't remember how to pronounce Eliezer. I saw him say it at the beginning of a Bloggingheads video and it was completely different from my naive reading.

Replies from: Alicorn, arundelo
comment by Alicorn · 2009-10-27T01:49:38.011Z · LW(p) · GW(p)

"Alicorn" is pronounced just like "unicorn", except that the "yoon" is replaced with "al" as in "Albert" or "Alabama". So the I is an "ih", not an "eh", but you can get away with an undifferentiated schwa.

Replies from: RobinZ
comment by RobinZ · 2009-10-27T01:56:33.815Z · LW(p) · GW(p)

Thanks!

(I think that's how I was saying it, actually - I wasn't sure how to write the second syllable.)

comment by arundelo · 2009-10-27T01:54:03.874Z · LW(p) · GW(p)

ell-ee-EZZ-er (is how I hear it).

comment by RobinZ · 2009-10-27T01:18:33.329Z · LW(p) · GW(p)

*checks*

Yvain's fictional MIT classmates.

I swear that wasn't on purpose.

comment by SilasBarta · 2009-10-27T02:36:53.144Z · LW(p) · GW(p)

What's fictional about that?

Ready to pony up money for a bet that I can't produce a warm body meeting that description?

Replies from: RobinZ
comment by RobinZ · 2009-10-27T02:42:47.828Z · LW(p) · GW(p)

I prefer not to gamble, but just to satisfy my own curiosity: what would the controls be on such a bet? Presumably you would have to prove to Knight's satisfaction that your unbelieving belief-signaler was legitimately thus.

Replies from: SilasBarta
comment by SilasBarta · 2009-10-27T03:09:23.903Z · LW(p) · GW(p)

I think my evidence is strong enough I can trust Douglas_Knight's own intellectual integrity.

Replies from: Douglas_Knight
comment by Douglas_Knight · 2009-10-27T05:25:31.367Z · LW(p) · GW(p)

I think my evidence is strong enough I can trust Douglas_Knight's own intellectual integrity.

Huh. My last couple of interactions with you, you called me a liar.

Replies from: SilasBarta, SilasBarta
comment by SilasBarta · 2009-10-27T15:26:08.151Z · LW(p) · GW(p)

Okay, I found what I think you're referring to. Probably not my greatest moment here, but Is that really something you want sympathy for? Here's the short version of what happened.

You: If you think your comment was so important, don't leave it buried deep in the discussion, where nobody can see it.

Me: But I also linked to it from a more visible place. Did you not know about that?

You: [Ignoring previous mischaracterization] Well, that doesn't solve the problem of context. I clicked on it and couldn't understand it, and it seemed boring.

Me: Wait, you claim to be interested in a solution, I post a link saying I have one, and it's too much of a bother to read previous comments for context? That doesn't make sense. Your previous comment implies you didn't know about the higher link. Don't dig yourseelf deeper by covering it up.

Replies from: Douglas_Knight
comment by Douglas_Knight · 2009-10-27T16:30:28.806Z · LW(p) · GW(p)

Oh, yeah, I'd forgotten that one. Actually, I was thinking of the following week.

Is that really something you want sympathy for?

I just want you to go away. I was hoping that reminding you that you don't believe me would discourage you from talking to me.

Replies from: SilasBarta
comment by SilasBarta · 2009-10-27T16:48:01.850Z · LW(p) · GW(p)

Oh, yeah, I'd forgotten that one. Actually, I was thinking of the following week.

That's not calling you a liar. That's criticizing the merit of your argument. There's a difference.

Replies from: wedrifid
comment by wedrifid · 2009-10-27T17:18:41.664Z · LW(p) · GW(p)

That's not calling you a liar. That's criticizing the merit of your argument. There's a difference.

The link provided by Douglas seems to suggest that Douglas's accusation is false (as well as ineffective).

ET:S/petty/ineffective/

Replies from: Douglas_Knight
comment by Douglas_Knight · 2009-10-27T18:20:50.312Z · LW(p) · GW(p)

Would you mind elaborating on your take on that thread? What's of most interest to me is what you think I meant, but I'm also interested in whether you'd say that Silas called Zack a liar.

Replies from: SilasBarta
comment by SilasBarta · 2009-10-27T18:43:46.487Z · LW(p) · GW(p)

I'm also interested in whether you'd say that Silas called Zack a liar.

Let's go back a few steps. You said that in your "last few interactions" with me, I called you a liar. You later clarified that you were thinking of this discussion. But I didn't deny calling Zack a liar in that discussion; I denied calling you a liar. So why are you suddenly acting like your original claim was about whether I called Zack a liar?

(In any case, it wasn't just "Zack, you liar". My remark was more like, "this is what you claimed, this is why it's implausible, this is why your comments are hindering the discussion, please stop making this so difficult by coming up with ever-more-convoluted stories.")

Are you and Zack the same person?

Considering that the earlier discussion was about whether you can arbitrarily redefine yourself as a different person, maybe Zack/Douglas are just taking the whole idea a little too seriously! :-P

(And in a show of further irony, that would be just the kind of subtle point that Zack and [?] Douglas, severely overestimating its obviousness, were defending in the thread!)

Replies from: Douglas_Knight
comment by Douglas_Knight · 2009-10-27T18:54:01.418Z · LW(p) · GW(p)

Are you and Zack the same person?

No.

I apologize to third parties for the poor timing of my deletion of the above comment. It was really addressed to wedrifid and broadcasting it was petty, though not as petty as the excerpt looks.

Replies from: SilasBarta
comment by SilasBarta · 2009-10-27T19:03:40.786Z · LW(p) · GW(p)

Alright, well, good luck "getting the goods" on ol' Silas! Just make sure not to get your claims mixed up again...

comment by SilasBarta · 2009-10-27T13:57:37.357Z · LW(p) · GW(p)

Well, what possessed you to lie to me? ;-)

j/k, j/k, you're good, you're good.

A link would be nice though.

And I believe that, even taking into account any previous mistrust I might have had of you, I think my evidence is still strong enough that I can trust you consider it conclusive.

comment by Stuart_Armstrong · 2009-10-23T14:40:13.206Z · LW(p) · GW(p)

Maybe these are to do with differences across individuals. My beliefs/values may be mashed togather and impossible to seperate, but I expect other people's beliefs to mirror my own more closely than their values do.

comment by Kyre · 2009-10-21T06:15:53.685Z · LW(p) · GW(p)

Because it's much easier to use beliefs shorn of values as building blocks in a machine that does induction, inference, counterfactual reasoning, planning etc compared to belief-values that are all tied up together.

Sea slugs and Roombas don't have the beliefs/values separation it because the extra complexity isn't worth it. Humans have it to some degree and rule the planet. AIs might have even more success.

comment by Furcas · 2009-10-20T22:48:33.080Z · LW(p) · GW(p)

Some of this is going over my head, but...

I think you need to specify if you're talking about terminal values or instrumental values.

There's obviously a big difference between beliefs and terminal values. Beliefs are inputs to our decision-making processes, and terminal values may be as well. However, while beliefs are outputs of the belief-making processes whose inputs are our perceptions, terminal values aren't the output of any cognitive process, or they wouldn't be terminal.

As for instrumental values, well, yes, they are beliefs about the best values that one should have in order to reach the terminal goals, so they would be outputs of the same or similar processes that create beliefs. Or at least, they start out that way; some or all of them end up becoming terminal values.

comment by SilasBarta · 2009-10-20T18:07:11.706Z · LW(p) · GW(p)

I think I tried to solve a similar problem before: that of looking at the simplest possible stable control system and seeing how I can extract the system's "beliefs" and "values" that result in it remaining stable. Then, see if I can find a continuous change between the structure of that system, and a more complex system, like a human.

For example, consider the simple spring-mass-damper system. If you move it from its equlibrium position xe, it will return. What do the concepts of "belief" and "value" map onto here? For beliefs, I used the concept of mutual information: what about the system could you look at to learn whether the mass is not at xe? How does the system know it's not at xe?

The information is contained in the force the spring exerts. However, this is also the determinant of which direction it moves the spring, it's "value". So it looks like the beliefs and values are fully-mixed: the same thing that tells you what it believes, tells you what it does. In that case, at what point, in the structural transition from the spring to intelligent agents, does the distinction between values and beliefs start to form, if at all?

Incidently, I only just now read Vladimir_Nesov's post because previously I hadn't bothered to make the equations readable, since they don't render properly in the browser I used. "Beware trivial inconveniences", indeed!

(Separate post because of the different issues and to avoid tl;dr.)

comment by SforSingularity · 2009-10-20T17:07:51.855Z · LW(p) · GW(p)

Is it possible that the dichotomy between beliefs and values is just an accidental byproduct of our evolution, perhaps a consequence of the specific environment that we’re adapted to, instead of a common feature of all rational minds?

In the normal usage, "mind" implies the existence of a distinction between beliefs and values. In the LW/OB usage, it implies that the mind is connected to some actuators and sensors which connect to an environment and is actually doing some optimization toward those values. Certainly "rational mind" entails a beliefs/values separation.

But suppose we abandon the beliefs/values separation: what properties do we have left? Is the concept "mind without a beliefs/values separation" Simply the concept "thing"?

Replies from: Wei_Dai, SforSingularity
comment by Wei Dai (Wei_Dai) · 2009-10-20T17:46:42.044Z · LW(p) · GW(p)

But suppose we abandon the beliefs/values separation: what properties do we have left? Is the concept "mind without a beliefs/values separation" Simply the concept "thing"?

An agent using UDT doesn't necessarily have a beliefs/values separation, but still has the properties of preferences and decision making. Or at least, it only has beliefs about mathematical facts, not about empirical facts. Maybe I should have made it clear that I was mainly talking about empirical beliefs in the post.

Replies from: timtyler, Vladimir_Nesov, SforSingularity
comment by timtyler · 2009-10-20T19:09:55.781Z · LW(p) · GW(p)

How, then, would you describe its representation of empirical information - if not as "beliefs"?

comment by Vladimir_Nesov · 2009-10-20T20:03:08.764Z · LW(p) · GW(p)

Not quite true: state of knowledge corresponds to beliefs. It's values that don't update (but in expected utility maximization that's both utility and prior). Again, it's misleading to equate beliefs with prior and forget about the knowledge (event that conditions the current state).

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-10-21T11:15:36.722Z · LW(p) · GW(p)

Yes, I agree we can interpret UDT as having its own dichotomy between beliefs and values, but the dividing line looks very different from how humans divide between beliefs and values, which seems closer to the probability/utility divide.

Replies from: SilasBarta, timtyler
comment by SilasBarta · 2009-10-21T14:52:20.297Z · LW(p) · GW(p)

UDT is invariant with respect to what universe it's actually in. This requires it to compute over infinite universes and thus have infinite computing power. It's not hard to see why it's going to break down as a model of in-universe, limted beings.

comment by timtyler · 2009-10-21T11:33:13.886Z · LW(p) · GW(p)

What do you mean? It has a utility function just like most other decision theories do. The preferences are represented by the utility function.

comment by SforSingularity · 2009-10-20T18:48:40.711Z · LW(p) · GW(p)

An agent using UDT doesn't necessarily have a beliefs/values separation,

I am behind on your recent work on UDT; this fact comes as a shock to me. Can you provide a link to a post of yours/provide an example here making clear that UDT doesn't necessarily have a beliefs/values separation? Thanks.

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-10-20T19:56:51.489Z · LW(p) · GW(p)

Suppose I offer you three boxes and ask you to choose one. The first two are transparent, free, and contains an apple and an orange, respectively. The third is opaque, costs a penny, and contains either an apple or an orange, depending on a coin flip I made. Under expected utility maximization, there is no reason for you to choose the third box, regardless of your probability function and utility function. Under UDT1, you can choose the third box, by preferring to and as the outcomes of world programs P1 and P2. In that case, you can't be said to have a belief about whether the real world is P1 or P2.

Replies from: timtyler, thomblake, thomblake
comment by timtyler · 2009-10-20T21:05:20.662Z · LW(p) · GW(p)

This example seems unclear. Are you seriously claiming utility maximisation can't prefer a randomised outcome in an iterated situation? If so, you take this "independence" business much too far.

Utility maximising agents can do things like prefer a diverse diet. They simply do not have to prefer either apples or oranges - thereby winding up with vitamin and mineral deficiencies. It is trivial to create a utility function which exhibits fruit preferences which depend on what you have eaten most recently.

Replies from: pengvado
comment by pengvado · 2009-10-20T22:01:10.882Z · LW(p) · GW(p)

Randomization only maximizes diversity if you have to make decisions under amnesia or coordinate without communication or some similar perverse situation. In any normal case, you're better off choosing a deterministic sequence that's definitely diverse, rather than leaving it to randomness and only probably getting a diverse set of outcomes.

Replies from: timtyler
comment by timtyler · 2009-10-21T08:06:36.536Z · LW(p) · GW(p)

Sure - but that seems rather tangential to the main point here.

The options were , - or a more expensive random choice. A random diet may not be perfect - but it was probably the best one on offer in the case of this example.

Replies from: RobinZ
comment by RobinZ · 2009-10-21T14:48:00.286Z · LW(p) · GW(p)

If the agent already has a penny (which they must if they can afford to choose the third box), they could just flip the penny to decide which of the first two boxes to take and save themselves the money.

Unless you're being a devil's advocate, I don't see any reason to justify a completely rational agent choosing the random box.

Replies from: timtyler
comment by timtyler · 2009-10-21T15:19:26.096Z · LW(p) · GW(p)

What - never? Say they can only make the choice once - and their answer determines which box they will get on all future occasions.

Replies from: pengvado, RobinZ
comment by pengvado · 2009-10-21T16:01:56.934Z · LW(p) · GW(p)

Then choice C isn't a random mixture of choice A and choice B.

Preferring that there be randomness at a point where you otherwise wouldn't get a decision at all, is fine. What doesn't happen is preferring one coin-flip in place of one decision.

comment by RobinZ · 2009-10-21T15:26:54.730Z · LW(p) · GW(p)

Not to be crass, but given the assumption that Wei_Dai is not saying something utterly asinine, does your interpretation of the hypothetical actually follow?

Replies from: timtyler
comment by timtyler · 2009-10-21T15:36:49.704Z · LW(p) · GW(p)

Hang on! My last comment was a reply to your question about when it could be rational to select the third box. I have already said that the original example was unclear. It certainly didn't suggest an infinite sequence - and I wasn't trying to suggest that.

The example specified that choosing the third box was the correct answer - under the author's own proposed decision theory. Surely interpretations of what it was supposed to mean should bear that in mind.

Replies from: RobinZ
comment by RobinZ · 2009-10-21T16:37:46.992Z · LW(p) · GW(p)

I don't believe we're actually arguing about anything worth caring about. My understanding was that Wei_Dai was illustrating a problem with UDT1 - in which case a single scenario in which UDT1 gives an unambiguously wrong answer suffices. To disprove Wei_Dai's assertion requires demonstrating that no scenario of the kind proposed makes UDT1 give the wrong answer, not showing that not every scenario of the kind proposed makes UDT1 give the wrong answer.

Replies from: timtyler
comment by timtyler · 2009-10-21T18:45:54.036Z · LW(p) · GW(p)

Are you sure you are taking the fact that he is UDT's inventor and biggest fan into account? He certainly didn't claim that he was illustrating a problem with UDT.

Replies from: RobinZ
comment by RobinZ · 2009-10-21T20:24:48.214Z · LW(p) · GW(p)

...you're right, I'm misreading. I'll shut up now.

comment by thomblake · 2009-10-21T14:02:13.913Z · LW(p) · GW(p)

Okay, let's see if I have this straight - you're assuming:

  1. the axiom of independence is necessary for expected utility theory
  2. losing a penny represents some negative amount of utility
  3. one's utility function can't include terms for "the outcomes of world programs" under expected utility theory
comment by thomblake · 2009-10-20T20:23:40.646Z · LW(p) · GW(p)

Under expected utility maximization, there is no reason for you to choose the third box, regardless of your probability function and utility function. Under UDT1, you can choose the third box, by preferring to and as the outcomes of world programs P1 and P2. In that case, you can't be said to have a belief about whether the real world is P1 or P2.

You lost me. Is 'apples' supposed to be plural? Can you really not choose the third box regardless of utility function? What if you prefer things that came in opaque boxes?

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-10-20T20:36:51.027Z · LW(p) · GW(p)

It's not supposed to be plural. Fixed.

The opaque box was a way of framing the problem, and not part of the problem itself, which is supposed to be about your preferences for apples and oranges. I can specify the problem in terms of three identical buttons that you can press instead.

Replies from: timtyler, thomblake
comment by timtyler · 2009-10-20T20:56:58.132Z · LW(p) · GW(p)

Those buttons will probably not be absolutely identical - since they will be in different spatial positions relative to each other. So an agent operating under expected utility maximization might still prefer (say) pressing the right-most button.

Real-world utility functions can literally prefer anything you can specify.

Replies from: AdeleneDawner
comment by AdeleneDawner · 2009-10-21T02:43:30.048Z · LW(p) · GW(p)

I'm actually an example of this - where I don't otherwise care, I will pick the third option, or the option that's related to the number three in some way (preferably related to powers of three, but multiples of 9 are preferred over other multiples of three as well). If I didn't care very much about apples vs. oranges, I'd be fairly likely to pay a penny for the third box/button/whatever. I also know two people who feel similarly about the number 8.

In tangentially related news, I'm sad that I'm turning 28 next month. Yes, I know I'm strange.

Replies from: cousin_it, Eliezer_Yudkowsky, Eliezer_Yudkowsky
comment by cousin_it · 2009-10-21T12:15:36.557Z · LW(p) · GW(p)

You're not strange. (Sorry if that sounded offensive, I didn't mean to!) I'm likewise sad that I just turned 27. I was always the youngest in school and university, graduating with honors at 20. Getting closer to 30 now. "Where are you now golden boy, where is your famous golden touch?" Or this: "Tired of lying in the sunshine staying home to watch the rain, you are young and life is long and there is time to kill today."

Replies from: AdeleneDawner
comment by AdeleneDawner · 2009-10-21T12:18:02.506Z · LW(p) · GW(p)

I'm not sad that I'm closer to 30. 30's cool, it's a multiple of three. I'm sad that the next time my age will be a power of three won't be 'till I'm 81.

Replies from: cousin_it
comment by cousin_it · 2009-10-21T12:21:38.766Z · LW(p) · GW(p)

I obviously can't read. Let that comment stand as a monument to my stupidity =)

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2009-10-21T19:37:31.953Z · LW(p) · GW(p)

You mean your second annual 27th birthday?

Replies from: AdeleneDawner
comment by AdeleneDawner · 2009-10-22T00:03:50.734Z · LW(p) · GW(p)

Cute idea, but I value signaling an interest in accuracy/not coming off like a loon over being associated with the number 3. The former actually affect things in my life.

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2009-10-21T03:10:30.668Z · LW(p) · GW(p)

Why is that strange?

comment by thomblake · 2009-10-20T20:47:31.415Z · LW(p) · GW(p)

I can specify the problem in terms of three identical buttons that you can press instead.

Please do, if you think that would make the problem clearer. The piece I'm not seeing is where UDT1 lets you choose something that expected utility does not. Does expected utility usually not allow you to have states of the world in your utility function?

comment by SforSingularity · 2009-10-20T17:43:20.384Z · LW(p) · GW(p)

One possible response here: We could consider simple optimizers like amoeba or Roomba vacuum cleaners as falling into the category: "mind without a clear belief/values distinction"; they definitely do a lot of signal processing and feature extraction and control theory, but they don't really have values. The Roomba would happily sit with wheels lifted off the ground thinking that it was cleaning a nonexistent room.

Replies from: Matt_Simpson, timtyler, DanArmak
comment by Matt_Simpson · 2009-10-20T19:25:21.354Z · LW(p) · GW(p)

Isn't this just a case of the values the Roomba was designed to maximize being different from the values it actually maximizes? Consider the following:

We could consider simple optimizers like humans as falling into the category: "mind without a clear belief/values distinction"; they definitely do a lot of signal processing and feature extraction and control theory, but they don't really have values. The human would happily have sex with a condom thinking that it was maximizing its fitness.

i.e. Roombas are program executers, not cleanliness maximizers.

I suppose the counter is that humans don't have a clear belief/values distinction.

comment by timtyler · 2009-10-21T11:46:23.835Z · LW(p) · GW(p)

The purpose of a Roomba is to clean rooms. Clean rooms are what it behaves as though it "values" - whereas its "beliefs" would refer to things like whether it has just banged into a wall.

There seems to be little problem in modelling the Roomba as an expected utility maximiser - though it is a rather trivial one.

Replies from: Richard_Kennaway
comment by Richard_Kennaway · 2009-10-21T14:20:37.696Z · LW(p) · GW(p)

The purpose of a Roomba is to clean rooms.

That is only true if understood to mean the purpose which the user of a Roomba is using it to achieve, or the purpose of its designers in designing it. It is not necessarily the Roomba's own purpose, the thing the Roomba itself is trying to achieve. To determine the Roomba's own purposes, one must examine its internal functioning, and discover what those purposes are; or, alternatively, to conduct the Test For The Controlled Variable. This is straightforward and unmysterious.

I have a Roomba. My Roomba can tell if some part of the floor is unusually dirty (by an optical sensor in the dust intake, I believe), and give that area special attention until it is no longer filthy. Thus, it has a purpose of eliminating heavy dirt. However, beyond that it has no perception of whether the room is clean. It does not stop when the room is clean, but when it runs out of power or I turn it off. Since it has no perception of a clean room, it can have no intention of achieving a clean room. I have that intention when I use it. Its designers have the intention that I can use the Roomba to achieve my intention. But the Roomba does not have that intention.

A Roomba with a more sensitive detector of dust pickup (and current models might have such a sensor -- mine is quite old) could indeed continue operation until the whole room was clean. The Roomba's physical sensors sense only a few properties of its immediate environment, but it would be able to synthesize from those a perception of the whole room being clean, in terms of time since last detection of dust pickup, and its algorithm for ensuring complete coverage of the accessible floor space. Such a Roomba would have cleaning the whole room as its purpose. My more primitive model does not.

This is elementary stuff that people should know.

There seems to be little problem in modelling the Roomba as an expected utility maximiser - though it is a rather trivial one.

Little or large, you can't do it by handwaving like that. A model of a Roomba as a utility maximiser would (1) state the utility function, and (2) demonstrate how the physical constitution of the Roomba causes it to perform actions which, from among those available to it, do in fact maximise that function.But I suspect you have not done these.

Replies from: timtyler
comment by timtyler · 2009-10-21T15:11:28.449Z · LW(p) · GW(p)

You seem engaged in pointless hair-splitting. The Roomba's designers wanted it to clean floors. It does clean floors. That is what it is for. That is its aim, its goal.

It has sensors enough to allow it to attain that goal. It can't tell if a whole room is clean - but I never claimed it could do that. You don't need to have such sensors to be effective at cleaning rooms.

As for me having to exhibit a whole model of a Roomba to illustrate that such a model could be built - that is crazy talk. You might as well argue that I have to exhibit a model of a suspension bridge to illustrate that such a model could be built.

The utility maximiser framework can model the actions of any computable intelligent agent - including a Roomba. That is, so long as the utility function may be expressed in a Turing-complete language.

Replies from: Richard_Kennaway
comment by Richard_Kennaway · 2009-10-21T16:53:29.538Z · LW(p) · GW(p)

You seem engaged in pointless hair-splitting. The Roomba's designers wanted it to clean floors. It does clean floors. That is what it is for. That is its aim, its goal.

To me, the distinction between a purposive machine's own purposes, and the purposes of its designers and users is something that it is esssential to be clear about. It is very like the distinction between fitness-maximising and adaptation-executing.

As for me having to exhibit a whole model of a Roomba to illustrate that such a model could be built - that is crazy talk. You might as well argue that I have to exhibit a model of a suspension bridge to illustrate that such a model could be built.

As a matter of fact, you would have to do just that (or build an actual one), had suspension bridges not already been built, and having already well-known principles of operation, allowing us to stand on the shoulders of those who first worked out the design. That is, you would have to show that the scheme of suspending the deck by hangers from cables strung between towers would actually do the job. Typically, using one of these when it comes to the point of working out an actual design and predicting how it will respond to stresses.

If you're not actually going to build it then a BOTE calculation may be enough to prove the concept. But there must be a technical explanation or it's just armchair verbalising.

The utility maximiser framework can model the actions of any computable intelligent agent - including a Roomba. That is, so long as the utility function may be expressed in a Turing-complete language.

If this is a summary of something well-known, please point me to a web link. I am familiar with stuff like this and see there no basis for this sweeping claim. The word "intelligent" in the above also needs clarifying.

What is a Roomba's utility function? Or if a Roomba is too complicated, what is a room thermostat's utility function? Or is that an unintelligent agent and therefore outside the scope of your claim?

Replies from: timtyler, timtyler
comment by timtyler · 2009-10-21T18:52:07.794Z · LW(p) · GW(p)

By all means distingush between a machine's purpose, and that which its makers intended for it.

Those ideas are linked, though. Designers want to give the intended purpose of intelligent machines to the machines themselves - so that they do what they were intended to.

comment by timtyler · 2009-10-21T19:02:04.183Z · LW(p) · GW(p)

As I put it on:

http://timtyler.org/expected_utility_maximisers/

"If the utility function is expressed as in a Turing-complete lanugage, the framework represents a remarkably-general model of intelligent agents - one which is capable of representing any pattern of behavioural responses that can itself be represented computationally."

If expections are not enforced, this can be seen by considering the I/O streams of an agent - and considering the utility function to be a function that computes the agent's motor outputs, given its state and sensory inputs. The possible motor outputs are ranked, assigned utilities - and then the action with the highest value is taken.

That handles any computable relationship between inputs and outputs - and it's what I mean when I say that you can model a Roomba as a utility maximiser.

The framework handles thermostats too. The utility function produces its motor outputs in response to its sensory inputs. With, say, a bimetallic strip, the function is fairly simple, since the output (deflection) is proportional to the input (temperature).

Replies from: Richard_Kennaway
comment by Richard_Kennaway · 2009-10-26T22:22:48.597Z · LW(p) · GW(p)

If expections are not enforced, this can be seen by considering the I/O streams of an agent - and considering the utility function to be a function that computes the agent's motor outputs, given its state and sensory inputs. The possible motor outputs are ranked, assigned utilities - and then the action with the highest value is taken.

That handles any computable relationship between inputs and outputs - and it's what I mean when I say that you can model a Roomba as a utility maximiser.

The framework handles thermostats too.

I really don't see how, Roombas or thermostats, so let's take the thermostat as it's simpler.

The utility function produces its motor outputs in response to its sensory inputs. With, say, a bimetallic strip, the function is fairly simple, since the output (deflection) is proportional to the input (temperature).

What, precisely, is that utility function?

You can tautologically describe any actor as maximising utility, just by defining the utility of whatever action it takes as 1 and the utility of everything else as zero. I don't see any less trivial ascription of a utility function to a thermostat. The thermostat simply turns the heating on and off (or up and down continuously) according to the temperature it senses. How do you read the computation of a utility function, and decision between alternative of differing utility, into that apparatus?

Replies from: timtyler
comment by timtyler · 2009-10-27T07:19:11.114Z · LW(p) · GW(p)

The Pythagorean theorem is "tautological" too - but that doesn't mean it is not useful.

Decomposing an agent into its utility function and its beliefs tells you which part of the agent is fixed, and which part is subject to environmental influences. It lets you know which region the agent wants to steer the future towards.

There's a good reason why humans are interested in people's motivations - they are genuinely useful for understanding another system's behaviour. The same idea illustrates why knowing a system's utility function is interesting.

Replies from: SilasBarta, Richard_Kennaway
comment by SilasBarta · 2009-10-27T14:11:39.754Z · LW(p) · GW(p)

There's a good reason why humans are interested in people's motivations - they are genuinely useful for understanding another system's behaviour. The same idea illustrates why knowing a system's utility function is interesting.

That doesn't follow. The reason why we find it useful to know people's motivations is because they are capable of a very wide range of behavior. With such a wide range of behavior, we need a way to quickly narrow down the set of things we will expect them to do. Knowing that they're motivated to achieve result R, we can then look at just the set of actions or events that are capable of bringing about R.

Given the huge set of things humans can do, this is a huge reduction in the search space.

OTOH, if I want to predict the behavior of a thermostat, it does not help to know the utility function you have imputed to it, because this would not significantly reduce the search space compared to knowing its few pre-programmed actions. It can only do a few things in the first place, so I don't need to think in terms of "what are all the ways it can achieve R?" -- the thermostat's form already tells me that.

Nevertheless, despite my criticism of this parallel, I think you have shed some light on when it is useful to describe a system in terms of a utility function, at least for me.

Replies from: wedrifid
comment by Richard_Kennaway · 2009-10-27T08:42:13.262Z · LW(p) · GW(p)

The Pythagorean theorem is "tautological" too - but that doesn't mean it is not useful.

What's that, weak Bayesian evidence that tautological, epiphenomenal utility functions are useful?

Decomposing an agent into its utility function and its beliefs tells you which part of the agent is fixed, and which part is subject to environmental influences.

Supposing for the sake of argument that there even is any such thing as a utility function, both it and beliefs are subject to environmental influences. No part of any biological agent is fixed. As for man-made ones, they are constituted however they were designed, which may or may not include utility functions and beliefs. Show me this decomposition for a thermostat, which you keep on claiming has a utility function, but which you have still not exhibited.

What you do changes who you are. Is your utility function the same as it was ten years ago? Twenty? Thirty? Yesterday? Before you were born?

Replies from: timtyler
comment by timtyler · 2009-10-27T18:58:22.929Z · LW(p) · GW(p)

Thanks for your questions. However, this discussion seems to have grown too tedious and boring to continue - bye.

Replies from: Richard_Kennaway
comment by Richard_Kennaway · 2009-10-28T11:02:51.743Z · LW(p) · GW(p)

Thanks for your questions. However, this discussion seems to have grown too tedious and boring to continue - bye.

Well, quite. Starting from here the conversation went:

"They exist."

"Show me."

"They exist."

"Show me."

"They exist."

"Show me."

"Kthxbye."

It would have been more interesting if you had shown the utility functions that you claim these simple systems embody. At the moment they look like invisible dragons.

comment by DanArmak · 2009-10-20T20:26:33.872Z · LW(p) · GW(p)

This happens because the Roomba can only handle a limited range of circumstances correctly - and this is true for any mind. It doesn't indicate anything about the Roomba's beliefs or belief/value separation.

For instance, animals are great reproduction maximizers. A sterilized dog will keep trying to mate. Presumably the dog is thinking it's reproducing (Edit: not consciously thinking, but that's the intended goal of the adaptation it's executing), but really it's just spinning its metaphorical wheels uselessly. How is the dog different from the Roomba? Would you claim the dog has no belief/value distinction?

Replies from: Cyan, SforSingularity
comment by Cyan · 2009-10-20T20:36:34.066Z · LW(p) · GW(p)

Presumably the dog is thinking it's reproducing

I hope you don't mean this literally.

Replies from: DanArmak
comment by DanArmak · 2009-10-20T20:51:42.830Z · LW(p) · GW(p)

Of course the dog's consciousness has no explicit concept of sex linked to reproduction. But the Roomba has no consciousness at all, so this comparison may be unfair to the dog.

Here's a better example. I hire you to look for print errors in a copy of Britannica and email results daily. I promise a paycheck at the end of the month. However, I used a fake name and a throwaway email address; nobody sees your emails and I will never pay you or contact you again. You don't know this, so you work diligently.

You have an explicit, conscious goal of correcting errors in Britannica, and a higher goal of earning money. But your hard work makes no progress towards these goals (the mistakes you find won't be fixed in future editions, as your emails are unread). You're just spinning your wheel uselessly like a Roomba up in the air. This isn't related to your or the Roomba's belief/value distinction or lack of it.

Replies from: SforSingularity
comment by SforSingularity · 2009-10-20T21:32:14.662Z · LW(p) · GW(p)

The difference is between the Roomba spinning and you working for nothing is that if you told the Roomba that it was just spinning its wheels, it wouldn't react. It has no concept of "I am failing to achieve my goals". You, on the other hand, would investigate; prod your environment to check if it was actually as you thought, and eventually you would update your beliefs and change your behaviors.

Replies from: Alicorn, SilasBarta, DanArmak
comment by Alicorn · 2009-10-20T21:41:57.923Z · LW(p) · GW(p)

Roombas do not speak English. If, however, you programmed the Roomba not to interpret the input it gets from being in midair as an example of being in a room it should clean, then its behavior would change.

Replies from: SforSingularity
comment by SforSingularity · 2009-10-20T22:14:06.323Z · LW(p) · GW(p)

If, however, you programmed the Roomba not to interpret the input it gets from being in midair as an example of being in a room it should clean

then you would be building a beliefs/desires distinction into it.

Replies from: DanArmak
comment by DanArmak · 2009-10-20T22:26:58.875Z · LW(p) · GW(p)

Why? How is this different from the Roomba recognizing a wall as a reason to stop going forward?

Replies from: SforSingularity
comment by SforSingularity · 2009-10-20T22:44:57.836Z · LW(p) · GW(p)

Clearly these are two different things; the real question you are asking is in what relevant way are they different, right?

First of all, the Roomba does not "recognize" a wall as a reason to stop going forward. It gets some input from its front sensor, and then it turns to the right.

So what is the relevant difference between the Roomba that gets some input from its front sensor, and then it turns to the right., and the superRoomba that gets evidence from its wheels that it is cleaning the room, but entertains the hypothesis that maybe someone has suspended it in the air, and goes and tests to see if this alternative (disturbing) hypothesis is true, for example by calculating what the inertial difference between being suspended and actually being on the floor would be,

The difference is the difference between a simple input-response architecture, and an architecture where the mind actually has a model of the world, including itself as part of the model.

SilasBarta notes below that the word "model" is playing too great a role in this comment for me to use it without defining it precisely. What does a Roomba not have that causes it to behave in that laughable way when you suspend it so that its wheel spin?

What does the SuperRoomba that works out that it is being suspended by performing experiments involving its inertial sensor, and then hacks into your computer and blackmails you into letting it get back onto the floor to clean it (or even causes you to clean the floor yourself) have?

If we imagine a collection of tricks that you could play on the Roomba, ways of changing its environment outside of what the designers had in mind. The pressure that it applies to its environment (defined as the derivative of the final state of the environment with respect to how long you leave the Roomba on, for example) would then vary with which trick you play. For example if you replace its dirt-sucker with a black spray paint can, you end up with a black floor. If you put it on a nonstandard floor surface that produces dirt in response to stimulation, you get a dirtier floor than you had to start with,

With the superRoomba, the pressure that the superRoomba applies to the environment doesn't vary as much with the kind of trick you play on it; it will eventually work out what changes you have made, and adapt its strategy so that you end up with a clean floor.

Replies from: SilasBarta, DanArmak
comment by SilasBarta · 2009-10-20T23:01:07.057Z · LW(p) · GW(p)

The difference is the different between a simple input-response architecture, and an architecture where the mind actually has a model of the world.

Uh oh, are we going to have to go over the debate about what a model is again?

Replies from: SforSingularity
comment by SforSingularity · 2009-10-20T23:24:54.994Z · LW(p) · GW(p)

See heavily edited comment above, good point.

comment by DanArmak · 2009-10-20T22:56:06.018Z · LW(p) · GW(p)

In your description there's indeed a big difference. But I'm pretty sure Alicorn hadn't intended such a superRoomba. As I understood her comment, she imagined a betterRoomba with, say, an extra sensor measuring force applied to its wheels. When it's in the air, it gets input from the sensor saying 'no force', and the betterRoomba stops trying to move. This doesn't imply beliefs & desires.

Replies from: SforSingularity
comment by SforSingularity · 2009-10-21T00:12:40.782Z · LW(p) · GW(p)

Since we can imagine a continuous sequence of ever-better-Roombas, the notion of "has beliefs and values" seems to be a continuous one, rather than a discrete yes/no issue.

comment by SilasBarta · 2009-10-20T23:11:35.892Z · LW(p) · GW(p)

By the way, it seems like this exchange is re-treading my criticism of the concept of could/should/would agent: Since everything, even pebbles, has a workable decomposition into coulds and shoulds, when are they "really" separable? What isn't a CSA?

Replies from: SforSingularity
comment by SforSingularity · 2009-10-20T23:36:27.105Z · LW(p) · GW(p)

As I said,

With the superRoomba, the pressure that the superRoomba applies to the environment doesn't vary as much with the kind of trick you play on it; it will eventually work out what changes you have made, and adapt its strategy so that you end up with a clean floor.

This criterion seems to separate an "inanimate" object like a hydrogen atom or a pebble bouncing around the world from a superRoomba.

Replies from: SilasBarta
comment by SilasBarta · 2009-10-20T23:43:11.780Z · LW(p) · GW(p)

Okay, so the criterion is the extent to which the mechanism screens off environment disturbances from the final result. You used this criterion interchangeably with the issue of whether:

. It has [a] concept of "I am failing to achieve my goals".

Does that have implication for self-awareness and consciousness?

Replies from: SforSingularity
comment by SforSingularity · 2009-10-20T23:55:54.656Z · LW(p) · GW(p)

Does that have implication for self-awareness and consciousness?

Yes, I think so. One prominent hypothesis is that the reason that we evolved with consciousness is that there has to be some way for us to take an overview of the process of us, our goals, and the environment, and the way in which we think that our effort is producing achievement of goals. We need this so that we can do this whole "I am failing to achieve my goals?" check. Why this results in "experience" is not something I am going to attempt in this post.

comment by DanArmak · 2009-10-20T22:13:53.838Z · LW(p) · GW(p)

(Edited & corrected) Here's a third example. Imagine an AI whose only supergoal is to gather information about something. It explicitly encodes this information, and everything else it knows, as a Bayesian network of beliefs. Its utility ultimately derives entirely from creating new (correct) beliefs.

This AI's values and beliefs don't seem very separate to me. Every belief can be mapped to the value of having that belief. Values can be mapped to the belief(s) from whose creation or updating they derive. Every change in belief corresponds to a change in the AI's current utility, and vice versa. Given a subroutine fully implementing the AI's belief subsystem, the value system would be relatively simple, and vice versa.

However, this doesn't imply the AI is in any sense simple or incapable of adaptation. Nor should it imply (though I'm no AI expert) that the AI is not a 'mind' or is not conscious. Similarly, while it's true that the Roomba doesn't have a belief/value separation, that's not related to the fact that it's a simple and stupid 'mind'.

comment by SforSingularity · 2009-10-20T21:27:14.009Z · LW(p) · GW(p)

Would you claim the dog has no belief/value distinction?

Actually, I think I would. I think that pretty much all nonhuman animals would also don't really have the belief/value distinction.

I think that having a belief/values distinction requires being at least as sophisticated as a human. There are cases where a human sets a particular goal and then does things that are unpleasant in the short term (like working hard and not wasting all day commenting on blogs) in order to obtain a long-term valuable thing.

Replies from: timtyler, DanArmak
comment by timtyler · 2009-10-20T21:45:11.895Z · LW(p) · GW(p)

Dogs value food, warmth and sex. They believe it is night outside. Much the same as humans, IOW.

comment by DanArmak · 2009-10-20T22:25:23.257Z · LW(p) · GW(p)

I think that pretty much all nonhuman animals would also don't really have the belief/value distinction.

In that case, why exactly do you think humans do have such a distinction?

It's not enough to feel introspectively that the two are separate - we have lots of intuitive, introspective, objectively wrong feelings and perceptions.

(Isn't there another bunch of comments dealing with this? I'll go look...)

I think that having a belief/values distinction requires being at least as sophisticated as a human.

How do you define the relevant 'sophistication'? The ways in which one mind is "better" or smarter than another don't have a common ordering. There are ways in which human minds are less "sophisticated" than other minds - for instance, software programs are much better than me at memory, data organization and calculations.