Where do selfish values come from?

By "selfish" I mean how each human (apparently) cares about himself more than others, which needs an explanation because there can't be a description of himself embedded in his brain at birth.

Pointing at self is possible, which looks like a reasonable description of self, referring to all the details of a particular person. That is, interpretation of individual's goal representation depends on the fact that the valued individual is collocated with the individual-as-agent.

Just as how a file offset value stored in memory of my computer won't be referring to the same data if used on (moved to) your computer that has different files; its usefulness depends on the fact that it's kept on the same computer; and it will continue to refer to same data if we move the whole computer around.

No, that's not the meaning I had in mind.

Now I'm confused again, as I don't see how these senses (one I suggested and one you explained in parent comment) differ, other than on the point of caring vs. not caring about counterfactual versions of same individual. You said, "each human (apparently) cares about himself more than others, which needs an explanation", and it reads to me as asking how can humans have the individual-focused utility I suggested, that you then characterized as not the meaning you had in mind...

By “selfish” I mean how each human (apparently) cares about himself more than others, which needs an explanation because there can’t be a description of himself embedded in his brain at birth.

It's hardly a mystery: you're only plumbed into your own nervous system , so you only feel your own pleasures and pains. That creates a tendency to be only concerned about them as well. It's more mysterious that you would be concerned about pains you can't feel

I think there's enough science on the subject - here's the first paper I could find with a quick Google - to sketch out an approximate answer to the question of how self-care arises in an individual life. The infant first needs to form the concept of a person (what Bischof calls self-objectification), loosely speaking a being with both a body and a mind. This concept can be applied to both self and others. Then, depending on its level of emotional contagion (likelihood of feeling similarly to others when observing their emotions) it will learn, through sophisticated operant conditioning, self-concern and other-concern at different rates.

Since the typical human degree of emotional contagion is less than unity, we tend to be selfish to some degree. I'm using the word "selfish" just as you've indicated.

there can't be a description of himself embedded in his brain at birth.

Why not, or what do you mean by this? Common sense suggests that we do know ourselves from others at a very low, instinctive level.

Just as altruism can be related to trust, selfishness can be related to distrust.

An agent which has a high prior belief in the existence of deceptive adversaries would exhibit "selfish" behaviors.

No, that's not the meaning I had in mind.

What is your meaning then? What would you call "caring about the welfare of a particular individual (that happens to be myself)"?

No one can mix another person's actual utility function into their own. You can mix in your estimate of it. You can mix in your estimate of what you think it should be. But the actual utility function of another person is in that other person, and not in you.

there's a sense in which humans don't have values, just things like thoughts and behaviors.

Between values, thoughts, and behaviors, it seems like the larger gap is between behaviors on the one hand, and thoughts and values on the other. Given a neurological description of a human being, locating "thoughts" in that description would seem roughly comparable in difficulty to locating "values" therein. Not that I take this to show there are neither thoughts nor values. Such a conclusion would likely indicate overly narrow definitions of a thought and a value.

I could see neuroscience results arguing for one side or the other, but I think the question of what exact neuroscience results would argue for which answer is a morally loaded one.

I think so too.

What I meant by my post is that starting with random preferences, those that we designate as selfish survive. So what we intuitively think of selfishness - me-first, a utility function with an index pointing to myself - arises naturally from non-indexical starting points (evolving agents with random preferences).

If it arose this way, then it is less mysterious as to what it is, and we could start looking at evolutionary stable decision theories or suchlike. You don't even have to have evolution, simply "these are preferences that would be advantageous should the AI be subject to evolutionary pressure".

Does that make sense?

It used to, as Tim notes, but I'm not so sure now. AIXI works with its distribution over programs and sequences of observations, not with states of a world and its properties. If presented with a sequence of observations generated by a program, it quickly figures out what the following observations are, but it's more tricky here.

With other types of agents, we usually need to stipulate that the problem statement is somehow made clear to the agent. The way in which this could be achieved is not specified, and it seems very difficult to arrange through presenting an actual sequence of observations. So the shortcut is to draw the problem "directly" on agent's mind in terms of agent's ontology, and usually it's possible in a moderately natural way. This all takes place apart from the agent observing the state of the coin.

However in case of AIXI, it's not as clear how the elements of the problem setting should be expressed in terms of its ontology. Basically, we have two worlds corresponding to the different coin states, which could for simplicity be assumed to be generated by two programs. The first idea is to identify the programs generating these worlds with relevant AIXI's hypotheses, so that observing "tails" excludes the "heads"-programs, and therefore the "heads"-world, from consideration.

But there are many possible "tails"-programs, and AIXI's response depends on their distribution. For example, the choice of a particular "tails"-program could represent the state of other worlds. What does it say about this distribution that the problem statement was properly explained to the AIXI agent? It must necessarily be more than just observing "tails", the same as for other types of agents (if you only toss a coin and it falls "tails", this observation alone doesn't incite me to pay up). Perhaps "tails"-programs that properly model CM also imply paying the mugger.

It makes sense, but the conclusion apparentlly depends on how AIXI's utility function is written. Assuming it knows Omega is trustworthy...

• If AIXI's utility function says to maximise revenue in this timeline, it does not pay.

• If it says to maximise revenue across all its copies in the multiverse, it does pay.

The first case - if I have analysed it correctly - is kind-of problematical for AIXI. It would want to self-modify.,,

Something equivalent to precommitment is: it just being in your nature to trust counterfactual muggers. Then, recomputing your actions in every time-step is fine, and it doesn't necessarilly indicate that you don't have a nature that alllows you to pay counterfactual muggers.

This doesn't really clarify anything. You can consider AIXI as a formal definition of a strategy that behaves a certain way; whether this definition "understands" something is a wrong question.

Er, that is unnecessarily hostile and insulting nonsense. Please think before you post public insults.