Rationality versus Short Term Selves

So when a superintelligence arises that, despite being Friendly and having the correct goals, does the AGI equivalent of scrolling 9gag, eating Pringles and drinking booze all day long, tell the programmers that the concept of Self, Personal Identity, Agent, or Me-ness was not sufficiently well described, and vit cares too much for vits short-term selves

This is improper anthropomorphising of something we could presumably write down the math for.

Future solidarity is a preference. Different strengths of that preference lead to different time preferences.

Having said that, I think it's true that people generally have a strong future solidarity preference with themselves, at least in theory, so that this is not a valid argument against the long view as the best strategy for winning, given the preferences that people have.

A better argument comes from uncertainty about events and preferences in the future.

In the marshmallow case, how is the child supposed to know that he can trust the Marshmallow Man? As the seconds tick by, is he sure he heard Marshmallow Man right? Maybe he misunderstood. Is he sure he remembers what the Marshmallow Man said?

Have they factored trust out of the marshmallow experiment? Analyzed correlations between trust and holding off on eating the marshmallow?

For grown up problems, we don't know know what the future holds. Remember when a college education was supposed be your key to a secure future? Ha! The world is changing faster than ever. And how good it will be is largely out of your hands. It's unlikely that you, personally, will be the one to defeat Death. You don't know what the world will be like, and you don't even know what you will be like. How have your preferences for money, experience, leisure, people, and places changed in the past? How much do you think they'll change in the future? How will this interact with changes in the state of the world?

In the face of such uncertainty, "do what you love" is likely the most self negating you should be, while "Live Now" is looking more and more reasonable.

A final thought. It occurs to me that backward solidarity is likely as important as future solidarity for long term thinking. Maybe more so. If you feel free to create yourself anew each day, is it likely you'll have the commitment to follow through on a project?

With regards to the singularity, and given that we haven't solved 'morality' yet, one might just value "human well-being" or "human flourishing" without referring to a long-term self concept. I.e. you just might care about a future 'you', even if that person is actually a different person. As a side effect you might also equally care about everyone else in to future too.

Very interesting post.

Our selves post-singularity may well be to us as we are to our ten years of age version. Even now, looking back a few years can feel like having been implanted with the memories of a stranger.

But at least there's physical continuity, right? Talk about brain uploads, our molecules getting switched out all the time anyways, we seem to care only about the pattern, which is fine. The physical continuity is an illusion, and the perceptual continuity broken up every time you go through a deep sleep cycle. So if we solely value the pattern, should we identify more with our peers in a similar situation than with our future selves? (Taken to the extreme, discounting short term benefits may lead to continually sacrificing in the present so that last present moment we experience - on our deathbed? - is supposed to be healthy bliss?)

Well, hopefully the superintelligence can sort out this mess ... deus ex machine indeed.

Your term "piconomics" is non-standard and generates largely irrelevant hits in search. Please stick with the somewhat more standard term "picoeconomics".

Additionally, I'm not sure a new term is useful, given that the issue is pretty much covered in hyperbolic discounting.

I think the interesting question is why we care for our future selves at all.

As kids, we tend not to. It's almost a standard that a child has a holiday, and a bit of homework to do during that holiday, then they will decide not to do the work at the beginning of the break. The reason is they care about their current selves, and not about their future self. Of course in due time the future becomes the present, and that same child has to spend the entire time at the end of their holiday working furiously on everything that's been left to the last minute. At that point, they wish that their past self had chosen an alternative plan. This is still not really wisdom, as they don't much care about their past self either - they care about their present self who now has to do the homework.

Summarising - if your utility function changes over time, then you will, as you mentioned, have conflict between your current and future self. This prevents your plans for the future from being stable - a plan that maximises utility when considered at one point no longer maximises it when considered again later. You cannot plan properly - and this undermines the very point of planning. (You may plan to diet tomorrow, but when tomorrow comes, dieting no longer seems the right answer....)

I think this is why the long view becomes the rational view - if you weight future benefits equally to your present ones, assuming (as you should) that your reward function is stable, then a plan you make now will still be valid in the future.

In fact the mathematical form that works is any kind of exponential - it's OK to have the past be more important than the future, or the future more important than the past as long as this happens as an exponential function of time. Then as you pass through time, the actual sizes of the allocated rewards change, but the relative sizes remain the same, and planning should be stable. In practice an exponential rise pushes all the importance of reward far out into the indefinite future, and is useless for planning. Exponential decays push all the important rewards into your past, but since you can't actually change that, it's almost workable. But the effect of it is that you plan to maximise your immediate reward to the neglect of the future, and since when you reach the future you don't actually think that it was worthwhile that your past self enjoyed these benefits at the expense of your present self, this doesn't really work either as a means of having coherent plans.

That leaves the flat case. But this is a learned fact, not an instinctive one.

There is also the e-marshmallow test (link is not very relevant), in which children have to face the tough choice between surfing an open connected computer with games, internet etc... and waiting patiently for the experimenter to get back. Upon the experimenter's arrival, they get a pile of marshmallows. I presume it also correlates with interesting things, though haven't found much on it.

What about kids who value internet more than marshmallows? A better test would be to promise them more time if they waited.