A question on rationality.

Of course Bob becomes a monster superintelligence hell bent on using all the energy in the universe for his own selfish reasons. I mean, duh! It's just that "his own selfish reasons" involves things like cute puppies. If Bob cares about cute puppies, then Bob will use his monstrous intelligence to bend the energy of the universe towards cute puppies. And love and flowers and sunrises and babies and cake.

And killing the unbelievers if he's a certain sort - I don't want to make this sound too great. But power doesn't corrupt. Corruption corrupts. Power just lets you do what you want, and people don't want "to stay alive." People want friends and cookies and swimming with dolphins and ice skating and sometimes killing the unbelievers.

I think the most relevant idea here is the distinction between wanting, liking, and approving.

Suppose we have the capability, for any behavior, to change whether we want it, or whether we like it, or whether we approve of it. It seems reasonable to self-modify in such a way that these three things coincide -- that way, you want only things that you like, and you approve of your own behavior. Seen through this lens, Bob's decision to delete his alcohol abuse makes sense: he wants alcohol, but he may or may not like it, and he does not approve of it. So he changes his values so that none of the three apply.

I think a general definition of "approving" is "would self-modify to like and want this, if possible". So the fixed point of Bob's self-modification is not putting flags of +want, +like, +approve on self-preservation, and forgetting everything else. Rather, you would decide what behavior you approve of (which may well include relationships and hobbies -- it certainly doesn't involve being a selfish monster superintelligence) and make yourself want to do it and enjoy doing it.

There are two definitions of rationality to keep in mind: epistemic rationality and instrumental rationality. An agent is epistemically rational to the extent it update their beliefs about the world based on the evidence and in accordance with probability theory - notably Bayes rule.

On the other hand, an agent is instrumentally rational to the extent it maximizes it's utility function (i.e. satisfies it's preferences).

There is no such thing as "rational preferences," though much ink has been spilled trying to argue for them. Clearly preferences can't be rational in an epistemic sense because, well, preferences aren't beliefs. Now can preferences be rational in the instrumental sense? Well, actually, yes but only in the sense that having a certain set of preferences may maximize the preferences you actually care about - not in the sense of some sort of categorical imperative. Suppose a rational agent has the ability to modify their own utility function (i.e. preferences) - maybe an AI that can rewrite its own source code. Would it do it? Well, only if it maximizes that agent's utility function. In other words, a rational agent will change its utility function if and only if it maximizes expected utility according to that same utility function - which is unlikely to happen under most normal circumstances.

As for Bob, presumably he's a human. Humans aren't rational, so all bets are off as far as what I said above. However, let's assume at least with respect to utility function changing behavior Bob is rational. Will he change his utility function? Again, only if he expects it to better help him maximize that same utility function. Now then, what do we make of him editing out his alcoholism? Isn't that a case of editing his utility function? Actually, it isn't - it's more of a constraint of the hardware that Bob is running on. There are a lots of programs running inside Bob's head (and yours), but only a subset are Bob. The difficult part is figuring out which parts of Bob's head are Bob and which aren't.

For #2: Let's assume for the sake of simplicity that Bob is a perfect rational agent with a stable utility function -- Bob's utility function (the function itself, not the output) doesn't change over time. If Bob goes FOOM, Bob is still going to support his original utility function (by definition, because I stipulated that it was stable wrt time above.) I think that you're wondering whether this stability conflicts with Bob being a perfect (instrumentally) rational agent. It doesn't -- Bob being a perfect rational agent just means that he makes the best choices possible to maximize his utility function given the information and computing power that he has.

You can have a rational paperclip maximizer, or a rational bunny maximizer...pretty much whatever you like.

Note: The whole utility function thing gets more complicated for humans, because human's utility functions tend not to be stable wrt time, and it gets tricky when you utility is changing while you're trying to maximize it. Also, we have intuitions which make us happier/more content/whatever when we're trying to maximize certain types of utility functions, and less so with others (having a nice meal will probably make you happier than would a close friend dying, to use an extreme example) -- we usually want to consider this when picking utility functions.

2.If the answer is no, there is no "inherent rationality" absence of a PVG, then what would preclude the possibility that a perfect rationalist, given enough time and resources, will eventually become a perfectly self interested entity with only one overall goal which is to perpetuate his existence, at the sacrifice of everything and everyone else?

That would require having a PVG toward self-interest.

A being without a PVG is not an optimizer. It could not be accurately described as intelligent, let alone rational.

Value is Fragile

You can't derive an ought from an is.