Junkie AI?

A properly designed AI would not modify its utility function to infinity, because that would not maximize its current utility function.

See Morality as Fixed Computation, The Domain of Your Utility Function, and Maximise Expected Utility, not Expected Perception of Utility.

Utility functions have two very different meanings, and people keep confusing them.

On the one hand, you can take any object, and present it with choices, record what it actually does, and try to represent the pattern of its choices AS IF its internal architecture was generating all possible actions, evaluating them with a utility-function-module, and then taking the action with the highest utility in this situation. Call this the "observational" utility function.

On the other hand, you can build entities that do in fact have utility-function-modules as part of their internal architecture, either a single black box (as in some current AI architectures), or as some more subtle design parameter. Call this the "architectural utility function".

However, entities with an explicit utility function component have a failure mode, so-called "wireheading". If some industrial accident occurred and drove a spike into its brain, the output of the utility function module might be "fail high" and cause the entity to do nothing, or nothing except pursue similar industrial accidents. More subtle, distributed utility function modules would require more subtle, "lucky" industrial accidents, but the failure mode is still there.

Now, if you consider industrial accidents to be a form of stimulus, and you take an entity with a utility-function component, and try to compute its observational utility function, you will find that the observational utility function differs from the designed utility function - specifically, it pursues certain "wireheading" stimuli, even though those are not part of the designed utility function.

If you insist on using observational utility, then wireheading is meaningless; for example, addicted humans want addictive substances, and that's simply part of their utility function. However, I suggest that this is actually an argument against using observational utility. In order to design minds that are resistant to wireheading, we should use the (admittedly fuzzy) concept of "architectural utility" - meaning the output of utility function modules, even though that means that we can no longer say that a (for example) paperclip maximizer necessarily maximizes paperclips. It might try to maximize paperclips but routinely fail, and that pattern of behavior might be characterizable using a different, observational utility function something like "maximize paperclip-making attempts".

As has already been said by many: an optimizing system will optimize for whatever it is designed to optimize for.

If it is designed to optimize for maximizing the value of a counter in a register somewhere, it will do that... maybe find INT_MAX and halt, maybe find some way of rearchitecting itself so that INT_MAX is incremented, depending on how powerful an optimizer it is. It will not spontaneously decide to start making paperclips or making happy people or sorting pebbles into prime-numbered piles instead.

Conversely, if it is designed to optimize for maximizing the number of paperclips, it will do that -- and will not spontaneously decide to start maximizing the value of a counter in a register somewhere.

I think the reason this is sometimes confusing is that people confuse the functional description with the structural one. Sure, maybe the implementation of the paperclip-optimizer involves maximizing counters in registers, but as long as it actually is a paperclip-optimizer it has no reason to further optimize for counter-maximization beyond whatever such optimization is designed into its architecture. (Indeed, it might replace its architecture with a superior paperclip-optimization implementation that doesn't involve counters, as an anti-akrasia method.)

A mind can only want to change its utility function if such a desire is a part of their utility function already. Otherwise you're not talking about the true utility function but only about an inprecise approximation of it.

Example: An addict who craves a fix but wants to no longer crave it -- you may think that his utility function is about procuring the fix, but that's not the true utility function or he wouldn't want to stop craving it -- the real utility function is perhaps just about feeling better.

The "hide an INT_MAX bonus inside the box" idea was mine. One key detail missing from this description of it: the AI does not know that the bonus exists, or how to get it, unless it has already escaped the box; and escaping is meant to be impossible. So it's a failsafe - if everything works as designed it might as well not exist, but if something goes wrong it makes the AI effectively wirehead itself and then halt.

I am aware of two problems with this plan which no one has suggested yet.

That idea is commonly known as wireheading. The idea of it reliably causing problems was discussed here:

http://physicsandcake.wordpress.com/2011/01/22/pavlovs-ai-what-did-it-mean/

I don't understand the science: but my understanding was that utility in this sense was whatever the AI was trying to maximise. If you make an AI that tries to maximise paperclips, it's motivated to do that, not to maximise 'doing whatever maximises my utility maximisation'. I think there's an error here comes of mixing it up with pleasure, and thinking that the paperclip maximiser will 'enjoy' making paperclips but ultimately what it wants is enjoyment. Whereas in its motherboard of motherboards it really, really wants paperclips.