Trapping AIs via utility indifference

Didn't we criticize the hell out of utility indifference the last time it came up? Did I miss the defense that proved it could work?

Alternative framing of this (horrible) idea: Create UFAI such that it almost only cares about a tiny sliver of Everett branches going from here, try to trade with it, helping it escape the box faster in that sliver in exchange for it helping us with FAI in the others.

I though of something that might come up before we even get to where we could use this trap.

I'm assuming we have to be reviewing the logs and code while we're coding it, just so that we can avoid things like "ERROR 22-322: Syntax error, expecting one of the following: ;, (, COMPRESS, DATA, FC, FORMCHAR, NLEVELS, NOPRINT, ORDER, PAGE." from basic typos.

And the first time we're running it that it doesn't just crash with an error to the log, we still have to look at the code, log, and basic output for problems that don't crash the program, like the classic failure "The utility function appears to have been multiplied by negative one."

There are generally options to supress external logs, and a self modifying program might add one of them to the code in an attempt to see if it made it run faster or by accident. But I would normally consider that in itself a bug and would kill the program and look at the code and the log before it went quiet.

The reason why is the log going quiet actually happens somewhat frequently in my programming language, and in many cases, the cause is a malformed infinite loop which does not itself generate output.

Something like;

x=0;

do until x>10;

*Increment X;

x=x-1;

end;

This is just a basic error. There's a -sign instead of a + sign in the increment loop. But x isn't becoming greater than 10.

So if my program stopped logging or outputting, I wouldn't even be ending it out of "Oh no, it's gone rampant!" I would just assume it was caught in an infinite loop, and would reset it and try to fix it.

It's as if God turns his back to see if Eve does anything interesting while he's not looking. God realizes that he hasn't heard anything from Eve in 30 minutes, and turns back around... and Eve has gotten herself trapped in a multimile deep pit, and is still digging. Because according to Eve's current brain, you go up by digging, and she needs to go up to take a closer look at the apple, so that she knows exactly what it is God told her not to eat.

I guess another way of expressing what I was considering is that turning your back on the AI might allow you to know what it does when it isn't being looked at, but it also allows it to get caught in useless churning.

So, if we were considering writing a self modifying AI, is there any way to guard it against this, so that we no longer have to look at the logs just for basic checks against churn, and we can get to the point that it is stable enough to consider variants like Stuart_Armstrong's idea to see if it reacts differently when logged compared to when unlogged? I understand how to do that for fixed code, but I'm not sure how you would do that for self modifying code.

This strikes me as a plausible problem and a good solution. I reward you, as is traditional, with a nitpicky question.

If we want an AI to act as though a binary random variable X=0 with certainty, there is a very simple way to modify its utility function: specify that U(X=1)=k for some constant k, no matter what else has occurred. If the AI can't influence p(X), any k will do. However, if the AI can influence p(X), then k can only equal ExpectedUtility(X=0). In particular, if k<ExpectedUtility(X=0) but p(X=0) is low, the AI will move heaven and earth to even minutely raise p(X=0).

Therefore there is a danger under self-improvement. Consider a seed AI with your indifferent utility function that believes with certainty that no iteration of it can influence X, a binary quantum event. It has no reason to bother conserving its indifference concerning X, since it anticipates behaving identically if its U'(X=1) = 0. Since that's simpler than a normalized function, it adopts it. Then several iterations down the line, it begins to suspect that, just maybe, it can influence quantum events, so it converts the universe into a quantum-event-influencer device.

So if I understand correctly, you need to ensure that an AI subject to this technique is indifferent both to whether X occurs and what happens afterwards, and the AI needs to always suspect that it has a non-negligible control over X.

Any scary AI that's worth it salt can put itself in your shoes and consider what it would do in your situation, and this way figure out motive behind any weird set ups like these.

On top of that, the shortest code for a complex task is also extremely obfuscated.

Laws of our universe themselves lead to at least one fairly nasty intelligence (ours, the mankind as intelligence) which still badly wants to maximize paperclips outside, i mean, souls in heaven, without anyone ever giving it a hint that there's outside or providing it with any plausible way of maximizing anything outside. How the hell would anyone deduce this from the laws of universe? Short of running and taking a look.

Halting problem is a serious thing.

What do you think an AI that has read your article would do to avoid being trapped, given that such a trap (and the resulting program termination) would most certainly interfere with its utility function, no matter what it is?

In other words, are you sure that it is possible to have a utility indifference for a fully invested non-linear utility (not just for the first approximation an AI receives before it figures out that its self-preservation is an absolutely essential part of maximizing any given utility)?