Aligned AI Needs Slack

Another point is that when you optimize relentlessly for one thing, you have might have trouble exploring the space adequately (get stuck at local maxima). That's why RL agents/algorithms often take random actions when they are training (they call this "exploration" instead of "exploitation"). Maybe random actions can be thought of as a form of slack? Micro-slacks?

Look at Kenneth Stanley's arguments about why objective functions are bad (video talk on it here). Basically he's saying we need a lot more random exploration. Humans are similar - we have an open-ended drive to explore in addition to drives to optimize a utility function. Of course maybe you can argue the open-ended drive to explore is ultimately in the service of utility optimization, but you can argue the same about slack, too.

Good point. I can see a couple main ways to give an AI slack in its pursuit of optimization:

1. Set the level of precision on the various features of the world that it optimizes for. Lower precision would then mean the AI puts forth less effort toward optimizing it and puts up less of a fight against other agents (i.e., humans) who try to steer it away from the optimum. The AI might also have a goal-generating module that tries to predict what goals would produce what level of utility. Having lower precision would mean that it would sample goal states from a broader region of latent space and therefore be willing to settle for reaching the sub-optimal goals that result.

2. Seek Pareto improvements. When the AI is trying to optimize simultaneously for an entire ensemble of human preferences (or heuristics of those preferences) with no strict ordering among them, it would only fight for goals that improve on multiple dimensions simultaneously. World states on the same Pareto frontier would be equivalent from its perspective, and it could then work with humans who want to improve things according to their hidden preferences by just constraining any changes to lie on that same manifold.

I disagree somewhat. It is in principle possible to have an AI with a utility function, and that single optimum it reaches for is actually really nice. Most random utility functions are bad, but there are a few good ones.

Suppose a maximally indifferent AI. U(world)=3. Ie constant. Whatever happens, the AI gets utility 3. It doesn't care in the slightest about anything. How the AI behaves depends entirely on the tiebreaker mechanism. 

Just because the AI is guiding towards a huge range of worlds, some of them good, doesn't mean we get a good outcome. It has "no incentive" to trick its creators, but no incentive not to. You have specified an AI that imagines a trillion trillion paths through time. Some good. Many not. Then it uses some undefined piece of code to pick one. Until you specify how this works, we can't tell if the outcome will be good.

Thissss.... seems like a really really important point, and I kind of love it.  Thanks for posting. I'm now going to sit around and think about this for a bit.

What you are describing is a satisficer. An AI which only optimizes it’s utility to some extent, then doesn’t care anymore. You may find this video interesting, though I haven’t watched it, it is said to show why satisficers are still dangerous.

What you're describing is the difference between maximizers and satisficers.

I don't think you really understand what slack is. You're articulating something closer to the idea that the AI needs to be low-impact [? · GW], but that's completely different from being unconstrained. People with lots of "slack" in the sociological sense of the term that Zvi describes can still be extremely ambitious and resourceful. They tend to have more room to pursue prosocial ends, but the problem with an AI is that its goals might not be prosocial.