SMBC comic: poorly programmed average-utility-maximizing AI

Go classical - 'lotus-eating'.

There's no clear line between 'hiding' and 'not showing'. You can leave just a million people or so, to be put around the monster, and simply not show him the rest. It is not like the AI is making every wall into the screen displaying the suffering on the construction of pyramids. Or you can kill those people and show it in such a way that the monster derives pleasure from it. At any rate, anyone whose death would go unnoticed by the monster, or whose death does not sufficiently distress the monster, would die, if the AI is to focus on average pleasure.

edit: I think those solutions really easily come to mind when you know of what a soviet factory would do to exceed the five year plan.

The only similarity between those cases is that they involve utility calculations you disagree with. Otherwise every single detail is completely different. (e. g. the sort of utility considered, two negative utilities being traded against each other vs. trading utility elsewhere (positive and negative) for positive utility, which side of the trade the single person with the large individual utility difference is on, the presence of perverse incentives, etc, etc).

If anything it would be more logical to equate Felix with the tortured person and treat this as a reductio ad absurdum of your position on the dust speck problem. (But that would be wrong too, since the numbers aren't actually the problem with Felix, the fact that there's an incentive to manipulate your own utility function that way is (among other things).)

Felix is essentially a Utility Monster: a thought experiment that's been addressed here before. As that family of examples shows, happiness-maximization breaks down rather spectacularly when you start considering self- or other-modification or any seriously unusual agents. You can bite that bullet, if you want, but not many people here do; fortunately, there are a few other ways you can tackle this if you're interested in a formalization of humanlike ethics. The "Value Stability and Aggregation" post linked above touches on the problem, for example, as does Eliezer's Fun Theory sequence.

You don't need any self-modifying or non-humanlike agents to run into problems related to "Torture vs. Dust Specks", though; all you need is to be maximizing over the welfare of a lot of ordinary agents. 3^^^3 is an absurdly huge number and leads you to a correspondingly counterintuitive conclusion (one which, incidentally, I'd estimate has led to more angry debate than anything else on this site), but lesser versions of the same tradeoff are quite realistic; unless you start invoking sacred vs. profane values or otherwise define the problem away, it differs only in scale from the same utilitarian calculations you make when, say, assigning chores.

In one case, (Torture to avoid the specks) the larger portion of people is better off if you pick the single person. In the other case, (Build pyramids to please Felix) the larger portion of people is worse off if you pick the single person.

So if my position was "The majority should win" It would be right to torture the person and it would be right to depose Felix.

I'm not sure if it's a fundamental difference or a good difference, but I think that means I can lay out the following 4 distinct answer pairs:

Depose Felix, Torture Man: Majority wins.

Adore Felix, Speck people: Minority wins.

Adore Felix, Torture Man: Mean Happiness wins.

Depose Felix, Speck People: Minimum happiness wins. (Assuming either Felix is happier about being deposed than an average person with a dust speck in their eye, or dead, and no longer counted for minimum happiness.)

So I think I can see all 4 distinct positions, if I'm not missing something.

Edit: Fixed spacing.

What's the fundamental difference between those two cases? I don't see it, do you?

One fundamental difference is that I don't care about Felix's further happiness. After some point, I may even resent it, which would make his additional happiness of negative utility to me.

Another difference is that happiness may be best represented as a percentage with an upper bound of e.g. 100% happy, rather than be an integer you can keep adding to without end.

I think Felix's case may be an interesting additional scenario to consider, in order to be sure that AIs don't fall victims to it (e.g. by creating a superintelligence and making it super-happy, to the expense of normal human happiness). But it's not the same scenario as the specks.

this AI which does absolutely nothing in our world is no more useful than AI that does nothing in all possible worlds, or just a brick.

Sorry, I wasn't being clear. The point was saying that no AI can do better than humanity implies that our world is optimal out of all similar worlds. (I believe there are much stronger arguments than this against what you are saying, but this one should suffice)

Name 1 then.

The Unobtrusive Guardian. An FAI that concludes that humanity's aversion to being 'straightjacketed' is such that it is never ok for it to interfere with what humans do themselves. It proceeds to navigate itself out of the way and wait until it spots an external threat like a comet or hostile aliens. It then destroys those threats.

(The above is not a recommended FAI design. It is a refutation by example of an absolute claim that would exclude the above.)