Welcome to Heaven

I think you are missing the point.

First throw out the FAI part of this argument; we can consider an FAI just as a tool to help us achieve our goals. Any AI which does not do at least this this is insufficiently friendly (and thus counts as a paperclipper, possibly).

Thus, the actual question is what are our goals? I don't know about you, but I value understanding and exploration. If you value pleasure, good! Have fun being a wirehead.

It comes down to the fact that a world where everyone is a wirehead is not valued by me or probably by many people. Even though this world would maximize pleasure, it wouldn't maximize utility of people designing the world (I think this is the util/hedon distinction, but I am not sure). If we don't value that world, why should we create it, even if we would value after we create it?

What I'm observing in the various FAI debates is a tendency of people to shy away from wire-heading as something the FAI should do. This reluctance is generally not substantiated or clarified with anything other than "clearly, this isn't what we want". This is not, however, clear to me at all.

I don't want that. There, did I make it clear?

If you are a utilitarian, and you believe in shut-up-and-multiply, then the correct thing for the FAI to do is to use up all available resources so as to maximize the number of beings, and then induce a state of permanent and ultimate enjoyment in every one of them.

Since when does shut-up-and-multiply mean "multiply utility by number of beings"?

If you don't want to be "reduced" to an eternal state of bliss, that's tough luck.

Heh heh.

denis, most utilitarians here are preference utilitarians, who believe in satisfying people's preferences, rather than maximizing happiness or pleasure.

To those who say they don't want to be wireheaded, how do you really know that, when you haven't tried wireheading? An FAI might reason the same way, and try to extrapolate what your preferences would be if you knew what it felt like to be wireheaded, in which case it might conclude that your true preferences are in favor of being wireheaded.

But I don't want to be a really big integer!

One reason I might not want to become a ball of ecstasy is that I don't really feel it's ME. I'm not even sure it's sentient, since it doesn't learn, communicate, or reason, it just enjoys.

Not for the Sake of Happiness (Alone) is a response to this suggestion.

I just so happened to read Coherent Extrapolated Volition today. Insofar as this post is supposed to be about "what an FAI should do" (rather than just about your general feeling that objections to wire-heading are irrational), it seems to me that this post all really boils down to navel-gazing once you take CEV into account. Or in other words, this post isn't really about FAI at all.

A number of people mention this one way or another, but an explicit search for "local maximum" doesn't match any specific comment - so I wanted to throw it out here.

Wireheading is very likely to put oneself in a local maximum of bliss. Though a wirehead may not care or even ponder about whether or not there exist greater maxima, it's a consideration that I'd take into account prior to wiring up.

Unless one is omniscient, the act of a permanent (-ish) state of wireheading means foregoing the possibility of discovering a greater point of wireheaded happiness.

I guess the very definition of wireheadedness bakes in the notion that you wouldn't care about that anymore - good for those taking the plunge and hooking up I suppose. Personally, the universe would have to throw me an above average amount of negative derivates before I'd say enough is enough, screw potential for higher maxima, I'll take this one...

If you are a utilitarian, and you believe in shut-up-and-multiply, then the correct thing for the FAI to do is to use up all available resources so as to maximize the number of beings, and then induce a state of permanent and ultimate enjoyment in every one of them.

Rewritting that: if you are an altruistic, total utilitarian whose utility function includes only hedonistic pleasure and with no birth-death asymmetry, then the correct thing for the FAI to do is...

If you're considering the ability to rewire one's utility function, why simplify the function rather than build cognitive tools to help people better satisfy the function? What you've proposed is that an AI destroys human intelligence, then pursues some abstraction of what it thinks humans wanted.

Your suggestion is that an AI might assume that the best way to reach its goal of making humans happy (maximizing their utility) is to attain the ends of humans' functions faster and better than we could, and rewire us to be satisfied. There are two problems here that I see.

First, the means are an end. Much of what we value isn't the goal we claim as our objective, but the process of striving for the goal. So here you have an AI that doesn't really understand what humans want.

Second, most humans aren't interested in every avenue of exploration, creation and pleasure. Our interests are distinct. They also do change over time (or some limited set of parameters do anyhow). We don't always notice them change, and when they do, we like to track down what decisions they made that led them to their new preferences. People value the (usually illusory) notion that they control changes to their utility functions. The offer to be a "wirehead" is an action which intrinsically violates peoples' utility in the illusion of autonomy. This doesn't apply to everyone - hedonists can apply to your heaven. I suspect that few others would want it.

Also,

If you don't want to be "reduced" to an eternal state of bliss, that's tough luck... The FAI can simply modify your preferences so you want an eternally blissful state.

That is not friendly.

I think you have an idea that there is a "global human utility function" and that FAI is that which satisfies this. Humans have commonalities in their functions, but they are localized around the notion of self. Your "FAI" generalizes what most people want in some form except for experience and autonomy, but the other values it extracts are, in humans, not independent from those.

I'd like to be a wirehead, but have no particular desire to impose that condition on others.

I think people shy away from wireheading because a future full of wireheads would be very boring indeed. People like to think there's more to existence than that. They wan't to experience something more interesting than eternal pleasure. And that's exactly what an FAI should allow.

If all the AI cares about is the utility of each being times the number of beings, and is willing to change utility functions to get there, why should it bother with humans? Humans have all sorts of "extra" mental circuitry associated with being unhappy, which is just taking up space (or computer time in a simulator). Instead, it makes new beings, with easily satisfied utility functions and as little extra complexity as possible.

The end result is just as unFriendly, from a human perspective, as the naive "smile maximizer".

Even from a hedonistic perspective, 'Shut up and multiply' wouldn't necessarily equate to many beings experiencing pleasure.

It could come out to one superbeing experiencing maximal pleasure.

Actually, I think thinking this out (how big are entities, who are they?) will lead to good reasons why wireheading is not the answer.

Example: If I'm concerned about my personal pleasure, then maximizing the number of agents isn't a big issue. If my personal identity is less important than total pleasure maximizing, then I get killed and converted to orgasmium (be it one being or many). If my personal identity is more important... well, then we're not just multiplying hedons anymore.

This is a very relevant post for me because I've been asking these questions in one form or another for several months. A framework of objective value (FOV) seems to be precluded by physical materialism. However, without it, I cannot see any coherent difference between being happy (or satisfied) because of what is going on in a simulation and what is going on in reality. Since value (that is, our personal, subjective value) isn't tied to any actual objective good in the universe, it doesn't matter to our subjective fulfillment if the universe is modified to be 'better' (with respect to our POV), a simulation we're in is modified to be better, or our preferences are modified.

For example, I asked the question several weeks ago here.

When I began to complain (at length...) that without FOV I felt like I was trapped in a machine carrying out instructions to satisfy preferences I neither care about nor am able to abort, it was recommended that I replace my preference for objective value with a preference for subjective value.

If it is true that the only solution to my problem with the non-existence of an FOV is to change my preference -- and I've already understood that the logical consequence of this is that any kind of preference fulfillment is equivalent to wire-heading -- then I'm simply not going to be very sympathetic to objections to wire-heading based on having preferences for not being wire-headed. It's simply not coherent; there's no difference.

This is one of the most horrifying things I have ever read. Most of the commenters have done a good just of poking holes in it, but I thought I'd add my take on a few things.

This reluctance is generally not substantiated or clarified with anything other than "clearly, this isn't what we want".

Some good and detailed explanations are here, here, here, here, and here.

If you are a utilitarian, and you believe in shut-up-and-multiply, then the correct thing for the FAI to do is to use up all available resources so as to maximize the number of beings,

No, the correct thing for an FAI to do is to use some resources to increase the number of beings and some to increase the utility of existing beings. You are assuming that creating new beings does not have diminishing returns. I find this highly unlikely. Most activities generate less value the more we do them. I don't see why this would change for creating new beings.

Having new creatures that enjoy life is certainly a good thing. But so is enhancing the life satisfaction of existing creatures. I don't think one of these things is categorically more valuable than the other. I think they are both incrementally valuable.

In other words, as I've said before, the question is not, "Should we maximize total utility or average utility?" It's "How many resources should be devoted to increasing total utility, and how many to increasing average utility?"

and then induce a state of permanent and ultimate enjoyment in every one of them.

Wouldn't it be even more efficient to just create creatures that feel nothing except a vague preference to keep on existing, which is always satisfied?

Or maybe we shouldn't try to minmax morality. Maybe we should understand that phrases like "maximize pleasure" and "maximize preference satisfaction" are just rules of thumb that reflect a deeper and more complex set of moral values.

The most energy efficient way to create any kind of enjoyment, however, is to stimulate the brain-equivalent directly.

Again, you're assuming all enjoyments are equivalent and don't generate diminishing returns. Pleasure is valuable, but it has diminishing returns, you get more overall value by increasing lots of different kinds of positive things, not just pleasure.

In fact, I can hardly tell this apart from the concept of a Christian Heaven, which appears to be a place where Christians very much want to get.

If you're right this is just proof that Christians are really bad at constructing Heaven. But I don't think you are, most Christians I know think heaven is far more complex than just sitting around feeling good.

If you don't want to be "reduced" to an eternal state of bliss, that's tough luck. The alternative would be for the FAI to create an environment for you to play in, consuming precious resources that could sustain more creatures in a permanently blissful state.

The alternative you suggest is a very good alternative. Creating all those blissful creatures would be a waste of valuable resources that could be used to enhance the preferences of already existing creatures. Again, creating new creatures is often a good thing, but it has diminishing returns.

Now for some rebuttals to your statements in the comments section:

If we all think it's so great to be autonomous, to feel like we're doing all of our own work, all of our own thinking, all of our own exploration - then why does anyone want to build an AI in the first place?

Again, complex values and diminishing returns. Autonomy is good, but if an FAI can help us obtain some other values it might be good to cede a little of our autonomy to it.

I find that comparable to a depressed person who doesn't want to cure his depression, because it would "change who he is". Well, yeah; but for the better.

It's immoral and illegal to force people to medicate for a reason. That being said, depression isn't a disease that changes what your desires are. It's a disease that makes it harder to achieve your desires. If you cured it you'd be better at achieving your desires, which would be a good thing. If a cure radically changed what your desires were it would be a bad thing.

That being said, I wouldn't necessarily object to rewiring humans so that we feel pleasure more easily,, as long as it fulfilled two conditions:

1. That pleasure must have a referent. You have to do something to trigger the reward center in order to feel it, stimulating the brain directly would be bad.
2. The increase must be proportional. I should still enjoy a good movie better then a bad movie, even if I enjoy them both a lot more.

Wei explains that most of the readership are preference utilitarians, who believe in satisfying people's preferences, not maximizing pleasure.

That's fine enough, but if you think that we should take into account the preferences of creatures that could exist, then I find it hard to imagine that a creature would prefer not to exist, than to exist in a state where it permanently experiences amazing pleasure.

I don't think that that it's ethical, or possible to take into account the hypothetical preferences of nonexistant creatures. That's not even a logically coherent concept. If a creature doesn't exist, then it doesn't have preferences. I don't think it's logically possible to prefer to exist if you don't already. Besides, as I said before, it would be even more efficient to create a creature that can't feel pleasure, that just has a vague preference to keep on existing that would always be satisfied as long as it existed. But I doubt you would want to do that.

Besides, for every hypothetical creature that wants to exist and feel pleasure, there's another hypothetical creature that wants that creature to not exist, or feel pain. Why are we ignoring those creature's preferences?

The only way preference utilitarianism can avoid the global maximum of Heaven is to ignore the preferences of potential creatures. But that is selfish.

No, it isn't. Selfishness is when you severely thwart someone's preferences to mildly enhance your own. It's not selfish to thwart nonexistant preferences because they don't exist. That's like saying it's gluttonous to eat nonexistant food, or vain to wear nonexistant costume jewelry.

The reason some people find the idea that you have to respect the preferences of all potential creatures is that they believe (correctly) that they have an obligation to make sure people who exist in the future will have satisfied preferences. But that isn't because nonexistant people's preference have weight. It's because it's good for whoever exists at the moment to have highly satisfied preferences, so as soon as a creature comes into existence you have a duty to make sure it is satisfied. And the reason those people's preferences are highly satisfied should be that they are strong, powerful, and have lots of friends, not because they were genetically modified to have really really unambitious preferences.

If you don't want Heaven, then you don't want a universally friendly AI. What you really want is an AI that is friendly just to you.

I want a universally friendly AI, but since nonexistant creatures don't exist in this universe, not creating them isn't universally unfriendly.

Also, I find it highly suspect, to say the least, that you start by arguing for "Heaven" because you think that all human desires can be reduced to the desire to feel certain emotions, but then when the commenters have poked holes in that idea you suddenly change and use a completely different justification (the logically incoherent idea that we have to respect the nonexistant preferences of nonexistant people) to defend it.

The infinite universe argument can be used as an excuse to do pretty much anything. Why not just torture and kill everyone and everything in our Hubble volume? ... If there are infinite copies of everyone and everything, then there's no harm done.

I find it helpful to think of having a copy as a form of life extension, except done serially instead of linearly. An exact duplicate of you who lives for 70 years is similar to living an extra 70 years. So torturing everyone because they have duplicates would be equivalent to torturing someone for half their lifespan and then saying that it's okay because they still have half a lifespan leftover.

Whatever happens outside of our Hubble volume has no consequence for us, and neither adds to nor alleviates our responsibility.

Again, if these creatures exist somewhere else, then if you create them you aren't really creating them, you're extending their lifespan. Now, having a long lifespan is one way of having a high quality of life, but it isn't the only way, and it does have diminishing returns, especially when it's serial instead of linear, and you don't share your copy's memories. So it seems logical that, in addition to focusing on making people live longer, we should increase their quality of life in other ways, such as devoting resources to making them richer and more satisfied.

If we take for granted that an AI that is friendly to all potential creatures is out of the question - that the only type of FAI we really want is one that's just friendly to us - then the following is the next issue I see.

If we all think it's so great to be autonomous, to feel like we're doing all of our own work, all of our own thinking, all of our own exploration - then why does anyone want to build an AI in the first place?

Isn't the world, as it is, lacking an all-powerful AI, perfectly suited to our desires of control and autonomy?

Suppose an AI-friendly-to-you exists, and you know that you can always ask it to expand your mind, and download into you everything it knows about the issues you care for, short-circuiting thousands of years of work that it would otherwise take for you to make the same discoveries.

Doesn't it seem pointless to be doing all that work, if you know that FAI can already provide you with all the answers?

Furthermore, again supposing an AI-friendly-to-you exists - you know that you can always ask it to wire-head you. In any given year, there is a negligible, but non-zero probability that you'll succumb to the temptation. Once you do succumb to the temptation, it will feel so great that you will never ever want to do the boring "thinking and doing stuff" again. You will be constantly blissful, and anything you want to know about the universe will be immediately available to you, through a direct interface with FAI.

It doesn't take much to see that, whether it takes a thousand years or a million years before you succumb, you will eventually choose to be wire-headed; you will choose this much sooner than the universe ends; and the vast majority of your total existence will be lived that way.

It is worth noting that in Christian theology, heaven is only reached after death, and both going there early and sending people there early are explicitly forbidden.

While an infinite duration of bliss has very high utility, that utility must be finite, since infinite utility anywhere causes things to go awry when handling small probabilities of getting that utility. It is also not the only term in a human utility function; living as a non-wirehead for awhile to collect other types of utilons and then getting wireheaded is better than getting wireheaded immediately. Therefore, it seems like the sensible thing for a FAI to do is to offer wireheading as an option, but not to force the issue except in cases of imminent death.

This reminds me of a thought I had recently - whether or not God exists, God is coming - as long as humans continue to make technological progress. Although we may regret it (for one, brief instant) when he gets here. Of course, our God will be bound by the laws of the universe, unlike the Theist God.

The Christian God is an interesting God. He's something of a utilitarian. He values joy and created humans in a joyful state. But he values freedom over joy. He wanted humans to be like himself, living in joy but having free will. Joy is beautiful to him, but it is meaningless if his creations don't have the ability to choose not-joy. When his creations did choose not-joy, he was sad but he knew it was a possibility. So he gave them help to make it easier to get back to joy.

I know that LW is sensitive to extended religious reference. Please forgive me for skipping the step of translating interesting moral insights from theology into non-religious speak.

I do hope that the beings we make which are orders of magnitude more powerful than us have some sort of complex value system, and not anything as simple as naive algebraic utilitarianism. If they value freedom first, then joy, then they will not enslave us to the joy machines - unless we choose it.

(Side note: this post is tagged with "shut-up-and-multiply". That phrase trips the warning signs for me of a fake utility function, as it always seems to be followed by some naive algebraic utilitarian assertion that makes ethics sound like a solved problem).

edit: Whoa, my expression of my emotional distaste for "shut up and multiply" seems to be attracting down-votes. I'll take it out.

In biology, 1 and 2 are proximate goals and 3 is an implementation detail.

So this begs(?) the question: Our brains pleasure circuitry is the ultimate arbitrator on whether an action is "good" [y/N]?

I would say that our pleasure centre is, like our words-feel-like-meaningful, our map-feels-like-territory, our world-feels-agent-driven, our consciousness-feels-special, etc. It is a good enough evolutionary heuristic that made our ancestors survive to breed us.

I am at this point tempted to shout "Stop the presses, pleasure isn't the ultimate good!" Yes, wire heading is of course the best way to fulfil that little feeling-good part of your brain. Is it constructive? Meh.

I would trade my pleasure centre for intuitive multiplication any day.

Point 3. doesn´t seem to belong in the same category as 1. and 2.

What if we could create a wirehead that made us feel as though we were doing 1 or 2? Would that be satisfactory to more people?

Oh heaven Heaven is a place A place where nothing Nothing ever happens

The idea of wireheading violates my aesthetic sensibilities. I'd rather keep experiencing some suffering on my quest for increasing happiness, even if my final subjective destination were the same as that of a wirehead, which I doubt, because I consider my present path as important as the end goal. I doubt value and morality can be fully deconstructed through reason.

How is wireheading different from this http://i.imgur.com/wKpLx.jpg ? I think James Hughes makes a very good case for what is wrong with current transhumanist thought in his 'Problems of Transhumanism' http://ieet.org/index.php/IEET/more/hughes20100105/Problems

I'll just comment on what most people are missing, because most reactions seem to be missing a similar thing.

Wei explains that most of the readership are preference utilitarians, who believe in satisfying people's preferences, not maximizing pleasure.

That's fine enough, but if you think that we should take into account the preferences of creatures that could exist, then I find it hard to imagine that a creature would prefer not to exist, than to exist in a state where it permanently experiences amazing pleasure.

Given that potential creatures outnumber existing creatures many times over, the preferences of existing creatures - that we wish to selfishly keep the universe's resources to ourselves, so we can explore and think and have misguided lofty impressions about ourselves, and whatnot - all of those preferences don't count that much in the face of many more creatures that would prefer to exist, and be wireheaded, than not to exist at all.

The only way preference utilitarianism can avoid the global maximum of Heaven is to ignore the preferences of potential creatures. But that is selfish.

If you don't want Heaven, then you don't want a universally friendly AI. What you really want is an AI that is friendly just to you.

If you are a utilitarian, and you believe in shut-up-and-multiply, then the correct thing for the FAI to do is to use up all available resources so as to maximize the number of beings, and then induce a state of permanent and ultimate enjoyment in every one of them. This enjoyment could be of any type - it could be explorative or creative or hedonic enjoyment as we know it. The most energy efficient way to create any kind of enjoyment, however, is to stimulate the brain-equivalent directly. Therefore, the greatest utility will be achieved by wire-heading. Everything else falls short of that.

That's why utilitarianism is a bad idea, expecially if you are allowed to modify the agents. Think about it, humans would be a terrible waste of energy if their only purpose was to have their hedonic pleasure maximized. Mice would be more efficient. Or monocellular organisms. Or registers inside the memory of a computer that get incremented as fast as possible.

What I don't quite understand is why everyone thinks that this would be such a horrible outcome. As far as I can tell, these seem to be cached emotions that are suitable for our world, but not for the world of FAI. In our world, we truly do need to constantly explore and create, or else we will suffer the consequences of not mastering our environment.

You have it backwards. Why do you need not to suffer the consequences of not mastering our environment?

In a world where FAI exists, there is no longer a point, nor even a possibility, of mastering our environment. The FAI masters our environment for us, and there is no longer a reason to avoid hedonic pleasure. It is no longer a trap.

There is no longer any "us" in your hedonic "heaven". It is a world populated by minimalistic agents, all equal to each other, with no memories, no sense of personal identity, no conscious experiences. Life and death would be meanignless concepts to those things, like any other concept, since they wouldn't be capable of anything close to what we call thinking.

Is that what you want the world to become?

Since the FAI can sustain us in safety until the universe goes poof, there is no reason for everyone not to experience ultimate enjoyment in the meanwhile. In fact, I can hardly tell this apart from the concept of a Christian Heaven, which appears to be a place where Christians very much want to get.

Yes, and in fact the Christian Heaven is not a coherent concept. There can't be happiness without pain. No satisfaction without unquenched desire. If you give an agent anything it can possibly ever want, it stops being an agent.

These parallels reinforce my belief that Singularitarianism is just a thinly veiled version of Christianity.

What of Buddhist nirvana as another goal? Buddhist post-nirvana in a transhuman universe could consist of literally obliterating one's consciousness and becoming one with the universe, but it could be many things.