The Fun Theory Sequence

post by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2009-01-25T11:18:23.000Z · LW · GW · Legacy · 30 comments

(A shorter gloss of Fun Theory is "31 Laws of Fun", which summarizes the advice of Fun Theory to would-be Eutopian authors and futurists.)

Fun Theory is the field of knowledge that deals in questions such as "How much fun is there in the universe?", "Will we ever run out of fun?", "Are we having fun yet?" and "Could we be having more fun?"

Many critics (including George Orwell) have commented on the inability of authors to imagine Utopias where anyone would actually want to live.  If no one can imagine a Future where anyone would want to live, that may drain off motivation to work on the project.  The prospect of endless boredom is routinely fielded by conservatives as a knockdown argument against research on lifespan extension, against cryonics, against all transhumanism, and occasionally against the entire Enlightenment ideal of a better future.

Fun Theory is also the fully general reply to religious theodicy (attempts to justify why God permits evil).  Our present world has flaws even from the standpoint of such eudaimonic considerations as freedom, personal responsibility, and self-reliance.  Fun Theory tries to describe the dimensions along which a benevolently designed world can and should be optimized, and our present world is clearly not the result of such optimization.  Fun Theory also highlights the flaws of any particular religion's perfect afterlife - you wouldn't want to go to their Heaven.

Finally, going into the details of Fun Theory helps you see that eudaimonia is complicated - that there are many properties which contribute to a life worth living.  Which helps you appreciate just how worthless a galaxy would end up looking (with very high probability) if the galaxy was optimized by something with a utility function rolled up at random.  This is part of the Complexity of Value Thesis and supplies motivation to create AIs with precisely chosen goal systems (Friendly AI).

Fun Theory is built on top of the naturalistic metaethics summarized in Joy in the Merely Good; as such, its arguments ground in "On reflection, don't you think this is what you would actually want for yourself and others?"

Posts in the Fun Theory sequence (reorganized by topic, not necessarily in the original chronological order):


Comments sorted by oldest first, as this post is from before comment nesting was available (around 2009-02-27).

comment by Jadagul · 2009-01-25T14:10:13.000Z · LW(p) · GW(p)

It occurred to me at some point that Fun Theory isn't just the correct reply to Theodicy; it's also a critical component of any religious theodicy program. And one of the few ways I could conceive of someone providing major evidence of God's existence.

That is, I'm fairly confident that there is no god. But if I worked out a fairly complete version of Fun Theory, and it turned out that this really was the best of all possible worlds, I might have to change my mind.

comment by Roko · 2009-01-25T14:25:31.000Z · LW(p) · GW(p)

Unfortunately, it seems to me that moral anti-realism and axiological anti-realism place limits on our ability to "optimize" the universe.

To put the argument in simple terms:

  1. Axiological/Moral anti-realism states that there are no categorically good states of the universe. On this we agree. The goodness of states of the universe is contingent upon the desires and values of those who ask the question; in this case us.

  2. Human minds can only store a finite amount of information in our preferences. Humans who have spent more time developing their character beyond the evolutionarily programmed desires [food, sex, friendship, etc] will fare slightly better than those who haven't, i.e. their preferences will be more complicated. But probably not by very much, information theoretically speaking. The amount of information your preferences can absorb by reading books, by having life experiences, etc is probably small compared to the information implicit in just being human.

  3. The size of the mutually agreed preferences of any group of humans will typically be smaller than the preferences of any one human. Hence it is not surprising that in the recent article on "Failed Utopia 4-2" there was a lot of disagreement regarding the goodness of this world.

  4. The world that we currently live in here in the US/UK/EU fails to fulfill a lot of the base preferences that are common to all humans, with notable examples being the dissatisfaction with the opposite sex, boring jobs, depression, aging, etc, etc...

  5. If one optimized over these unfulfilled preferences, one would get something that resembled - for most people - a low grade utopia that looked approximately like Banks' Culture. This low grade utopia would probably only be a small amount of information away from the world we see today. Not that it isn't worth doing, of course!

This explains a lot of things. For example, the change of name of the WTA from "transhumanist" to "humanity plus". Humanity plus is code for "low grade utopia for all". "Transhumanist" is code for futures that various oddball individuals envisage in which they (somehow) optimize themselves way beyond the usual human preference set. These two futures are eminently compatible - we can have them both, but most people show no interest in the second set of possibilities. It will be interesting to think about the continuum between these two goals. It's also interesting to wonder whether the goals of "radical" transhumanists might be a little self-contradictory. With a limited human brain, you can (as a matter of physical fact) only entertain thoughts that constrain the future to a limited degree. Even with all technological obstacles out of the way, our imaginations might place a hard limit on how good a future we can try to build for ourselves. Anyone who tries to exceed this limit will end up (somehow) absorbing noise from their environment and incorporating it into their preferences. Not that I have anything against this - it is how we got our preferences in the first place - though it is not a strong motivator for me to fantasize about spending eternity fulfilling preferences that I don't have yet and which I will generate at random at some point in the future when I realize that my extant preferences have "run out of juice".

This, I fear, is a serious torpedo in the side of the transhumanist ideal. I eagerly await somebody proving me wrong here...

comment by Vladimir_Nesov · 2009-01-25T15:31:23.000Z · LW(p) · GW(p)

Roko, preferences are not flat, they depend and act on the state of the world in general and on themselves in particular. They can grow very detailed, and include a state quite remote from current world as desirable. The problem with derailed aspect of transhumanism in not in remoteness from currently human, but in mistaken preferences arrived at mostly by blind leaps of imagination. We define the preferences over remote future implicitly, without being able to imagine it, only gradually becoming able to actually implement them, preserving or refining the preference through growth.

comment by Roko · 2009-01-25T15:53:26.000Z · LW(p) · GW(p)

I response to my own question: I think that the information difference between innate biological prefs that we have and explicitly stated preferences is a lot bigger than I thought.

For example, I can state the following:

(1) I wish to be smart enough to understand all human science and mathematics published to this date, and to solve all outstanding scientific and philosophical questions including intelligence, free will and ethics. I want to know the contents and meaning of every major literary work in print and every major film, to understand the history of every major civilization, to fall in love with the person who is most compatible with me in the world.

Now if I make all these wishes, how much have I cut down future states of the universe? How much optimizing power in bits have I wished for?

I expressed the wish in about 330 characters, which according to Shannon means I have expressed 330 bits of information, roughly equivalent to specifying the state of a 20X20 grid of pixels each one of which can be either on or off. I feel that this is something of an underestimate in terms of how much I have cut down future states of the universe. Another way of calculating the complexity of the above wish is to bound it by the log of the number of psychologically distinguishable states of my mind. Given the FHI brain emulation roadmap, this upper bound could be a very large number indeed. Here is another ~300-char wish:

(2) I want to be as rich as Bill Gates. I want to have ten mansions, each with ten swimming pools and a hundred young, willing female virgins to cater to my every whim. I want my own private army and an opposing force who I will trounce in real combat every weekend. I want an 18-inch penis and muscles the size of Arnie in his younger days, and I want to be 6'7''. I want to be able to eat galaxy chocolate flavored ice cream all day without getting fat or getting bored with it. I want a car that goes at 5000 miles an hour without any handling problems or danger of accident, and I want to be able to drive it around the streets of my city and leave everyone in the dust.

Now it appears to me that this wish probably did only cut down the future by 300 bits... that it is a far less complex wish than the first one I gave. Presumably the difference between those who end up in low grade heaven and those who end up as superintelligent posthumans inhabiting a Dyson sphere, or having completely escaped from our physics lies in the difference between the former wish and the latter. Again, it is fruitful and IMO very important to explore the continuum between the two.

comment by Carl_Shulman · 2009-01-25T17:39:16.000Z · LW(p) · GW(p)

Roko, the Minimum Message Length of that wish would be MUCH greater if you weren't using information already built into English and our concepts.

comment by Jon2 · 2009-01-25T18:45:20.000Z · LW(p) · GW(p)

I can certainly understand your dissatisfaction with medieval depictions of heaven. However, your description of fun theory reminds me of the Garden of Eden. i.e. in Genesis 1-2, God basically says:

"I've created the two of you, perfectly suited for one another physically and emotionally, although the differences will be a world to explore in itself. You're immortal and I've placed you in a beautiful garden, but now I'm going to tell you to go out and be fruitful and multiply and fill the earth and subdue it and have dominion over all living things; meaning build, create, procreate, invent, explore, and enjoy what I've created, which by the way is really really big and awesome. I'll always be here beside you, and you'll learn to live in perfect communion with me, for I have made you in my own image to love the process of creation as I do. But if you ever decide that you don't want that, and that you want to go it alone, rejecting my presence and very existence, then there's this fruit you can take and eat. But don't do it, because if you do, you will surely die."

It seems that the point of disagreement is that your utopia doesn't have an apple. The basic argument of theodicy is that Eden with the apple is better than Eden sans apple. To the extent that free will is good, a utopia must have an escape option.

Or, to put it another way, obedience to the good is a virtue. Obedience to the good without the physical possibility evil is a farce.

It's easy to look around and say, "How could a good God create THIS." But the real question is, "How could a good God create a world in which there is a non-zero probability of THIS."

Replies from: tygorton, Odinn
comment by tygorton · 2012-05-19T09:02:34.506Z · LW(p) · GW(p)

This logic assumes that a beyond human intelligence in a redesigned world would still find inherent value in free will. Isn't it possible that such an intelligence would move beyond the need to experience pain in order to comprehend the value of pleasure?

According to the bible, god created different aspects of the world across six days and after each creation he "saw that it was good". Yet nothing ELSE existed. If there had never been a "world" before, and evil had not yet been unleashed, by what method was this god able to measure that his creation was good? One must assume that god's superior intelligence simply KNEW it to be good and had no need to measure it against something "bad" in order to know it. Couldn't the eventual result of AI be the attainment of the same ability... the ability to KNOW pleasure without the existence of its opposite?

Isn't the hope (or should I say fun?) of considering the potential of AI that such a vast intelligence would move life BEYOND the anchors to which we now find ourselves locked? If AI is simply going to be filled with the same needs and methods of measuring "happiness" as we currently deal with, what is the point of hoping for it at all?

This is a bit of an afterthought, but even at our current level of intelligence, humans have no way of knowing if we would value pleasure if pain did not exist. Pain does now and has always existed. "Evil" (or what we perceive as evil) has existed since the dawn of recorded human existence. How can we assume that we are not already capable of recognizing pleasure as pleasure and good as good without their opposites to compare them to? We have never had the opportunity to try.

Replies from: thespianic
comment by thespianic · 2012-09-21T14:57:29.602Z · LW(p) · GW(p)

I beg to differ on the aspect of there being non-existence predating the creation. A subtle nuance in the first verse of Genesis offers an insight into this. Gen 1:1 "In the beginning God created the heavens and the earth. And the earth was without form, and void; and darkness was upon the face of the deep." Original manuscripts offer a translation that is closer to "and the earth 'became' without form (sic), and void". It may so very well be that in the assumption that God looked on his creation and saw that it was good, there was a pre-existential basis for this. Also to point out another simple example, there would be no record of wrong without a sort of legal system that says that an act is actually defined as wrong. I agree with the idea that there had to be an apple in the garden to bring to the front the difference between good and bad. Utopia can therefore only exists where there is an understanding or mere knowledge of dystopia.

comment by Odinn · 2015-08-03T03:06:47.602Z · LW(p) · GW(p)

I knew there would come a day when almost a decade of mandatory bible classes in private school would pay off. (That's not true, I've generally written it off as a really depressing waste of my mental resources... still) You've got the order of events in the Garden of Eden backwards. After God finished up and took off for Miller Time, Adam and Eve had nothing to do. They didn't need clothes or shelter, all animals were obedient and gentle, they had to live of fruit for eternity which would get old, the weather and season (singular) was always the same and they were the only those two people in existence with no concept of there ever being any more. Sure, they would have lived forever, but there was no challenge, inspiration, reason or stimulation. Only AFTER the forbidden fruit and the knowledge of good and evil does God start up Eve's biological clock and issue the 'be fruitful and multiply' command, society starts to develop, there's a ton of implicit incest (er... bonus?) and they can cook up a nice lamb shank to break up the monotony. Once again, the literal interpretation of the bible leaves a lot to be desired in a literary sense, because the Garden of Eden is one of the most depressing 'paradises' ever devised. Also, here I go again responding to many-years-cold comments.

Replies from: None
comment by [deleted] · 2015-08-03T03:45:45.417Z · LW(p) · GW(p)

and they can cook up a nice lamb shank to break up the monotony.

Well, no. That's not until Noah is issued permission to eat meat after the Flood.

because the Garden of Eden is one of the most depressing 'paradises' ever devised

It's not that depressing. It's just a park. The depressing part is that God gets angry and says, "Oh, you don't want to spend 100% of all your existence in this park for all eternity with literally nothing else? FUCK YOU AND LITERALLY DIE." A good God would have allowed much larger portions of possible life-space to be explored with fewer or even no penalties.

Eden is indeed more interesting for having the Apple, but damnation is so totally uninteresting that religious people had to go and invent Redemption, which is the simpering and undignified version of having your cake and eating it too.

comment by Matt5 · 2009-01-25T21:21:25.000Z · LW(p) · GW(p)

Apparently having 72 virgins at your disposal is a utopia for many. EY should look into this...

comment by nazgulnarsil3 · 2009-01-26T06:17:17.000Z · LW(p) · GW(p)

but an eden with a reversible escape option is surely better than an eden with a non-reversible escape option yes?

Replies from: Rinon
comment by Rinon · 2012-08-06T15:24:00.457Z · LW(p) · GW(p)

Most religions believe that the escape option is reversible - otherwise there wouldn't be much point.

comment by Roko · 2009-01-26T17:08:23.000Z · LW(p) · GW(p)

@ Carl Shulman

yes, I am aware that human "concepts" are acting as a big multiplier on how much you can wish for in a small number of words. But I want to know whether certain wishes make better use ir worse use of this, and I want toget some idea of exactly how much more a human can feasibly wish for.

I think that by using established human concepts to make a wish ("I want to understand and solve all current scientific problems"), you are able to constrain the future more, but you have less understanding of what you'll actually get. You trade in some safety and get more mileage.

comment by Roko · 2009-01-26T17:09:45.000Z · LW(p) · GW(p)

@ Nesov: "Roko, preferences are not flat..."

I don't understand quite what you're saying. Perhaps it would help if i attempt to make my own post a bit clearer.

comment by Vladimir_Nesov · 2009-01-26T18:57:22.000Z · LW(p) · GW(p)

@Roko: As I understood, one of the points you made was about how preferences of both individual people and humanity as a whole are quite coarse-grained, and so strong optimization of environment is pointless. Beyond certain precision, the choices become arbitrary, and so continuing systematic optimization, forcing choices to be non-arbitrary from the updated perspective, basically consists in incorporating noise into preferences.

I reply that a formula for pi can be written down in much fewer bytes that it'd take to calculate 10000th digit in its decimal expansion. Human embodying a description of morality, just like note containing the formula for pi, can have no capacity for imagining (computing) some deeper property of that description, and still precisely determine that property. What we need in both cases is a way around the limitations of media presenting the description, without compromising its content.

comment by Roko · 2009-01-27T00:46:37.000Z · LW(p) · GW(p)


Yes, you understood my message correctly, and condensed it rather well.

Now, what would it mean for human axiology to be like pi? A simple formula that unfolds into an "infinitely complex looking" pattern? Hmmm. This is an interesting intuition.

If we treat our current values as a program that will get run to infinity in the future, we may find that almost all of the future output of that program is determined by things that we don't really think of as being significant; for example, very small differences in the hormone levels in our brains when we first ask our wish granting machine for wishes.

I would only count those features of the future that are robust to very small perturbations in our psychological state to be truly the result of our prefs. On the other hand, features of the future that are entirely robust to our minds are also not the result of our prefs.

And still there is the question of what exactly this continued optimization would consist of. the 100th digit of pi makes almost no difference to its value as a number. Perhaps the hundredth day after the singularity will make almost no difference to what our lives are like in some suitable metric. Maybe it really will look like calculating the digits of pi: pointless after about digit number 10.

To satisfy the robustness criterion and this nonconvergence criterion seems hard.

comment by Vladimir_Nesov · 2009-01-27T02:33:13.000Z · LW(p) · GW(p)

If a computer program computes pi to 1,000,000 instead of 100 places, it doesn't make the result more dependent on thermal noise. You can run arbitrarily detailed abstract computations, without having the outcome depend on irrelevant noise. When you read a formula for pi from a note, differences in writing style don't change the result. AI should be only more robust.

Think of digits playing out in time, so that it's important to get each of them right at the right moment. Each later digit could be as important in the future as earlier digits now.

comment by Roko · 2009-01-27T16:07:43.000Z · LW(p) · GW(p)


It is an open question whether our values and our lives will behave more like you have described or not.

For a lot of people, the desire to conform and not to be too weird by current human standards might make them converge over time. These people will live in highly customized utopias that suit their political and moral views, e.g. Christians in a mini-world where everyone has had their mind altered so that they can't doubt God, can't commit any sin, etc. E.g. ordinary modern semi-hedonists who live in something like the Culture. (Like pi as a number we use for engineering: the digits after the 100th convey almost no new information)

For others, boredom and curiosity will push them out into new territory. But the nature of both of these emotions is to incorporate environmental noise into one's prefs. They'll explore new forms of existence, new bodies, new emotions, etc, which will make them recursively weirder. These people will behave like the co-ordinates of a chaotic system in phase space: they will display very very high sensitivity to initial conditions, chance events "oh look, a flock of birds. I wonder what it would be like to exist as a swarm intelligence. I know, I'll try it".

The only group of people who I can see behaving the way you want are scientists. We have an abstract desire to understand how the world works. We will alter ourselves to become more intelligent in order to do so, and we have no idea what we will discover along the way. We are surely in for surprises as big as the discovery of evolution and quantum mechanics. Each new level of intelligence and discovery will be truly new, but hopefully the nature of truth is an abstract universal invariant that doesn't depend upon the details of the path you take to get to it.

In essence, scientists are the only ones for whom long term optimization of our world has the kind of unbounded value that singularitarians want. Ordinary people will only get a limited amount of value out of a positive singularity. Thus their apathy about it is understandable.

comment by Roko · 2009-01-27T16:12:34.000Z · LW(p) · GW(p)

Thus their apathy about it is understandable.

... given that they don't think it is very likely, and they discount the future.

Note that I used "scientist" in a very general sense: anyone who really wants to understand reality for the sake of understanding it, anyone who has that natural curiosity.

comment by Abigail · 2009-01-29T15:02:52.000Z · LW(p) · GW(p)

I want to be able easily to come back to this. Would you create a category of "Sequences", and post it to that, so that there is a link on the side bar? I think there is at least another such sequence.

comment by Uni · 2011-03-28T22:05:22.475Z · LW(p) · GW(p)

Eliezer_Yudkowsky wrote: "We don't want the AI's models of people to be people - we don't want conscious minds trapped helplessly inside it. So we need how to tell that something is definitely not a person, and in this case, maybe we would like the AI itself to not be a person, which would simplify a lot of ethical issues if we could pull it off. Creating a new intelligent species is not lightly to be undertaken from a purely ethical perspective; if you create a new kind of person, you have to make sure it leads a life worth living."

I do want AI models of people to be people. If an AI model of a human lacks the important part of the person that we call emotions, there is the risk that it will be a dangerously faulty model of that person. If it can't suffer, it can't understand why humans so eagerly want to get rid of suffering, and so it may in many situations think it knows better than us what is most important to achieve. It may regard the diminishing of suffering in humans just as irrational as we would regard turning the planet into paper clips.

From a utilitarian perspective, even from a negative utilitarian perspective, building AI models of humans and equipping then with human emotions and ability to suffer would not necessarily be unethical. It might be what would in the long run minimize the suffering. Trying to minimize suffering too fast can be dangerous; it might take away something we really value, along with the suffering, even if we don't value the suffering in and of itself. Future superhuman AI will know how to get rid of suffering in a completely safe way, once they have become in every relevant way immensely smarter than humans - but perhaps only if the first transhuman AI we build are equipped with the ability to suffer just like humans. You often have to know a problem well before you can safely solve it, and you can't really know suffering unless you actually feel it. So let's not hesitate to create AI that can suffer (once we can create such AI).

Happiness will grow immensely and suffering will and should be abolished later on, but let's not rush it during the process of creating AI models of humans.

Replies from: nshepperd
comment by nshepperd · 2011-03-28T22:53:51.519Z · LW(p) · GW(p)

We're talking about giving the models subjective experience, not just "emotions". You want the AI to create conscious minds inside itself and torture them to find out whether torture is bad? And then again every time it makes a decision where torture is a conceivable outcome? I'd hope we can give the AI a model that accurately predicts how humans react to stimuli without creating a conscious observer. Humans seem to be able to do that, at least..

Beware of anthropomorphizing AIs. A Really Powerful Optimization Process shouldn't need to "suffer" for us to tell it what suffering is, and that we would like less of it.

comment by Uni · 2011-03-28T22:26:57.253Z · LW(p) · GW(p)

When we have gained total control of all the matter down to every single particle within, say, our galaxy, and found out exactly what kinds of combinations we need to put particles together in to maximize the amount of happiness produced per particle used (and per spacetime unit), then what if we find ourselves faced with the choice between 1) maximizing happiness short term but not getting control over more of the matter in the universe at the highest possible rate (in other words, not expanding maximally fast in the universe), and 2) maximizing said expansion rate at the cost of short term happiness maximation. What if this trade-off problem persists forever?

We might find ourselves in the situation where we, time after time, can either use all of our matter for maximizing the pace at which we take control over more and more matter, creating no short term happiness at all, or creating any non-zero amount of happiness short term at the expense of our ability to expand our ability to get us much more happiness in the future instead. We might find that, hey, if we postpone being happy for one year, we can be ten times as happy next year as we would otherwise be able to be, and that's clearly better. And next year, we are again in the same situation: postponing being happy one more year again seems rational. Next year, same thing. And so on.

Suppose that kind of development would never end, unless we ended it by "cashing in" (choosing short term happiness before maximum development). Then when should we "cash in"? After how many years? Any finite number of years seems too small, since you could always add one extra year to further improve the expected long term happiness gain. On the other hand, the answer "in infinitely many years from now" is not appealing either, as an infinity of years never passes, by definition, meaning we would never choose to be happy. So, when would you "cash in" and choose to be happy? After how many years?

Replies from: DSimon, rkyeun
comment by DSimon · 2011-09-16T07:01:09.780Z · LW(p) · GW(p)

This is an interesting problem. The correct solution probably lies somewhere in the middle: allocate X of our resources to expansion, and 1-X of our resources to taking advantage of our current scope.

comment by rkyeun · 2012-07-28T23:10:10.367Z · LW(p) · GW(p)

The maximum happy area for a happy rectangle is when both its happy sides are of equal happy length, forming a happy square.

comment by fuzzyshark · 2013-06-01T04:21:16.736Z · LW(p) · GW(p)

Who are you? It's way out there without question. If God conceives a circle, and then a square. (pi: pizza, fried, Jamaican, etc.) as a physical thing held in front of a mirror, it will always reflect an image of himself. If God flashes the image of a word. Pie. The reverse mirror image of 413 is the English word - PIE. Then the answer is approximately the circle squared. God exist inside a room of mirrors. All God can see is him. Could get boring after eternity. Unless God has fun.

comment by fuzzyshark · 2013-06-01T04:22:04.053Z · LW(p) · GW(p)


comment by [deleted] · 2015-04-09T19:25:56.963Z · LW(p) · GW(p)

This is an awesome sequence.

comment by [deleted] · 2016-02-18T01:17:25.388Z · LW(p) · GW(p)

There's a discount rate to money...How many years of your life would you have to get back for giving back everything you earned...the onlder you get the smaller that number gets...when you're on your deathbed you will give up every dollar in the bank for a few more days...You realise as you get older that it matters less and less and less

  • 56 minutes into Tim Ferris's second interview with Naval here

This really chiseled in the face validity of the eudemonia theory of hedonic value to me. Even if life was just like it is for me today, just that supply of stimulation feels better than the imagined counterfactual of non-experience.