High Challenge
post by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2008-12-19T00:51:08.000Z · LW · GW · Legacy · 75 commentsThere's a class of prophecy that runs: "In the Future, machines will do all the work. Everything will be automated. Even labor of the sort we now consider 'intellectual', like engineering, will be done by machines. We can sit back and own the capital. You'll never have to lift a finger, ever again."
But then won't people be bored?
No; they can play computer games—not like our games, of course, but much more advanced and entertaining.
Yet wait! If you buy a modern computer game, you'll find that it contains some tasks that are—there's no kind word for this—effortful. (I would even say "difficult", with the understanding that we're talking about something that takes 10 minutes, not 10 years.)
So in the future, we'll have programs that help you play the game—taking over if you get stuck on the game, or just bored; or so that you can play games that would otherwise be too advanced for you.
But isn't there some wasted effort, here? Why have one programmer working to make the game harder, and another programmer to working to make the game easier? Why not just make the game easier to start with? Since you play the game to get gold and experience points, making the game easier will let you get more gold per unit time: the game will become more fun.
So this is the ultimate end of the prophecy of technological progress—just staring at a screen that says "YOU WIN", forever.
And maybe we'll build a robot that does that, too.
Then what?
The world of machines that do all the work—well, I don't want to say it's "analogous to the Christian Heaven" because it isn't supernatural; it's something that could in principle be realized. Religious analogies are far too easily tossed around as accusations... But, without implying any other similarities, I'll say that it seems analogous in the sense that eternal laziness "sounds like good news" to your present self who still has to work.
And as for playing games, as a substitute—what is a computer game except synthetic work? Isn't there a wasted step here? (And computer games in their present form, considered as work, have various aspects that reduce stress and increase engagement; but they also carry costs in the form of artificiality and isolation.)
I sometimes think that futuristic ideals phrased in terms of "getting rid of work" would be better reformulated as "removing low-quality work to make way for high-quality work".
There's a broad class of goals that aren't suitable as the long-term meaning of life, because you can actually achieve them, and then you're done.
To look at it another way, if we're looking for a suitable long-run meaning of life, we should look for goals that are good to pursue and not just good to satisfy.
Or to phrase that somewhat less paradoxically: We should look for valuations that are over 4D states, rather than 3D states. Valuable ongoing processes, rather than "make the universe have property P and then you're done".
Timothy Ferris is again worth quoting: To find happiness, "the question you should be asking isn't 'What do I want?' or 'What are my goals?' but 'What would excite me?'"
You might say that for a long-run meaning of life, we need games that are fun to play and not just to win.
Mind you—sometimes you do want to win. There are legitimate goals where winning is everything. If you're talking, say, about curing cancer, then the suffering experienced by even a single cancer patient outweighs any fun that you might have in solving their problems. If you work at creating a cancer cure for twenty years through your own efforts, learning new knowledge and new skill, making friends and allies—and then some alien superintelligence offers you a cancer cure on a silver platter for thirty bucks—then you shut up and take it.
But "curing cancer" is a problem of the 3D-predicate sort: you want the no-cancer predicate to go from False in the present to True in the future. The importance of this destination far outweighs the journey; you don't want to go there, you just want to be there. There are many legitimate goals of this sort, but they are not suitable as long-run fun. "Cure cancer!" is a worthwhile activity for us to pursue here and now, but it is not a plausible future goal of galactic civilizations.
Why should this "valuable ongoing process" be a process of trying to do things—why not a process of passive experiencing, like the Buddhist Heaven?
I confess I'm not entirely sure how to set up a "passively experiencing" mind. The human brain was designed to perform various sorts of internal work that add up to an active intelligence; even if you lie down on your bed and exert no particular effort to think, the thoughts that go on through your mind are activities of brain areas that are designed to, you know, solve problems.
How much of the human brain could you eliminate, apart from the pleasure centers, and still keep the subjective experience of pleasure?
I'm not going to touch that one. I'll stick with the much simpler answer of "I wouldn't actually prefer to be a passive experiencer." If I wanted Nirvana, I might try to figure out how to achieve that impossibility. But once you strip away Buddha telling me that Nirvana is the end-all of existence, Nirvana seems rather more like "sounds like good news in the moment of first being told" or "ideological belief in desire" rather than, y'know, something I'd actually want.
The reason I have a mind at all, is that natural selection built me to do things—to solve certain kinds of problems.
"Because it's human nature" is not an explicit justification for anything. There is human nature, which is what we are; and there is humane nature, which is what, being human, we wish we were.
But I don't want to change my nature toward a more passive object—which is a justification. A happy blob is not what, being human, I wish to become.
I earlier argued that many values require both subjective happiness and the external objects of that happiness. That you can legitimately have a utility function that says, "It matters to me whether or not the person I love is a real human being or just a highly realistic nonsentient chatbot, even if I don't know, because that-which-I-value is not my own state of mind, but the external reality." So that you need both the experience of love, and the real lover.
You can similarly have valuable activities that require both real challenge and real effort.
Racing along a track, it matters that the other racers are real, and that you have a real chance to win or lose. (We're not talking about physical determinism here, but whether some external optimization process explicitly chose for you to win the race.)
And it matters that you're racing with your own skill at running and your own willpower, not just pressing a button that says "Win". (Though, since you never designed your own leg muscles, you are racing using strength that isn't yours. A race between robot cars is a purer contest of their designers. There is plenty of room to improve on the human condition.)
And it matters that you, a sentient being, are experiencing it. (Rather than some nonsentient process carrying out a skeleton imitation of the race, trillions of times per second.)
There must be the true effort, the true victory, and the true experience—the journey, the destination and the traveler.
75 comments
Comments sorted by oldest first, as this post is from before comment nesting was available (around 2009-02-27).
comment by Cameron_Taylor · 2008-12-19T03:30:56.000Z · LW(p) · GW(p)
We'll reject the games where everything is too easy. We will not want to live in a universe which is as we wish it were. It'll be unhealthy for us. We'll throw it off. Instead, we'll play games that are just as hard and frustrating as the life we live now.
Then, the robots will discover a way to circumvent thermodynamics and harness us as batteries and we'll all start saying "Woah!". Plus all the time playing computer games with help will make us so stupid that we give superintelligent, super-fast hostile ems a warning like "Dodge this!" before we shoot them because we don't comprehend their abilities. (Perhaps they meant that as a metaphor?)
comment by D._Alex · 2008-12-19T04:56:15.000Z · LW(p) · GW(p)
On the subject of computer games (an underrated area for the study of psychology, economy and even AI, IMO):
During the last 3 years, I have spent just over 1000 hours playing World of Warcraft. Why did I choose to spend (some, incl my wife, might say waste) my time on this? I am fairly wealthy and quite fit - just about any fun activity is open to me. So why do I like WoW? And why 10 million people around the world do the same?
Some important reasons why the game is so pleasurable seem to be:
a) the ultimate goals are pretty clear (so unlike real life...)
b) the "measures of progress" are likewise clear - and there is only one way to go, namely "up"! (again, so unlike real life, apart from possibly "youth" - is that what makes "youth" so good?)
c) the rewards are clear - and are earned so progressively that playing the game seems akin to wireheading (the trickle of XPs, "gold" and new items continuously stimulates some pleasure centre or other).
(Good play involves fairly sophisticated analysis, strategy and tactics, which maintains intetrest... but is beside the point I want to make. The game is attractive to good and poor players.)
With this on the table, I would like to offer the following for comments:
1) Can the 3 items above (clarity of goals, clear measures of achievement, progressive rewards) be mapped onto real life in a way that makes life more, um, fun? There are some mechanisms like that already - money most clearly, yet pursuit of money as a goal is not considered that worthwhile.
2) this is a bit out of left field, but: Is a setting like World of Warcraft a good medium for development of AI? Clear goals, clear measures of progress, sufficient complexity to provide an indication of when important insights are achieved, and a safe environment (in the sense that the path to paperclip AI seems unlikely)...?
(For Robin Hanson: have you heard about the economic studies carried out in WoW setting?)
OK, duty calls... I have more to say on this when time permits.
D. Alex
Replies from: Turgurth, Ksenia_Sazonova↑ comment by Ksenia_Sazonova · 2016-08-07T07:06:34.068Z · LW(p) · GW(p)
The life is entertaining too. If you want to gamify it, use https://habitica.com you'll set your goals, see your progress and achieve the rewards. What else is there to want to?
comment by Roland2 · 2008-12-19T05:12:43.000Z · LW(p) · GW(p)
@ D. Alex: Some important reasons why the game is so pleasurable seem to be:
a) the ultimate goals are pretty clear (so unlike real life...)
b) the "measures of progress" are likewise clear -
c) the rewards are clear -
This looks like real life without the hard parts. Sure, it makes it more fun, but at the end will you feel rewarded? If you look back now or in a few years to the time spent playing and consider what you could have achieved in real life if you invested the same time into real challenges how will you feel? From my own experience I can tell you that I regret every minute I wasted playing stupid games. Nowadays I still play chess ocasionally to relax, but I'm successfully getting rid of that habit. I avoid overly immersive/addictive games like the plague.
Replies from: MarsColony_in10years↑ comment by MarsColony_in10years · 2015-11-03T17:19:55.371Z · LW(p) · GW(p)
Sounds like WoW is optimized for System 1 pleasures, and you explicitly reject this. I think that brings up an important point: How can we build a society/world where there are strong optimization forces to enable people to choose System 2 preferences? Once such a world iterated on itself for a couple generations, what might it look like?
I don’t think this would be a world with no WoW-like activities, because a world without any candy or simple pleasures strikes me as deeply lacking. My System 2 seems to place at least a little value on System 1 being happy. So I’d guess the world would just have many fewer of such activities, and be structured in such a way as to make it easy to avoid choices we’d regret the next day.
If this turns out to a physically impossible problem to overcome for some reason, then I could imagine a world with no System 1 pleasures, but such a world would be deeply lacking, even if that loss was more than made up for by gains in our System 2 values.
As a side note, it'd be an interesting question how much of the theoretical per capita maximum value falls into which categories. An easier question is how much of our currently actualized value is immediate gratifications. I'd expect that to be heavily biased toward System 1, since we suffer from Akrasia, but it might still be informative.
Replies from: Lumifer, VoiceOfRa↑ comment by Lumifer · 2015-11-03T17:38:29.990Z · LW(p) · GW(p)
How can we build a society/world where there are strong optimization forces to enable people to choose System 2 preferences?
I think the real world qualifies quite well. People who listen to their System 2 achieve much more than people who are slaves to their System 1.
If you want stronger "optimization forces", take away the safety net. Hunger and pain are excellent incentives. Not many people would allow themselves to get addicted to WoW if it means they'll become homeless in a short while.
Replies from: MarsColony_in10years, None, ChristianKl↑ comment by MarsColony_in10years · 2015-11-03T21:07:09.628Z · LW(p) · GW(p)
That provided me with some perspective. I'd only been thinking of cases where we imposed limitations, such as those we use with Alcohol and addictive drugs. But, as you point out, there are also regulations which push us toward immediate gratification, rather than away. If, after much deliberation, we collectively decide that 99% of potential values are long term, then perhaps we'd wind up abolishing most or all such regulations, assuming that most System 2 values would benefit.
However, at least some System 2 values are likely orthogonal to these sorts of motivators. For instance, perhaps NaNoWriMo participation would go down in a world with fewer social and economic safety nets, since many people would be struggling up Maslow's Hierarchy of Needs instead of writing. I'm not sure how large of a fraction of System 2 values would be aided by negative reinforcement. There would be a large number of people who would abandon their long-term goals in order to remove the negative stimuli ASAP. If the shortest path to removing the stimuli gets them 90% of the way toward a goal, then I'd expect most people to achieve the remaining 10%. However, for goals that are orthogonal to pain and hunger, we might actually expect a lower rate of achievement.
If descriptive ethics research shows that System 2 preferences dominate, and if the majority of that weighted value is held back by safety nets, then it'll be time to start cutting through red tape. If System 2 preferences dominate, and the majority of moral weight is supported by safety nets, then perhaps we need more cushions or even Basic Income. If our considered preference is actually to "live in the moment" (System 1 preferences dominate) then perhaps we should optimize for wireheading, or whatever that utopia would look like.
More likely, this is an overly simplified model, and there are other concerns that I'm not taking into account but which may dominate the calculation. I completely missed the libertarian perspective, after all.
↑ comment by [deleted] · 2015-11-11T00:27:02.964Z · LW(p) · GW(p)
If you want stronger "optimization forces", take away the safety net. Hunger and pain are excellent incentives.
Actual experiments in doing this have proven it to be extremely counterproductive. The more human effort needs to be poured into avoiding hunger, homelessness, and base pain, the less ends up available for serving "self-actualizing" goals, conforming to socially-approved-of lifestyles, or even increasing economic productivity.
If you have an intuition which tells you that punishing people makes them act smarter, it is wrong. Punishing people makes them spend mental effort on avoiding getting caught transgressing your norms when they could have spent that effort on something that was actually important.
Replies from: Lumifer↑ comment by Lumifer · 2015-11-11T16:45:26.613Z · LW(p) · GW(p)
the less ends up available for serving "self-actualizing" goals, conforming to socially-approved-of lifestyles,
LOL. For a lot of people "self-actualization" ends up with sitting on a couch in front of an idiot box, eating chips. Nowadays it might be in front of their FB feed, but that's essentially the same. And I'm not sure what are "socially-approved-of lifestyles" -- that seem to depend a lot on the society in question.
If you have an intuition which tells you that punishing people makes them act smarter, it is wrong.
No. My intuition is that the threat of pain/hunger/etc. makes people act. Incentives matter.
Replies from: None↑ comment by [deleted] · 2015-11-13T03:27:37.706Z · LW(p) · GW(p)
LOL. For a lot of people "self-actualization" ends up with sitting on a couch in front of an idiot box, eating chips. Nowadays it might be in front of their FB feed, but that's essentially the same. And I'm not sure what are "socially-approved-of lifestyles" -- that seem to depend a lot on the society in question.
Look, the mere fact that you condescend at and disapprove of the actions of others doesn't mean you've proposed any kind of alternative (no, survivalism does not count, that problem was already solved), let alone demonstrated a metric by which your non-proposed alternative is superior (not even the "I like it" metric).
No. My intuition is that the threat of pain/hunger/etc. makes people act. Incentives matter.
Now explain why those actions or incentives matter, that is, what makes them superior to alternatives. No, sneering does not count.
↑ comment by ChristianKl · 2015-11-11T18:28:05.723Z · LW(p) · GW(p)
I think the real world qualifies quite well. People who listen to their System 2 achieve much more than people who are slaves to their System 1.
I think people who mainly listen to system 2 frequently suffer from akrasia. Productive people usually feel motivated to do what they are doing and that's system 1.
↑ comment by VoiceOfRa · 2015-11-10T00:23:21.029Z · LW(p) · GW(p)
The biggest problem isn't System 1 dominating System 2. It's system 2's being filled with BS and falsehoods.
Replies from: MarsColony_in10years↑ comment by MarsColony_in10years · 2015-11-10T15:09:30.091Z · LW(p) · GW(p)
Excellent point. Most people aren't trying and failing to achieve their dreams. We aren’t even trying. We don’t have well-articulated dreams, so trying isn’t even a reasonable course of action until we have a clear objective. I'd guess that most adults still don't know what they want to be when they grow up, and still haven't figured it out by the time they retire.
Replies from: Lumifer, VoiceOfRa↑ comment by Lumifer · 2015-11-10T18:00:36.460Z · LW(p) · GW(p)
Most people aren't trying and failing to achieve their dreams. We aren’t even trying. We don’t have well-articulated dreams
Evidence or typical mind fallacy..? X-)
Replies from: MarsColony_in10years↑ comment by MarsColony_in10years · 2015-11-10T18:59:21.191Z · LW(p) · GW(p)
Guilty. I've spent most of my life trying to articulate and rigorously define what our goals should be. It takes an extra little bit of cognitive effort to model others as lacking that sense of purpose, rather than merely having lots of different well-defined goals.
(EDIT, to avoid talking past each other: Not that people don't have any well defined sub-goals, mind you. Just not well defined terminal values, and well defined knowledge of their utility function. No well-defined answers to Life, The Universe, And Everything.)
Replies from: Lumifercomment by haig2 · 2008-12-19T05:52:04.000Z · LW(p) · GW(p)
What you describe as targets over '4D states' reminds me of Finite and Infinite Games by James Carse. For an example, playing a game of basketball with a winner/loser after an hour of play is a finite game. However, the sport of basketball overall, is an infinite game. So playing a specific video game to reach a score or pass the final level is a finite game, but being a 'gamer' is an infinite game, allowing ever more types of gaming to take place.
comment by ac · 2008-12-19T06:47:09.000Z · LW(p) · GW(p)
Unlike Roland, who is obviously a puritan, I rather enjoy the occasional spot of idleness. For a non-trivial number of people, playing WoW for a couple of hours a day is more fun that playing real life. Rather than make thinly veiled moral judgements about folks for their unproductivity, perhaps he should consider what makes certain games so engaging.
I spent a year playing a lot of WoW, attaining non-trivial sucess in both raiding and competitive PVP, but I gave it away, partly because the time commitment became too great and partly because what passed for progression started to lose its shine.
So, being suitably qualified, I'll take a stab at a few features that make this virtual social experience psychologically rewarding:
Competition with minimised risk. I love fighting. Seriously - nothing beats the adrenaline buzz, time compression and sheer physicality. Unfortunately even controlled fighting in the physical world entails a level of risk that as a father I'm not willing to assume. Simulated violence, while a poor substitute, helps to fill the void.
Persistent progress. Sure online FPS is fun, but when you log on you're always the same guy (more or less). There's also less risk of losing your shirt in an MMO compared to real life.
Social challenges. Much of the game content consists of elaborate logistical and combinatorial problems that require research, problem solving, and team coordination. All of which are fun.
I see a future where billions of uploaded humans exist primarily in a cartoonish sim, working out how to beat the latest uber boss the AI has dreamed up in order to get phat lewt.
comment by Manuel_Mörtelmaier · 2008-12-19T08:04:23.000Z · LW(p) · GW(p)
Heaven is being perfect.
(Richard Bach, Jonathan Livingston Seagull)
Replies from: SecondWind↑ comment by SecondWind · 2015-09-07T19:19:07.071Z · LW(p) · GW(p)
Jonathan Livingston Seagull doggedly pursues his passion, improving continuously, and as he does so, the needs and limits of his body fall away. By flying, he no longer needs to eat. By flying, he no longer needs to age. By flying, he no longer needs to die. Richard Bach allows his character to surpass realistic diminishing returns, distractions, and danger.
That, uh... that sounds pretty good.
comment by Vladimir_Golovin · 2008-12-19T08:07:56.000Z · LW(p) · GW(p)
EY: you'll find that it contains some tasks that are - there's no kind word for this - effortful. (I would even say "difficult", with the understanding that we're talking about something that takes 10 minutes, not 10 years.)
Some tasks in WoW can take months to complete, and it's clearly intended by WoW developers. Many tasks require 'raiding', which is an organized, coordinated activity involving up to 40 players, strategy, advance preparations, purchases, crafting etc. -- I have a friend who keeps a calendar of his evening raids and plans his real-world time in advance. When I played WoW, I didn't raid at all because it placed too much constraints on my real-world schedule.
EY: So in the future, we'll have programs that help you play the game
To a certain extent, we already do. Speaking of WoW again, we have Thottbot and Ludwig that help you instantly look up any item or spell (this function isn't build into WoW), talent/glyph calculators, forums where people calculate all these probabilities of critical strikes, and a huge number of addons -- for example Auctioneer, which lets people trade at the Auction House far more effectively (you see mean/median/average prices for a certain commodity, standard deviation, confidence ratings etc.), or Recount, which keeps stats about your fighting performance and displays them as graphs or pie charts.
Some of these 'helper' programs, however, are explicitly prohibited by the game vendors. Examples include WoW Glider, a bot that basically does the grindwork for you (it just repeatedly slays monsters to get you experience). Another example is from competitive FPS games -- there is a kind of helper programs, called aimbots, that take the task of weapon aiming off the player, he just does the running. FPS vendors ban aimbots, and players absolutely hate anyone who uses them.
EY: So this is the ultimate end of the prophecy of technological progress - just staring at a screen that says "YOU WIN", forever.
That definitely didn't happen in modern games -- the games that explored this area probably just didn't sell, otherwise we'd see a lot more games of this kind. On the other hand, we have wireheads, such as alcoholics or drug addicts, who (seem to) do precisely that. The addiction issue is complicated by the physiological component of addiction. Speaking cynically, it would be interesting to see how many addicts abandoned the habit if their drug of choice was not physiologically addicting.
Replies from: gwern↑ comment by gwern · 2009-10-19T15:20:54.916Z · LW(p) · GW(p)
Speaking cynically, it would be interesting to see how many addicts abandoned the habit if their drug of choice was not physiologically addicting.
From reading Ainslie's Breakdown of the Will: relatively few. Many addicts will go right back to their drug even after withdrawal is long over, and will even deviously work around their tools (like that one drug which makes any alcohol consumption induce vomiting).
comment by PK · 2008-12-19T08:08:00.000Z · LW(p) · GW(p)
MMO of the future lol(some swearing)
And just so I'm not completely off topic, I agree with the original post. There should be games, they should be fun and challenging and require effort and so on. AI's definetly should not do everything for us. A friendly future is a nice place to live in and not a place wher an AI does the living for us so we might as well just curl up in a fetal position and die.
comment by Vladimir_Golovin · 2008-12-19T08:20:40.000Z · LW(p) · GW(p)
D. Alex:this is a bit out of left field, but: Is a setting like World of Warcraft a good medium for development of AI? Clear goals, clear measures of progress, sufficient complexity to provide an indication of when important insights are achieved, and a safe environment (in the sense that the path to paperclip AI seems unlikely)...?
Ben Goertzel proposes a exactly that (but for different reasons): http://www.kurzweilai.net/meme/frame.html?main=/articles/art0710.html
comment by Vizikahn2 · 2008-12-19T08:44:15.000Z · LW(p) · GW(p)
How about making games that serve a purpose in the real world? Imagine a virtual world that generates and distributes quests and puzzles based on what kind of (robotic) work is needed in the real world. I guess this would go under "removing low-quality work to make way for high-quality work".
http://en.wikipedia.org/wiki/Game_with_a_purpose http://en.wikipedia.org/wiki/Human-based_computation
Replies from: Arandurcomment by Abigail · 2008-12-19T10:18:19.000Z · LW(p) · GW(p)
I watch kittens "playing", definitely building useful skills for the future. I enjoy effort on puzzles and games because each gives me a moment of pleasure on success, and no bad consequences of failure, but some games improve reactions or are otherwise useful.
The "Culture" sequence of novels by Iain M. Banks suggests how people might cope with machines doing all the work. One man works in a cafe, waiting on tables and cleaning up. Yes, the machines could do this work, but he gains happiness from the feeling of serving other people. Other people use machines which "surf" on lava flows, involving some danger, for excitement.
The problems around breeding would still provide interest, angst, challenge.
Artists could still work, though machines might be better able to communicate than humans.
comment by Vladimir_Golovin · 2008-12-19T11:58:48.000Z · LW(p) · GW(p)
EY: So this is the ultimate end of the prophecy of technological progress - just staring at a screen that says "YOU WIN", forever.
On second thought, playing a modern game IS staring at the "YOU WIN" screen.
Say, you just started playing a game. You did nothing at all, but you're already immortal, you look badass, you have fists the size of a boulder, and you can engage some mean-looking bad guys and win!
So, the actual product of the game industry is 4D "YOU WIN" screens.
Replies from: Kenny↑ comment by Kenny · 2013-07-27T18:41:43.134Z · LW(p) · GW(p)
I find the opposite to be true, at least for those games I've played recently. For example, Minecraft is so open-ended that I found myself expending more and more time and effort devising subgames and constructing various structures that I realized I was 'working' and further that I'd rather work on something more lastingly rewarding. Interestingly, I haven't found any more lastingly rewarding activity with which I've replaced it.
comment by jb5 · 2008-12-19T12:07:26.000Z · LW(p) · GW(p)
There's so much to consider here. For me at least, for something to be fun, I have to know that there's a challenge. For it to be a challenge, there has to be the possibility of failure. There has to be scary parts, that remind you of failure. There has to be multiple real, meaningful, obvious paths that suggest fun in the short term, but disaster in the long term, that you have to look at and reject. There have to be rewards that are enticing, but incredibly rare and difficult, and other rewards that are easier, faster and more localized.
But more importantly, to me, you have to believe it is real - you can't know you're sitting in a fun-generating machine (FGM) - you might wonder about it, but have no real ability to test it.
I can see the possibility of some super-advanced version of me thinking "I want to run a simulation where I live in the 20th and 21st centuries back on Earth", configuring the simulation in various ways, and then "jacking in" to the simulation. All of my memories are suppressed, and 38 years of apparent time later, I'm stuck here arguing on forums with you guys ;-)
Eventually I "die", and the simulation ends, super-advanced me "wakes up" and gets to look at my list of accomplishments and challenges, etc, and chuckle over the choices I've made, in my abysmal ignorance. Then, perhaps, he sits around every once in a while, wondering if his super-advanced life is just a simulation in some even-more-advanced entity's FGM.
It's turtles all the way down.
comment by luzr · 2008-12-19T12:26:24.000Z · LW(p) · GW(p)
Abigail:
"The "Culture" sequence of novels by Iain M. Banks suggests how people might cope with machines doing all the work."
Exactly, I think Culture is highly relevant to most topics discussed here. Obviously, it is just a fictional utopia, but I believe it gives plausible answer to "unlimited power future".
For the reference: http://en.wikipedia.org/wiki/The_Culture
comment by Hyphen · 2008-12-19T13:31:26.000Z · LW(p) · GW(p)
"Since you play the game to get gold and experience points, making the game easier will let you get more gold per unit time: the game will become more fun."
I know this statement is being set up to be knocked down but when I read it I recoiled in disgust. Many people already do not play games with the end goal of increasing the values in the computer that represent gold or experience, because that is a completely empty and pointless thing to care about. My goal when playing games is "to improve myself" and has been for several years. The most fun I have when playing a game is just after I improve my own skill enough to overcome some challenge in a game that I couldn't do before.
I mainly play competitive multiplayer games and extremely difficult singleplayer games. I refuse to play any multiplayer game with meaningful persistent character state because that inevitably makes the game revolve around grinding. Grinding is pointless and stupid and typically does not develop much skill.
"For me at least, for something to be fun, I have to know that there's a challenge. For it to be a challenge, there has to be the possibility of failure."
I go farther than this. For a game to be significantly fun for me I have to ACTUALLY FAIL. Repeatedly. Until I get good.
comment by Zubon · 2008-12-19T13:36:35.000Z · LW(p) · GW(p)
I find Greg Egan's Permutation City relevant here, particularly the character of Peer. He gives himself arbitrary desires over time, say collecting butterflies or making table legs with a lathe. Re-design the brain to enjoy some arbitrary, meaningless task for a finite time. It is one way of implementing jb's notion. At one point, he constructs a simulation that is perfectly circular: the experience is a closed loop that leads him back to his starting point and mental state. He realizes that he could maintain this one infinitely, or, with no loss, just run it once and permanently shut down.
Or as another character explains, how does it all feel? "However I want it to feel."
comment by nazgulnarsil3 · 2008-12-19T15:14:46.000Z · LW(p) · GW(p)
what's more fun? a holodeck that you have complete control over? or a holodeck with built in constraints?
playing god might be fun for awhile, but I think everyone would eventually switch over to programs with built in constraints to challenge themselves. the profession of highest prestige will probably people who write really really good holodeck programs.
comment by Marcello · 2008-12-19T16:32:44.000Z · LW(p) · GW(p)
"Though, since you never designed your own leg muscles, you are racing using strength that isn't yours. A race between robot cars is a purer contest of their designers."
Eliezer: While people don't design their muscles, they presently don't design their brains either, so a robot car-designing contest seems like just as impure a contest. Even if people did repeatedly redesign their brains, wouldn't this either result in convergence, in which case the contestants would be identical and the contest wouldn't be interesting, or alternatively, the arbitrary initial advantages and disadvantages would just be passed on in modified and perhaps even amplified form and the contest stays as impure as ever. Even if you try to measure the amount of effort the contestants put in, that's no good either because different people are born with unfairly different amounts of will-power.
So what on earth do you mean by "purer contest"?
Replies from: false_vacuum↑ comment by false_vacuum · 2011-02-05T02:55:46.924Z · LW(p) · GW(p)
This could function as an intuition pump for my own hypothesis that advanced self-modifying minds won't compete at arbitrary preselected goals for fun. (At least not exclusively for fun; they might design themselves to enjoy competing to achieve goals in a pedagogical framework, but actually achieving the goal [or acquiring the skills/knowledge/etc. necessary to achieve the goal] will be the point.)
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2008-12-19T16:45:23.000Z · LW(p) · GW(p)
I didn't say pure, I said purer. If you write your own legs, that's a purer contest than running on muscles you don't understand. If you rewrite your own mind, that's a purer contest than thinking with neural circuits you don't understand. You never made and can never make yourself from scratch, from nothing; but it's possible that the brain you wield could be Truly Part Of You. The implications of which are not exactly straightforward, but I haven't gotten to the section on introspection yet.
comment by Anonymous_Coward4 · 2008-12-19T16:56:51.000Z · LW(p) · GW(p)
What can I say, apart from "Progress Quest"
http://www.progressquest.com/ http://www.progressquest.com/info.php
Officially voted the Top Role Playing Game for Post-Singularity Sentient Beings.
Anonymous.
comment by Sean_C. · 2008-12-19T16:58:21.000Z · LW(p) · GW(p)
One common answer to the question "What will we do in the future when we've fixed all that is wrong with today" is "How the hell should I know?"
For example, imagine our neolithic ancestors asking each other the same question. "What will they do in the future when they don't have to worry about food, shelter, or even disease?" I think they could have imagined some things; "They'll make more complicated art." "They'll have more complicated sports."
But I don't they they would have imagined full time mathematicians, or video games, or TV, or even books. "Many many people will sit around and talk to each other while someone else gives them food and shelter in return." Unthinkable! Absurd!
So I would say that the technology of the future, insofar as it's impossible for us to imagine in detail, also makes the lifestyle of the future impossible to imagine in detail. Hence, the Singularity. Oh, and if you listen to those folks, they'll say that we'll merge with the AI's, so all of that 'automatic' engineering and whatnot will actually be performed by some sort of version of 'us'.
comment by steven · 2008-12-19T17:01:20.000Z · LW(p) · GW(p)
That one bothered me too. Perhaps you could say bodies are much more peripheral to people's identities than brains, so that in the running case what is being tested is meat that happens to be attached to you and in the robot case it's you yourself. On the other hand I'd still be me with some minor brain upgrades.
comment by Caledonian2 · 2008-12-19T17:07:57.000Z · LW(p) · GW(p)
Based on the comments here, it would seem that it's the people who reject ultimately-meaningless forms of play - that is, 'play' that doesn't develop skills useful to perpetuation - and concentrate on the "real world" who will end up existing.
And the Luddites will inherit the Earth...
comment by Caledonian2 · 2008-12-19T17:11:23.000Z · LW(p) · GW(p)
Or, to put it another way:
"Fixing" the future, in a way that renders human beings completely redundant and unnecessary even to themselves, isn't fixing anything. It's creating a problem of unlimited scope.
If that's the ultimate outcome of, say, producing superhuman minds - whether they're somehow enslaved to human preferences or not - then we're trying very hard to create a world in which the only rational treatment of humanity is extinction. Whether imposed from without or from within, voluntarily, is irrelevant.
comment by HermanTurnip · 2008-12-19T17:23:04.000Z · LW(p) · GW(p)
Of course, this could all be moot if we (re: the universe) are simply coded bits and bytes in a giant simulation program. Cue the Matrix...
comment by Philip_Goetz · 2008-12-19T17:41:02.000Z · LW(p) · GW(p)
What's most interesting to me is that lizards don't have fun.
Maybe they have fun. But if they do, I'm pretty sure worms don't have fun. A discussion like this one, carried on by lizards (or worms), wouldn't have included the concept "fun".
And if you keep going back in time or down in size, I'm sure you'll find organisms that don't experience pleasure.
Are there other types of possible experiences as qualitatively different and intrinsically good? Are there infinitely many of them? Is charting the course based on "fun theory" like lizards charting the course of the future based on "basking on a hot rock theory"?
Probably. And if the set of organisms that experience pleasure is a proper subset of the set of organisms that experience fun, then the answer is even more likely to be yes.
comment by Philip_Goetz · 2008-12-19T17:45:31.000Z · LW(p) · GW(p)
That should have said "as qualitatively different and intrinsically good as fun?"
"Heaven is being perfect.": Even a circle can't be perfect, in the classical sense of being the best possible circle. Is a circle of 2cm radius better than a circle of 1cm radius? It is much more nonsensical to talk of a person being perfect. It is even more nonsensical to talk of a still-evolving species being perfect.
comment by Philip_Goetz · 2008-12-19T18:30:27.000Z · LW(p) · GW(p)
I didn't mean "proper subset". I mean that if there are organisms that experience pleasure but not fun (or vice-versa), then it's more likely that there's an infinite number of possible "inherently good" noumena like pleasure, fun, and love; and that we've discovered only a small number of them.
comment by Sean_C. · 2008-12-19T19:28:46.000Z · LW(p) · GW(p)
And the mapping of attraction to those noumena is entirely subjective! Like in another Elizer Yudkowsky essay: http://lesswrong.com/lw/tn/the_true_prisoners_dilemma/
comment by Doug_S. · 2008-12-19T19:38:43.000Z · LW(p) · GW(p)
I'd say that "work" generally consists of activities that are only useful as a means, and not as an end. In general, "work" is anything that you'd rather have someone (or something) else do for you.
For example, you don't like cleaning the toilet, but you want the toilet to be clean, so you clean it anyway. Cleaning the toilet is work.
Anything you wouldn't volunteer to do if you weren't getting paid is work.
So, yeah, let's get rid of work!
comment by Roland2 · 2008-12-19T19:45:10.000Z · LW(p) · GW(p)
@ac:
Unlike Roland, who is obviously a puritan, I rather enjoy the occasional spot of idleness. For a non-trivial number of people, playing WoW for a couple of hours a day is more fun that playing real life. Rather than make thinly veiled moral judgements about folks for their unproductivity, perhaps he should consider what makes certain games so engaging.
I enjoyed playing games myself, so I know what you are talking about.
You mention idleness, which I agree is sometimes worthwhile. This is the package deal fallacy since there are other ways to achieve that, hanging out with your friends being one of them. Also I'm not making a moral judgement here and the point is not loss of productivity but foregoing much more rewarding things like:
-increasing your social circle(online friends don't count). -enjoying some hobby with real life benefits. -having quality time with your significant other. -if you don't have a significant other invest some time trying to find one. -improve your health by doing exercise, or eating better. -etc...
And if you don't have any ideas then instead of solving artificial game riddles think about what you could do to improve your life.
Replies from: taryneast↑ comment by taryneast · 2011-06-02T10:21:45.284Z · LW(p) · GW(p)
-increasing your social circle(online friends don't count).
I defy this conclusion.
online friends are maybe not as socially fulfilling as face-to-face friends, but I wouldn't count them as having zero positive benefits. Especially if you have an obscure hobby/interest only shared by some - or only by a few local people. Doubly-so if you happen not to live in a big city.
I have both online and offline friends and both sets of them make up my complete social circle.
If you also happen to travel frequently - offline contact with your old circle of friends/family may be the best way to keep in touch. As a member of a family who I can truthfully say the sun-never sets upon... this is a huge benefit.
comment by nazgulnarsil3 · 2008-12-19T20:50:28.000Z · LW(p) · GW(p)
caledonian: I agree. if we develop some sort of virtual reality that can provide any desire, we'll just be selecting for people who don't go in and never come out. If so the future will be populated by people who refuse such self gratification.
comment by Anonymous48 · 2008-12-20T00:21:24.000Z · LW(p) · GW(p)
On the subject of MMORPGs, I've enjoyed playing one for about a year and then it stopped being fun. In the beginning the interesting part was the world exploration, acts of learning new things, rules and interactions between various parts of the system, feeling of steady advance towards some clearly defined goal and work that was guaranteed to pay off. After a while grinding has started to become annoying and my interest shifted towards minmaxing everything and writing complex scripts to allow bots do the boring parts. Then realisation hit me. In real world an extraordinarily efficient way of doing things is good, it's called an invention. In a game it is called cheating. Nature doesn't care what smart tricks you used to achieve your goals. In a game if it wasn't anticipated by developers it probably counts as an exploit. The universe has a set of unchanging rules, a game is perpetually balanced by series of patches and crutches in unpredictable places. By being creative you are fighting against game developers, which is pointless because they will actively oppose and get rid of you through their control over the sandbox. You aren't expected to have hacker (in the original meaning of the word) kind of fun in such games.
Replies from: DSimon↑ comment by DSimon · 2011-09-06T06:11:10.457Z · LW(p) · GW(p)
You aren't expected to have hacker (in the original meaning of the word) kind of fun in such games.
You can be expected to have that kind of fun, but only in the better sorts of games. I'm thinking particularly of games like Minecraft where, after mastering the game's own systems, the player's next activity is "write some neat game systems of your own and share them around so other people can expand their experiences".
comment by Doug_S. · 2008-12-20T08:51:51.000Z · LW(p) · GW(p)
Then realisation hit me. In real world an extraordinarily efficient way of doing things is good, it's called an invention. In a game it is called cheating. Nature doesn't care what smart tricks you used to achieve your goals. In a game if it wasn't anticipated by developers it probably counts as an exploit. The universe has a set of unchanging rules, a game is perpetually balanced by series of patches and crutches in unpredictable places. By being creative you are fighting against game developers, which is pointless because they will actively oppose and get rid of you through their control over the sandbox. You aren't expected to have hacker (in the original meaning of the word) kind of fun in such games.
I think you need to go play some Magic: The Gathering. The kind of exploration you describe is actively encouraged by the game design - and you get the equivalent of a significant rules change roughly three and a half times a year, with the release of every new card set. The developers definitely do NOT think of everything, and they expect us to surprise them.
comment by Kaj_Sotala · 2008-12-20T15:01:37.000Z · LW(p) · GW(p)
EY: I'm not going to touch that one. I'll stick with the much simpler answer of "I wouldn't actually prefer to be a passive experiencer." If I wanted Nirvana, I might try to figure out how to achieve that impossibility. But once you strip away Buddha telling me that Nirvana is the end-all of existence, Nirvana seems rather more like "sounds like good news in the moment of first being told" or "ideological belief in desire" rather than, y'know, something I'd actually want.
Actually, to me Nirvana - or wireheading - sound like the opposite of the "sounds like good news at first" case. Most people, when being told about a scenario where they were turned into orgasmium, would instantly reject it and decide they wouldn't want it. But if they were made into orgasmium, even for a short while, then they wouldn't want it to ever stop...
comment by Aleksei_Riikonen · 2008-12-20T16:24:40.000Z · LW(p) · GW(p)
I don't understand what's the problem Eliezer has with advanced computer games. Why not "waste effort" if it's fun, and all the important work in the real world is anyway getting done?
No, the end of the road for the MMORPGer wouldn't be staring at an "YOU WIN" screen. That's not fun, I want to instead go waste some effort in e.g. advanced ancestor simulations, possibly with my memories of "the future" temporarily erased.
comment by Emile · 2008-12-20T17:19:55.000Z · LW(p) · GW(p)
Kaj: But if they were made into orgasmium, even for a short while, then they wouldn't want it to ever stop...
Well, yeah, but most people, if they had their brain removed and replaced by a slab of jelly, wouldn't raise any objections either.
I don't think wireheading is that different. A wireheaded brain is basically a hacked brain that doesn't work any more.
(you could also replace "made into orgasmium" by "given heroin")
comment by Kaj_Sotala · 2008-12-20T23:46:32.000Z · LW(p) · GW(p)
Emile: A wireheaded brain is basically a hacked brain that doesn't work any more.
Sure, if you want to define it that way... but I'd be wary of that kind of thinking. Doesn't work for what anymore? It certainly still works for experiencing phenomenal sensations. Doesn't work for the evolutionary purpose of maximizing fitness? Well, we should try to get past that anyhow. Doesn't work for self-preservation? That's just a poorly implemented wirehead. If one were, say, running as code, you could relatively easily have a subprogram that monitored the environment and brought the wirehead out of the Nirvana state for as long as needed to defend itself.
(For the record, I wouldn't want to be wirehead, nor would I particularly wish that anybody else became one, but this is a position I find immensly difficult to justify on intellectual grounds. Permutation City -style hardwired bliss out of certain tasks might be okay, though.)
comment by Z._M._Davis · 2008-12-21T16:56:07.000Z · LW(p) · GW(p)
"I sometimes think that futuristic ideals phrased in terms of 'getting rid of work' would be better reformulated as 'removing low-quality work to make way for high-quality work'."
Alternatively, you could taboo work and play entirely, speaking instead of various forms of activity, and their various costs and benefits.
comment by HopeFox · 2011-04-30T15:46:11.899Z · LW(p) · GW(p)
Do we even need the destination? When you consider "fun" as something that comes from a process, from the journey of approaching a goal, then wouldn't it make sense to disentangle the journey and the goal? We shouldn't need the destination in order to make the journey worthwhile. I mean, if the goal were actually important, then surely we'd just get our AI buddies to implement the goal, while I was off doing fun journey stuff.
For a more concrete example:
I like baking fruitcakes. (Something I don't do nearly often enough these days.) Mixing the raw ingredients is fun, and licking the bowl clean afterwards is always good times.
I also like eating fruitcake. Fruitcake is tasty.
Now, one of the things that induces me to bake a fruitcake rather than, say, play Baldur's Gate II is that, afterwards, there will be fruitcake. However, there have been times when other people (usually my mother) have been baking a fruitcake, and I have enthusiastically joined in the process, even though I know that she's better at it than I am, and even if I don't participate, there will still be fruitcake at the end of the day. So clearly I place some value on the process independently of the result.
I suspect, in fact, that actually getting the fruitcake at the end of the baking process is unnecessary to my enjoyment of the process. Maybe I'd be just as happy if we swapped the cake to another family for a cheesecake they'd just made. Maybe the need to be "rewarded" for participating in a process that was rewarding in itself, is just a cognitive bias that I can overcome. After all, if I really wanted a fruitcake, I could buy one, or just let my mother do the baking. The more I look at this, the more the fruitcake itself seems like fake justification for the baking process.
Now consider this situation in a world where optimal fruitcakes are constructed by nanomachines on demand. I should still be able to enjoy baking, even though the final product of the process is of trivial value. If I can separate the process from the goal - if, in fact, I can stop thinking of the baking process as a "journey" and instead just call it a goal in itself - a 4D goal - then I think that would be a substantial step towards being able to find fun in a post-work, post-scarcity world.
Damnit, now I want fruitcake.
Replies from: taryneast↑ comment by taryneast · 2011-06-02T10:47:19.062Z · LW(p) · GW(p)
I totally agree... there are heaps of processes that I enjoy far more than the actual end-result.
Crochet is my example.
I'm quite happy to continue crocheting something pretty (it has to be pretty - I don't enjoy crocheting abominations) for a long time and never "owning a crocheted thing" at the end.
Before I hit upon the solution, I spent a long time starting projects - some of which I finished, but lots I didn't... because I didn't care about finishing - just about doing. Of course, couple this with an aversion to destroying something I've already made (which might have solved the problem by turning it into a sisyphean task). and I got a lot of "why don't you ever finish anything?" from my mother.
The question usually comes as "why don't you ever finish anything, don't you want the [crocheted thing] you set out to create?" - and the honest answer is "no".... but if you say that - they ask "well why did you start making it in the first place?"
Most people don't seem to understand enjoying the process - at least not on a gut level...
I actually solved this particular dilemma by giving away my crocheted things to my grandma - who likes owning crocheted doilies et al. Works for embroidery projects too.
Unfortunately, I still tend to get lack of understanding from other people: "but why don't you ever make something for yourself?" I find it very hard to explain to goal-oriented people why I don't like crochet... I like crocheting.
I would definitely consider myself to be more process-oriented than goal-oriented. I like doing stuff... I like crocheting, not the goal of having crocheted something in particular. Especially, I like learning - not the feat of "having learned something".
So for me - it's very difficult to go to those "attain your goals" seminars etc - because I don't have set goals. I can't point at something and say I want to have achieved precisely that thing, because for me, the thing itself doesn't matter.
It can be frustrating, because I certainly do want to improve over time. I crochet better and more complicated things, I study more challenging topics that build on past learning that I enjoyed. but I can't necessarily quantify that I want to "learn X".
Because there is no X... or at least no specific X.
and then people tell me I'm drifting and that I'll "never accomplish anything"... but accomplishing specific things for me isn't the point. I enjoy the act, not necessarily the achievement.
Of course, over time, I do accomplish things - because if you continue to, say, crochet over a long period of time, eventually you will have piled up a very large back catalogue of doilies... and the same goes for learning of whatever other process you enjoy. Which I can then, of course, show to my mother...
who then invariably says "but why don't you finish the ones that are still in your cupboard?"
sigh
comment by pnrjulius · 2012-06-06T20:06:32.780Z · LW(p) · GW(p)
I just want to say that your point about valuing actual people and not (potentially illusory) experiences of people is a very important one, and one I wish I could explain better to people who think that maximizing stimulation to the opiate system is the final word on happiness.
Though I've never thought of myself as a Singularitarian, one thing I do look forward to that could be called a "singularity" is the point at which all the problems where winning is what counts have been won---no more war, no more cancer, no more poverty. Amazingly, we are about 80% there, and have risen from roughly 20% in the last few hundred years.
comment by Arandur · 2012-11-09T18:36:58.210Z · LW(p) · GW(p)
I'm not sure that the difference between 4D states and 3D states is meaningful, with respect to eudaimoniac valuations. Doesn't this overlook the fact that human memories are encoded physically, and are therefore part of the 3D state being looked at? I don't see any meaningful difference between a valuation over a 4D state, and a valuation over a 3D state including memories of the past.
In other words, I can think of no 3D state whose eudaimoniac valuation is worse than that of the 4D state having it as its endpoint.
(In fact, I can think of quite a few which may in fact be better, for pathological choices of 4D state, e.g. ones extending all the way back to the Dark Ages or before.)
P.S. Is there a standardized spelling for the term which I have chosen to spell as "eudaimoniac"? A quick Google search suggested this one as the best candidate.
comment by A1987dM (army1987) · 2013-10-24T10:16:38.636Z · LW(p) · GW(p)
realistic nonsentient
But but anti-zombie principle!
comment by NoriMori1992 · 2022-03-01T08:38:07.477Z · LW(p) · GW(p)
Though, since you never designed your own leg muscles, you are racing using strength that isn't yours. A race between robot cars is a purer contest of their designers.
How do you figure? You didn't design your brain, either, so using your intellect to design a robot car is also using strength that isn't yours.