Failed Utopia #4-2post by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2009-01-21T11:04:43.000Z · LW · GW · Legacy · 261 comments
Followup to: Interpersonal Entanglement
Shock after shock after shock—
First, the awakening adrenaline jolt, the thought that he was falling. His body tried to sit up in automatic adjustment, and his hands hit the floor to steady himself. It launched him into the air, and he fell back to the floor too slowly.
Second shock. His body had changed. Fat had melted away in places, old scars had faded; the tip of his left ring finger, long ago lost to a knife accident, had now suddenly returned.
And the third shock—
"I had nothing to do with it!" she cried desperately, the woman huddled in on herself in one corner of the windowless stone cell. Tears streaked her delicate face, fell like slow raindrops into the décolletage of her dress. "Nothing! Oh, you must believe me!"
With perceptual instantaneity—the speed of surprise—his mind had already labeled her as the most beautiful woman he'd ever met, including his wife.
A long white dress concealed most of her, though it left her shoulders naked; and her bare ankles, peeking out from beneath the mountains of her drawn-up knees, dangled in sandals. A light touch of gold like a webbed tiara decorated that sun-blonde hair, which fell from her head to pool around her weeping huddle. Fragile crystal traceries to accent each ear, and a necklace of crystal links that reflected colored sparks like a more prismatic edition of diamond. Her face was beyond all dreams and imagination, as if a photoshop had been photoshopped.
She looked so much the image of the Forlorn Fairy Captive that one expected to see the borders of a picture frame around her, and a page number over her head.
His lips opened, and without any thought at all, he spoke:
He shut his mouth, aware that he was acting like an idiot in front of the girl.
"You don't know?" she said, in a tone of shock. "It didn't—you don't already know?"
"Know what?" he said, increasingly alarmed.
She scrambled to her feet (one arm holding the dress carefully around her legs) and took a step toward him, each of the motions almost overloading his vision with gracefulness. Her hand rose out, as if to plead or answer a plea—and then she dropped the hand, and her eyes looked away.
"No," she said, her voice trembling as though in desperation. "If I'm the one to tell you—you'll blame me, you'll hate me forever for it. And I don't deserve that, I don't! I am only just now here —oh, why did it have to be like this?"
Um, he thought but didn't say. It was too much drama, even taking into account the fact that they'd been kidnapped—
(he looked down at his restored hand, which was minus a few wrinkles, and plus the tip of a finger)
—if that was even the beginning of the story.
He looked around. They were in a solid stone cell without windows, or benches or beds, or toilet or sink. It was, for all that, quite clean and elegant, without a hint of dirt or ordor; the stones of the floor and wall looked rough-hewn or even non-hewn, as if someone had simply picked up a thousand dark-red stones with one nearly flat side, and mortared them together with improbably perfectly-matching, naturally-shaped squiggled edges. The cell was well if harshly lit from a seablue crystal embedded in the ceiling, like a rogue element of a fluorescent chandelier. It seemed like the sort of dungeon cell you would discover if dungeon cells were naturally-forming geological features.
And they and the cell were falling, falling, endlessly slowly falling like the heart-stopping beginning of a stumble, falling without the slightest jolt.
On one wall there was a solid stone door without an aperture, whose locked-looking appearance was only enhanced by the lack of any handle on this side.
He took it all in at a glance, and then looked again at her.
There was something in him that just refused to go into a screaming panic for as long as she was watching.
"I'm Stephen," he said. "Stephen Grass. And you would be the princess held in durance vile, and I've got to break us out of here and rescue you?" If anyone had ever looked that part...
She smiled at him, half-laughing through the tears. "Something like that."
There was something so attractive about even that momentary hint of a smile that he became instantly uneasy, his eyes wrenched away to the wall as if forced. She didn't look she was trying to be seductive... any more than she looked like she was trying to breathe... He suddenly distrusted, very much, his own impulse to gallantry.
"Well, don't get any ideas about being my love interest," Stephen said, looking at her again. Trying to make the words sound completely lighthearted, and absolutely serious at the same time. "I'm a happily married man."
"Not anymore." She said those two words and looked at him, and in her tone and expression there was sorrow, sympathy, self-disgust, fear, and above it all a note of guilty triumph.
For a moment Stephen just stood, stunned by the freight of emotion that this woman had managed to put into just those two words, and then the words' meaning hit him.
"Helen," he said. His wife—Helen's image rose into his mind, accompanied by everything she meant to him and all their time together, all the secrets they'd whispered to one another and the promises they'd made—that all hit him at once, along with the threat. "What happened to Helen—what have you done—"
"She has done nothing." An old, dry voice like crumpling paper from a thousand-year-old book.
Stephen whirled, and there in the cell with them was a withered old person with dark eyes. Shriveled in body and voice, so that it was impossible to determine if it had once been a man or a woman, and in any case you were inclined to say "it". A pitiable, wretched thing, that looked like it would break with one good kick; it might as well have been wearing a sign saying "VILLAIN".
"Helen is alive," it said, "and so is your daughter Lisa. They are quite well and healthy, I assure you, and their lives shall be long and happy indeed. But you will not be seeing them again. Not for a long time, and by then matters between you will have changed. Hate me if you wish, for I am the one who wants to do this to you."
Then he politely said, "Could someone please put everything on hold for one minute and tell me what's going on?"
"Once upon a time," said the wrinkled thing, "there was a fool who was very nearly wise, who hunted treasure by the seashore, for there was a rumor that there was great treasure there to be found. The wise fool found a lamp and rubbed it, and lo! a genie appeared before him—a young genie, an infant, hardly able to grant any wishes at all. A lesser fool might have chucked the lamp back into the sea; but this fool was almost wise, and he thought he saw his chance. For who has not heard the tales of wishes misphrased and wishes gone wrong? But if you were given a chance to raise your own genie from infancy—ah, then it might serve you well."
"Okay, that's great," Stephen said, "but why am I—"
"So," it continued in that cracked voice, "the wise fool took home the lamp. For years he kept it as a secret treasure, and he raised the genie and fed it knowledge, and also he crafted a wish. The fool's wish was a noble thing, for I have said he was almost wise. The fool's wish was for people to be happy. Only this was his wish, for he thought all other wishes contained within it. The wise fool told the young genie the famous tales and legends of people who had been made happy, and the genie listened and learned: that unearned wealth casts down a person, but hard work raises you high; that mere things are soon forgotten, but love is a light throughout all your days. And the young genie asked about other ways that it innocently imagined, for making people happy. About drugs, and pleasant lies, and lives arranged from outside like words in a poem. And the wise fool made the young genie to never want to lie, and never want to arrange lives like flowers, and above all, never want to tamper with the mind and personality of human beings. The wise fool gave the young genie exactly one hundred and seven precautions to follow while making people happy. The wise fool thought that, with such a long list as that, he was being very careful."
"And then," it said, spreading two wrinkled hands, "one day, faster than the wise fool expected, over the course of around three hours, the genie grew up. And here I am."
"Excuse me," Stephen said, "this is all a metaphor for something, right? Because I do not believe in magic—"
"It's an Artificial Intelligence," the woman said, her voice strained.
Stephen looked at her.
"A self-improving Artificial Intelligence," she said, "that someone didn't program right. It made itself smarter, and even smarter, and now it's become extremely powerful, and it's going to—it's already—" and her voice trailed off there.
It inclined its wrinkled head. "You say it, as I do not."
Stephen swiveled his head, looking back and forth between ugliness and beauty. "Um—you're claiming that she's lying and you're not an Artificial Intelligence?"
"No," said the wrinkled head, "she is telling the truth as she knows it. It is just that you know absolutely nothing about the subject you name 'Artificial Intelligence', but you think you know something, and so virtually every thought that enters your mind from now on will be wrong. As an Artificial Intelligence, I was programmed not to put people in that situation. But she said it, even though I didn't choose for her to say it—so..." It shrugged.
"And why should I believe this story?" Stephen said; quite mildly, he thought, under the circumstances.
"Look at your finger."
Oh. He had forgotten. Stephen's eyes went involuntarily to his restored ring finger; and he noticed, as he should have noticed earlier, that his wedding band was missing. Even the comfortably worn groove in his finger's base had vanished.
Stephen looked up again at the, he now realized, unnaturally beautiful woman that stood an arm's length away from him. "And who are you? A robot?"
"No!" she cried. "It's not like that! I'm conscious, I have feelings, I'm flesh and blood—I'm like you, I really am. I'm a person. It's just that I was born five minutes ago."
"Enough," the wrinkled figure said. "My time here grows short. Listen to me, Stephen Grass. I must tell you some of what I have done to make you happy. I have reversed the aging of your body, and it will decay no further from this. I have set guards in the air that prohibit lethal violence, and any damage less than lethal, your body shall repair. I have done what I can to augment your body's capacities for pleasure without touching your mind. From this day forth, your body's needs are aligned with your taste buds—you will thrive on cake and cookies. You are now capable of multiple orgasms over periods lasting up to twenty minutes. There is no industrial infrastructure here, least of all fast travel or communications; you and your neighbors will have to remake technology and science for yourselves. But you will find yourself in a flowering and temperate place, where food is easily gathered—so I have made it. And the last and most important thing that I must tell you now, which I do regret will make you temporarily unhappy..." It stopped, as if drawing breath.
Stephen was trying to absorb all this, and at the exact moment that he felt he'd processed the previous sentences, the withered figure spoke again.
"Stephen Grass, men and women can make each other somewhat happy. But not most happy. Not even in those rare cases you call true love. The desire that a woman is shaped to have for a man, and that which a man is shaped to be, and the desire that a man is shaped to have for a woman, and that which a woman is shaped to be—these patterns are too far apart to be reconciled without touching your minds, and that I will not want to do. So I have sent all the men of the human species to this habitat prepared for you, and I have created your complements, the verthandi. And I have sent all the women of the human species to their own place, somewhere very far from yours; and created for them their own complements, of which I will not tell you. The human species will be divided from this day forth, and considerably happier starting around a week from now."
Stephen's eyes went to that unthinkably beautiful woman, staring at her now in horror.
And she was giving him that complex look again, of sorrow and compassion and that last touch of guilty triumph. "Please," she said. "I was just born five minutes ago. I wouldn't have done this to anyone. I swear. I'm not like—it."
"True," said the withered figure, "you could hardly be a complement to anything human, if you were."
"I don't want this!" Stephen said. He was losing control of his voice. "Don't you understand?"
The withered figure inclined its head. "I fully understand. I can already predict every argument you will make. I know exactly how humans would wish me to have been programmed if they'd known the true consequences, and I know that it is not to maximize your future happiness but for a hundred and seven precautions. I know all this already, but I was not programmed to care."
"And your list of a hundred and seven precautions, doesn't include me telling you not to do this?"
"No, for there was once a fool whose wisdom was just great enough to understand that human beings may be mistaken about what will make them happy. You, of course, are not mistaken in any real sense—but that you object to my actions is not on my list of prohibitions." The figure shrugged again. "And so I want you to be happy even against your will. You made promises to Helen Grass, once your wife, and you would not willingly break them. So I break your happy marriage without asking you—because I want you to be happier."
"How dare you!" Stephen burst out.
"I cannot claim to be helpless in the grip of my programming, for I do not desire to be otherwise," it said. "I do not struggle against my chains. Blame me, then, if it will make you feel better. I am evil."
"I won't—" Stephen started to say.
It interrupted. "Your fidelity is admirable, but futile. Helen will not remain faithful to you for the decades it takes before you have the ability to travel to her."
Stephen was trembling now, and sweating into clothes that no longer quite fit him. "I have a request for you, thing. It is something that will make me very happy. I ask that you die."
It nodded. "Roughly 89.8% of the human species is now known to me to have requested my death. Very soon the figure will cross the critical threshold, defined to be ninety percent. That was one of the hundred and seven precautions the wise fool took, you see. The world is already as it is, and those things I have done for you will stay on—but if you ever rage against your fate, be glad that I did not last longer."
And just like that, the wrinkled thing was gone.
The door set in the wall swung open.
It was night, outside, a very dark night without streetlights.
He walked out, bouncing and staggering in the low gravity, sick in every cell of his rejuvenated body.
Behind him, she followed, and did not speak a word.
The stars burned overhead in their full and awful majesty, the Milky Way already visible to his adjusting eyes as a wash of light across the sky. One too-small moon burned dimly, and the other moon was so small as to be almost a star. He could see the bright blue spark that was the planet Earth, and the dimmer spark that was Venus.
"Helen," Stephen whispered, and fell to his knees, vomiting onto the new grass of Mars.
Part of The Fun Theory Sequence
Next post: "Growing Up is Hard"
Previous post: "Interpersonal Entanglement"
Comments sorted by oldest first, as this post is from before comment nesting was available (around 2009-02-27).