Chapter 48: Utilitarian Priorities
post by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2015-03-14T19:00:22.695Z · LW · GW · 1 commentsContents
1 comment
It was Saturday, the first morning of February, and at the Ravenclaw table, a boy bearing a breakfast plate heaped high with vegetables was nervously inspecting his servings for the slightest trace of meat.
It might have been an overreaction. After he'd gotten over the raw shock, Harry's common sense had woken up and hypothesized that "Parseltongue" was probably just a linguistic user interface for controlling snakes...
...after all, snakes couldn't really be human-level intelligent, someone would have noticed by now. The smallest-brained creatures Harry had ever heard of with anything like linguistic ability were the African grey parrots taught by Irene Pepperberg. And that was unstructured protolanguage, in a species that played complex games of adultery and needed to model other parrots. While according to what Draco had been able to remember, snakes spoke to Parselmouths in what sounded like normal human language - i.e., full-blown recursive syntactical grammar. That had taken time for hominids to evolve, with huge brains and strong social selection pressures. Snakes didn't have much society at all that Harry had ever heard. And with thousands upon thousands of different species of snakes all over the world, how could they all use the same version of their supposed language, "Parseltongue"?
Of course that was all merely common sense, in which Harry was starting to lose faith entirely.
But Harry was sure he'd heard snakes hissing on the TV at some point - after all, he knew what that sounded like from somewhere - and that hadn't sounded to him like language, which had seemed a good deal more reassuring...
...at first. The problem was that Draco had also asserted that Parselmouths could send snakes on extended complex missions. And if that was true, then Parselmouths had to make snakes persistently intelligent by talking to them. In the worst-case scenario that would make the snake self-aware, like what Harry had accidentally done to the Sorting Hat.
And when Harry had offered that hypothesis, Draco had claimed that he could remember a story - Harry hoped to Cthulhu that this one story was just a fairy tale, it had that ring to it, but there was a story - about Salazar Slytherin sending a brave young viper on a mission to gather information from other snakes.
If any snake a Parselmouth had talked to, could make other snakes self-aware by talking to them, then...
Then...
Harry didn't even know why his mind was going all "then... then..." when he knew perfectly well how the exponential progression would work, it was just the sheer moral horror of it that was blowing his mind.
And what if someone had invented a spell like that to talk to cows?
What if there were Poultrymouths?
Or for that matter...
Harry froze in sudden realization just as the forkful of carrots was about to enter his mouth.
That couldn't, couldn't possibly be true, surely no wizard would be stupid enough to do THAT...
And Harry knew, with a dreadful sinking feeling, that of course they would be that stupid. Salazar Slytherin had probably never considered the moral implications of snake intelligence for even one second, just like it hadn't ever occurred to Salazar that Muggleborns were intelligent enough to deserve personhood rights. Most people just didn't see moral issues at all unless someone else was pointing them out...
"Harry?" said Terry from beside him, sounding like he was afraid he would regret asking. "Why are you staring at your fork like that?"
"I'm starting to think magic should be illegal," said Harry. "By the way, have you ever heard any stories about wizards who could speak with plants?"
Terry hadn't heard of anything like that.
Neither had any seventh-year Ravenclaws that Harry had asked.
And now Harry had returned to his place, but not yet sat down again, staring at his plate of vegetables with a forlorn expression. He was getting hungrier, and later in the day he would be visiting Mary's Place for one of their incredibly tasty dishes... Harry was finding himself sorely tempted to just revert back to yesterday's eating habits and be done with it.
You've got to eat something, said his inner Slytherin. And it's not all that much more likely that anyone sneezed self-awareness onto poultry than onto plants, so as long as you're eating food of questionable sentience either way, why not eat the delicious deep-fried Diracawl slices?
I'm not quite sure that's valid utilitarian logic, there -
Oh, you want utilitarian logic? One serving of utilitarian logic coming up: Even in the unlikely chance that some moron did manage to confer sentience on chickens, it's your research that stands the best chance of discovering the fact and doing something about it. If you can complete your work even slightly faster by not messing around with your diet, then, counterintuitive as it may seem, the best thing you can do to save the greatest number of possibly-sentient who-knows-whats is not wasting time on wild guesses about what might be intelligent. It's not like the house elves haven't prepared the food already, regardless of what you take onto your plate.
Harry considered this for a moment. It was a rather seductive line of reasoning -
Good! said Slytherin. I'm glad you see now that the most moral thing to do is to sacrifice the lives of sentient beings for your own convenience, to feed your dreadful appetites, for the sick pleasure of ripping them apart with your teeth -
What? Harry thought indignantly. Which side are you on here?
His inner Slytherin's mental voice was grim. You too will someday embrace the doctrine... that the end justifies the meats. This was followed by some mental snickering.
Ever since Harry had started worrying that plants might also be sentient, his non-Ravenclaw components had been having trouble taking his moral caution seriously. Hufflepuff was shouting Cannibalism! every time Harry tried to think about any food item whatsoever, and Gryffindor would visualize it screaming while he ate it, even if it was, say, a sandwich -
Cannibalism!
AIIIEEEE DON'T EAT ME -
Ignore the screams, eat it anyway! It's a safe place to compromise your ethics in the service of higher goals, everyone else thinks it's okay to eat sandwiches so you can't use your usual rationalization about a small probability of a large downside if you get caught -
Harry gave a mental sigh, and thought, Just so long as you're okay with us being eaten by giant monsters that didn't do enough research into whether we were sentient.
I'm okay with that, said Slytherin. Is everyone else okay with that? (Internal mental nods.) Great, can we go back to deep-fried Diracawl slices now?
Not until I've done some more research into what's sentient and what isn't. Now shut up. And Harry turned firmly away from his plate full of oh-so-tempting vegetables to head toward the library -
Just eat the students, said Hufflepuff. There's no doubt about whether they're sentient.
You know you want to, said Gryffindor. I bet the young ones are the tastiest.
Harry was starting to wonder if the Dementor had somehow damaged their imaginary personalities.
"Honestly," said Hermione. The young girl's voice sounded a little acerbic as her gaze scanned the bookshelves of the Herbology stacks in the Hogwarts library. Harry had left her a message asking if she could come to the library after she'd finished breakfast, which Harry had skipped; but then when Harry had introduced the day's topic she'd seemed a bit nonplussed. "You know your problem, Harry? You've got no sense of priorities. An idea gets into your head and you just go running straight off after it."
"I've got a great sense of priorities," said Harry. His hand reached out and grabbed Vegetable Cunning by Casey McNamara, and began to flip through the starting pages, searching for the table of contents. "That's why I want to find out whether plants can talk before I eat my carrots."
"Don't you think that maybe the two of us have more important things to worry about?"
You sound just like Draco, Harry thought, but of course didn't say out loud. Out loud he said, "What could possibly be more important than plants turning out to be sentient?"
There was a pregnant silence from beside him, as Harry's eyes went down the table of contents. There was indeed a chapter on Plant Language, causing Harry's heart to skip a beat; and then his hands began to rapidly turn the pages, heading for the appropriate page number.
"There are days," said Hermione Granger, "when I really, truly, have absolutely no idea what goes on inside that head of yours."
"Look, it's a question of multiplication, okay? There's a lot of plants in the world, if they're not sentient then they're not important, but if plants are people then they've got more moral weight than all the human beings in the world put together. Now, of course your brain doesn't realize that on an intuitive level, but that's because the brain can't multiply. Like if you ask three separate groups of Canadian households how much they'll pay to save two thousand, twenty thousand, or two hundred thousand birds from dying in oil ponds, the three groups will respectively state that they're willing to pay seventy-eight, eighty-eight, and eighty dollars. No difference, in other words. It's called scope insensitivity. Your brain imagines a single bird struggling in an oil pond, and that image creates some amount of emotion that determines your willingness to pay. But no one can visualize even two thousand of anything, so the quantity just gets thrown straight out the window. Now try to correct that bias with respect to a hundred trillion sentient blades of grass, and you'll realize that this could be thousands of times more important than we used to think the whole human species was... oh thank Azathoth, this says it's just mandrakes that can talk and they speak regular human language out loud, not that there's a spell you can use to talk with any plant -"
"Ron came to me at breakfast yesterday morning," Hermione said. Now her voice sounded a little quiet, a little sad, maybe even a little scared. "He said he'd been dreadfully shocked to see me kiss you. That what you said while you were Demented should've shown me how much evil you were hiding inside. And that if I was going to be a follower of a Dark Wizard, then he wasn't sure he wanted to be in my army anymore."
Harry's hands had stopped turning pages. It seemed that Harry's brain, for all its abstract knowledge, was still incapable of appreciating scope on any real emotional level, because it had just forcibly redirected his attention away from trillions of possibly-sentient blades of grass who might be suffering or dying even as they spoke, and toward the life of a single human being who happened to be nearer and dearer.
"Ron is the world's most gigantic prat," Harry said. "They won't be printing that in the newspaper anytime soon, because it's not news. So after you fired him, how many of his arms and legs did you break?"
"I tried to tell him it wasn't like that," Hermione went on in the same quiet voice. "I tried to tell him you weren't like that, and that it wasn't like that between the two of us, but it just seemed to make him even more... more like he was."
"Well, yes," Harry said. He was surprised that he wasn't feeling angrier at Captain Weasley, but his concern for Hermione seemed to be overriding that, for now. "The more you try to justify yourself to people like that, the more it acknowledges that they have the right to question you. It shows you think they get to be your inquisitor, and once you grant someone that sort of power over you, they just push more and more." This was one of Draco Malfoy's lessons which Harry had thought was actually pretty smart: people who tried to defend themselves got questioned over every little point and could never satisfy their interrogators; but if you made it clear from the start that you were a celebrity and above social conventions, people's minds wouldn't bother tracking most violations. "That's why when Ron came over to me as I was sitting down at the Ravenclaw table, and told me to stay away from you, I held my hand out over the floor and said, 'You see how high I'm holding my hand? Your intelligence has to be at least this high to talk to me.' Then he accused me of, quote, sucking you into the darkness, unquote, so I pursed my lips and went schluuuuurp, and after that his mouth was still making those talking noises so I put up a Quieting Charm. I don't think he'll be trying his lectures on me again."
"I understand why you did that," Hermione said, her voice tight, "I wanted to tell him off too, but I really wish you hadn't, it will make things harder for me, Harry!"
Harry looked up from Vegetable Cunning again, he wasn't getting any reading done at this rate; and he saw that Hermione was still reading whatever book she had, not looking up at him. Her hands turned another page even as he watched.
"I think you're taking the wrong approach by trying to defend yourself at all," Harry said. "I really do think that. You are who you are. You're friends with whoever you choose. Tell anyone who questions you to shove it."
Hermione just shook her head, and turned another page.
"Option two," Harry said. "Go to Fred and George and tell them to have a little talk with their wayward brother, those two are genuine good guys -"
"It's not just Ron," Hermione said in almost a whisper. "Lots of people are saying it, Harry. Even Mandy is giving me worried looks when she thinks I'm not looking. Isn't it funny? I keep worrying that Professor Quirrell is sucking you into the darkness, and now people are warning me just the same way I try to warn you."
"Well, yeah," said Harry. "Doesn't that reassure you a bit about me and Professor Quirrell?"
"In a word," said Hermione, "no."
There was a silence that lasted long enough for Hermione to turn another page, and then her voice, in a real whisper this time, "And, and Padma is going around telling everyone that, that since I couldn't cast the P-Patronus Charm, I must only be p-pretending to be n-nice..."
"Padma didn't even try herself!" Harry said indignantly. "If you were a Dark Witch who was just pretending, you wouldn't have tried in front of everyone, do they think you're stupid?"
Hermione smiled a little, and blinked a few times.
"Hey, I have to worry about actually going evil. Here the worst case scenario is that people think you're more evil than you really are. Is that going to kill you? I mean, is it all that bad?"
The young girl nodded, her face screwed up tight.
"Look, Hermione... if you worry that much about what other people think, if you're unhappy whenever other people don't picture you exactly the same way you picture yourself, that's already dooming yourself to always be unhappy. No one ever thinks of us just the same way we think of ourselves."
"I don't know how to explain to you," Hermione said in a sad soft voice. "I'm not sure it's something you could ever understand, Harry. All I can think of to say is, how would you feel if I thought you were evil?"
"Um..." Harry visualized it. "Yeah, that would hurt. A lot. But you're a good person who thinks about that sort of thing intelligently, you've earned that power over me, it would mean something if you thought I'd gone wrong. I can't think of a single other student, besides you, whose opinion I'd care about the same way -"
"You can live like that," whispered Hermione Granger. "I can't."
The girl had gone through another three pages in silence, and Harry had returned his eyes to his own book and was trying to regain his focus, when Hermione finally said, in a small voice, "Are you really sure I mustn't know how to cast the Patronus Charm?"
"I..." Harry had to swallow a sudden lump in his throat. He suddenly saw himself not knowing why the Patronus Charm didn't work for him, not being able to show Draco, just being told that there was a reason, and nothing more. "Hermione, your Patronus would shine with the same light but it wouldn't be normal, it wouldn't look like people think Patronuses should look, anyone who saw it would know there was something strange going on. Even if I told you the secret you couldn't demonstrate to anyone, unless you made them face the other way so they could only see the light, and... and the most important part of any secret is the knowledge that a secret exists, you could only show one or two friends if you swore them to secrecy..." Harry's voice trailed off helplessly.
"I'll take it." Her voice was still small.
It was very hard not to just blurt out the secret, right there in the library.
"I, I shouldn't, I really shouldn't, it's dangerous, Hermione, it could do a lot of harm if that secret got out! Haven't you heard the saying, three can keep a secret if two are dead? That telling just your closest friends is the same as telling everyone, because you're not just trusting them, you're trusting everyone they trust? It's too important, too much of a risk, it's not the sort of decision that should be made for the sake of fixing someone's reputation at school!"
"Okay," Hermione said. She closed the book and put it back on the shelf. "I can't concentrate right now, Harry, I'm sorry."
"If there's anything else I can do -"
"Be nicer to everyone."
The girl didn't look back as she walked out of the stacks, which might have been a good thing, because the boy was frozen in place, unmoving.
After a while, the boy started turning pages again.
1 comments
Comments sorted by top scores.
comment by andrew sauer (andrew-sauer) · 2023-12-10T20:19:06.745Z · LW(p) · GW(p)
Just so long as you're okay with us being eaten by giant monsters that didn't do enough research into whether we were sentient.
I'm okay with that, said Slytherin. Is everyone else okay with that? (Internal mental nods.)
I'd bet quite a lot they're not actually okay with that, they just don't think it will happen to them...