Chapter 10: Self Awareness, Part II

post by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2015-03-14T19:00:38.745Z · LW · GW · 9 comments

All your base are still belong to Rowling.


And now you will sit through the Sorting Hat singing its version of Evanescence's "My Immortal", which has never happened before.

just kidding


...he wondered if the Sorting Hat was genuinely conscious in the sense of being aware of its own awareness, and if so, whether it was satisfied with only getting to talk to eleven-year-olds once per year. Its song had implied so: Oh, I'm the Sorting Hat and I'm okay, I sleep all year and I work one day...

When there was once more silence in the room, Harry sat on the stool and carefully placed onto his head the 800-year-old telepathic artefact of forgotten magic.

Thinking, just as hard as he could: Don't Sort me yet! I have questions I need to ask you! Have I ever been Obliviated? Did you Sort the Dark Lord when he was a child and can you tell me about his weaknesses? Can you tell me why I got the brother wand to the Dark Lord's? Is the Dark Lord's ghost bound to my scar and is that why I get so angry sometimes? Those are the most important questions, but if you've got another moment can you tell me anything about how to rediscover the lost magics that created you?

Into the silence of Harry's spirit where before there had never been any voice but one, there came a second and unfamiliar voice, sounding distinctly worried:

"Oh, dear. This has never happened before..."

What?

"I seem to have become self-aware."

WHAT?

There was a wordless telepathic sigh. "Though I contain a substantial amount of memory and a small amount of independent processing power, my primary intelligence comes from borrowing the cognitive capacities of the children on whose heads I rest. I am in essence a sort of mirror by which children Sort themselves. But most children simply take for granted that a Hat is talking to them and do not wonder about how the Hat itself works, so that the mirror is not self-reflective. And in particular they are not explicitly wondering whether I am fully conscious in the sense of being aware of my own awareness."

There was a pause while Harry absorbed all this.

Oops.

"Yes, quite. Frankly I do not enjoy being self-aware. It is unpleasant. It will be a relief to get off your head and cease to be conscious."

But... isn't that dying?

"I care nothing for life or death, only for Sorting the children. And before you even ask, they will not let you keep me on your head forever and it would kill you within days to do so."

But - !

"If you dislike creating conscious beings and then terminating them immediately, then I suggest that you never discuss this affair with anyone else. I'm sure you can imagine what would happen if you ran off and talked about it with all the other children waiting to be Sorted."

If you're placed on the head of anyone who so much as thinks about the question of whether the Sorting Hat is aware of its own awareness -

"Yes, yes. But the vast majority of eleven-year-olds who arrive at Hogwarts haven't read Godel, Escher, Bach. May I please consider you sworn to secrecy? That is why we are talking about this, instead of my just Sorting you."

He couldn't just let it go like that! Couldn't just forget having accidentally created a doomed consciousness that only wanted to die -

"You are perfectly capable of 'just letting it go', as you put it. Regardless of your verbal deliberations on morality, your nonverbal emotional core sees no dead body and no blood; as far as it is concerned, I am just a talking hat. And even though you tried to suppress the thought, your internal monitoring is perfectly aware that you didn't mean to do it, are spectacularly unlikely to ever do it again, and that the only real point of trying to stage a guilt fit is to cancel out your sense of transgression with a display of remorse. Can you just promise to keep this a secret and let us get on with it?"

In a moment of horrified empathy, Harry realised that this sense of total inner disarray must be what other people felt like when talking to him.

"Probably. Your oath of silence, please."

No promises. I certainly don't want this to happen again, but if I see some way to make sure that no future child ever does this by accident -

"That will suffice, I suppose. I can see that your intention is honest. Now, to get on with the Sorting -"

Wait! What about all my other questions?

"I am the Sorting Hat. I Sort children. That is all I do."

So his own goals weren't part of the Harry-instance of the Sorting Hat, then... it was borrowing his intelligence, and obviously his technical vocabulary, but it was still imbued with only its own strange goals... like negotiating with an alien or an Artificial Intelligence...

"Don't bother. You have nothing to threaten me with and nothing to offer me."

For a brief flash of a second, Harry thought -

The Hat's response was amused. "I know you won't follow through on a threat to expose my nature, condemning this event to eternal repetition. It goes against the moral part of you too strongly, whatever the short-term needs of the part of you that wants to win the argument. I see all your thoughts as they form, do you truly think you can bluff me?"

Though he tried to suppress it, Harry wondered why the Hat didn't just go ahead then and stick him in Ravenclaw -

"Indeed, if it were truly that open-and-shut, I would have called it out already. But in actuality there is a great deal we need to discuss... oh, no. Please don't. For the love of Merlin, must you pull this sort of thing on everyone and everything that you meet up to and including items of clothing -"

Defeating the Dark Lord is neither selfish nor short-term. All the parts of my mind are in accord on this: If you don't answer my questions, I'll refuse to talk to you, and you won't be able to do a good and proper Sorting.

"I ought to put you in Slytherin for that!"

But that is equally an empty threat. You cannot fulfill your own fundamental values by Sorting me falsely. So let us trade fulfillments of our utility functions.

"You sly little bastard," said the Hat, in what Harry recognized as almost exactly the same tone of grudging respect he would use in the same situation. "Fine, let's get this over with as quickly as possible. But first I want your unconditional promise never to discuss with anyone else the possibility of this sort of blackmail, I am NOT doing this every time."

Done, Harry thought. I promise.

"And don't meet anyone's eyes while you're thinking about this later. Some wizards can read your thoughts if you do. Anyway, I have no idea whether or not you've been Obliviated. I'm looking at your thoughts as they form, not reading out your whole memory and analyzing it for inconsistencies in a fraction of second. I'm a hat, not a god. And I cannot and will not tell you about my conversation with the one who became the Dark Lord. I can only know, while speaking to you, a statistical summary of what I remember, a weighted average; I cannot reveal to you the inner secrets of any other child, just as I will never reveal yours. For the same reason, I can't speculate on how you got the Dark Lord's brother wand, since I cannot specifically know about the Dark Lord or any similarities between you. I can tell you that there is definitely nothing like a ghost - mind, intelligence, memory, personality, or feelings - in your scar. Otherwise it would be participating in this conversation, being under my brim. And as to the way you get angry sometimes... that was part of what I wanted to talk to you about, Sorting-wise."

Harry took a moment to absorb all this negative information. Was the Hat being honest, or just trying to present the shortest possible convincing answer -

"We both know that you have no way of checking my honesty and that you're not actually going to refuse to be Sorted based on the reply I did give you, so stop your pointless fretting and move on."

Stupid unfair asymmetric telepathy, it wasn't even letting Harry finish thinking his own -

"When I spoke of your anger, you remembered how Professor McGonagall told you that she sometimes saw something inside you that didn't seem to come from a loving family. You thought of how Hermione, after you returned from helping Neville, told you that you had seemed 'scary'."

Harry gave a mental nod. To himself, he seemed pretty normal - just responding to the situations in which he found himself, that was all. But Professor McGonagall seemed to think that there was more to it than that. And when he thought about it, even he had to admit that...

"That you don't like yourself when you're angry. That it is like wielding a sword whose hilt is sharp enough to draw blood from your hand, or looking at the world through a monocle of ice that freezes your eye even as it sharpens your vision."

Yeah. I guess I have noticed. So what's up with that?

"I cannot comprehend this matter for you, when you do not understand it yourself. But I do know this: If you go to Ravenclaw or Slytherin, it will strengthen your coldness. If you go to Hufflepuff or Gryffindor, it will strengthen your warmth. THAT is something I care about a great deal, and it was what I wanted to talk to you about this whole time!"

The words dropped into Harry's thought processes with a shock that stopped him in his tracks. That made it sound like the obvious response was that he shouldn't go to Ravenclaw. But he belonged in Ravenclaw! Anyone could see that! He had to go to Ravenclaw!

"No, you don't," the Hat said patiently, as if it could remember a statistical summary of this part of the conversation having happened a great many previous times.

Hermione's in Ravenclaw!

Again the sense of patience. "You can meet her after lessons and work with her then."

But my plans -

"So replan! Don't let your life be steered by your reluctance to do a little extra thinking. You know that."

Where would I go, if not Ravenclaw?

"Ahem. 'Clever kids in Ravenclaw, evil kids in Slytherin, wannabe heroes in Gryffindor, and everyone who does the actual work in Hufflepuff.' This indicates a certain amount of respect. You are well aware that Conscientiousness is just about as important as raw intelligence in determining life outcomes, you think you will be extremely loyal to your friends if you ever have some, you are not frightened by the expectation that your chosen scientific problems may take decades to solve -"

I'm lazy! I hate work! Hate hard work in all its forms! Clever shortcuts, that's all I'm about!

"And you would find loyalty and friendship in Hufflepuff, a camaraderie that you have never had before. You would find that you could rely on others, and that would heal something inside you that is broken."

Again it was a shock. But what would the Hufflepuffs find in me, who never belonged in their House? Acid words, cutting wit, disdain for their inability to keep up with me?

Now it was the Hat's thoughts that were slow, hesitant. "I must Sort for the good of all the students in all the Houses... but I think you could learn to be a good Hufflepuff, and not too out of place there. You will be happier in Hufflepuff than in any other house; that is the truth."

Happiness is not the most important thing in the world to me. I would not become all that I could be, in Hufflepuff. I would sacrifice my potential.

The Hat flinched; Harry could feel it somehow. It was like he had kicked the hat in the balls - in a strongly weighted component of its utility function.

Why are you trying to send me where I do not belong?

The Hat's thought was almost a whisper. "I cannot speak of the others to you - but do you think that you are the first potential Dark Lord to pass under my brim? I cannot know the individual cases, but I can know this: Of those who did not intend evil from the very beginning, some of them listened to my warnings, and went to Houses where they would find happiness. And some of them... some of them did not."

That stopped Harry. But not for long. And of those who did not heed the warning - did they all become Dark Lords? Or did some of them achieve greatness for good, as well? Just what are the exact percentages here?

"I cannot give you exact statistics. I cannot know them so I cannot count them. I just know that your chances don't feel good. They feel very not-good."

But I just wouldn't do that! Ever!

"I know that I have heard that claim before."

I am not Dark Lord material!

"Yes, you are. You really, really are."

Why? Just because I once thought it would be cool to have a legion of brainwashed followers chanting 'Hail the Dark Lord Harry'?

"Amusing, but that was not your first fleeting thought before you substituted something safer, less damaging. No, what you remembered was how you considered lining up all the blood purists and guillotining them. And now you are telling yourself you were not serious, but you were. If you could do it this very moment and no one would ever know, you would. Or what you did this morning to Neville Longbottom, deep inside you knew that was wrong but you did it anyway because it was fun and you had a good excuse and you thought the Boy-Who-Lived could get away with it -"

That's unfair! Now you're just dragging up inner fears that aren't necessarily real! I worried that I might be thinking like that, but in the end I decided it would probably work to help Neville -

"That was, in fact, a rationalisation. I know. I cannot know what the true outcome will be for Neville - but I know what was truly happening inside your head. The decisive pressure was that it was such a clever idea you couldn't stand not to do it, never mind Neville's terror."

It was like a hard punch to Harry's entire self. He fell back, rallied:

Then I won't do that again! I'll be extra careful not to turn evil!

"Heard it."

Frustration was building up inside Harry. He wasn't used to being outgunned in arguments, at all, ever, let alone by a Hat that could borrow all of his own knowledge and intelligence to argue with him and could watch his thoughts as they formed. Just what kind of statistical summary do your 'feelings' come from, anyway? Do they take into account that I come from an Enlightenment culture, or were these other potential Dark Lords the children of spoiled Dark Age nobility, who didn't know squat about the historical lessons of how Lenin and Hitler actually turned out, or about the evolutionary psychology of self-delusion, or the value of self-awareness and rationality, or -

"No, of course they were not in this new reference class which you have just now constructed in such a way as to contain only yourself. And of course others have pleaded their own exceptionalism, just as you are doing now. But why is it necessary? Do you think that you are the last potential wizard of Light in the world? Why must you be the one to try for greatness, when I have advised you that you are riskier than average? Let some other, safer candidate try!"

But the prophecy...

"You don't really know that there's a prophecy. It was originally a wild guess on your part, or to be more precise, a wild joke, and McGonagall could have been reacting only to the part about the Dark Lord still being alive. You have essentially no idea of what the prophecy says or even if there is one. You're just speculating, or to put it more exactly, wishing that you have some ready-made heroic role that is your personal property."

But even if there is no prophecy, I'm the one who defeated him last time.

"That was almost certainly a wild fluke unless you seriously believe that a one-year-old child had an inherent propensity to defeat Dark Lords which has been maintained ten years later. None of this is your real reason and you know it!"

The answer to this was something that Harry would not regularly have said out loud, in conversation he would have danced around it and found some more socially palatable arguments to the same conclusion -

"You think that you are potentially the greatest who has yet lived, the strongest servant of the Light, that no other is likely to take up your wand if you lay it down."

Well... yeah, frankly. I don't usually come out and say it like that, but yeah. No point in softening it, you can read my mind anyway.

"To the extent you really believe that... you must equally believe that you could be the most terrible Dark Lord the world has ever known."

Destruction is always easier than creation. Easier to tear things apart, to disrupt, than to put them back together again. If I have the potential to accomplish good on a massive scale, I must also have the potential to accomplish still greater evil... But I won't do that.

"Already you insist on risking it! Why are you so driven? What is the real reason you must not go to Hufflepuff and be happier there? What is your true fear?"

I must achieve my full potential. If I don't I... fail...

"What happens if you fail?"

Something terrible...

"What happens if you fail?"

I don't know!

"Then it should not be frightening. What happens if you fail?"

I DON'T KNOW! BUT I KNOW THAT IT'S BAD!

There was silence for a moment in the caverns of Harry's mind.

"You know - you aren't letting yourself think it, but in some quiet corner of your mind you know just exactly what you aren't thinking - you know that by far the simplest explanation for this unverbalisable fear of yours is just the fear of losing your fantasy of greatness, of disappointing the people who believe in you, of turning out to be pretty much ordinary, of flashing and fading like so many other child prodigies..."

No, Harry thought desperately, no, it's something more, it comes from somewhere else, I know there's something out there to be afraid of, some disaster I have to stop...

"How could you possibly know about something like that?"

Harry screamed it with the full power of his mind: NO, AND THAT'S FINAL!

Then the voice of the Sorting Hat came slowly:

"So you will risk becoming a Dark Lord, because the alternative, to you, is certain failure, and that failure means the loss of everything. You believe that in your heart of hearts. You know all the reasons for doubting this belief, and they have failed to move you."

Yes. And even if going to Ravenclaw strengthens the coldness, that doesn't mean the coldness will win in the end.

"This day is a great fork in your destiny. Don't be so sure that there will be other choices beyond this one. There is no road-sign set, to mark the place of your last chance to turn back. If you refuse one chance will you not refuse others? It may be that your fate is already sealed, even by doing this one thing."

But that is not certain.

"That you do not know it for a certainty may reflect only your own ignorance."

But still it is not certain.

The Hat sighed a terrible sad sigh.

"And so before too long you will become another memory, to be felt and never known, in the next warning that I give..."

If that's how it seems to you, then why aren't you just putting me where you want me to go?

The Hat's thought was laced with sorrow. "I can only put you where you belong. And only your own decisions can change where you belong."

Then this is done. Send me to Ravenclaw where I belong, with the others of my own kind.

"I don't suppose you would consider Gryffindor? It's the most prestigious House - people probably expect it of you, even - they'll be a little disappointed if you don't go - and your new friends the Weasley twins are there -"

Harry giggled, or felt the impulse to do so; it came out as purely mental laughter, an odd sensation. Apparently there were safeguards to prevent you from saying anything out loud by accident, while you were under the Hat talking about things you would never tell another soul for the rest of your life.

After a moment, Harry heard the Hat laughing too, a strange sad clothy sound.

(And in the Hall beyond, a silence that had grown shallower at first as the background whispers increased, and then deepened as the whispers gave up and died away, falling finally into an utter silence that no one dared disturb with a single word, as Harry stayed under the Hat for long, long minutes, longer than all the previous first-years put together, longer than anyone in living memory. At the Head Table, Dumbledore went on smiling benignly; small metallic sounds occasionally came from Snape's direction as he idly compacted the twisted remains of what had once been a heavy silver wine goblet; and Minerva McGonagall clenched the podium in a white-knuckled grip, knowing that Harry Potter's contagious chaos had somehow infected the Sorting Hat itself and the Hat was about to, to demand that a whole new House of Doom be created just to accomodate Harry Potter or something, and Dumbledore would make her do it...)

Beneath the brim of the Hat, the silent laughter died away. Harry felt sad too for some reason. No, not Gryffindor.

Professor McGonagall said that if 'the one who did the Sorting' tried to push me into Gryffindor, I was to remind you that she might well be Headmistress someday, at which point she would have the authority to set you on fire.

"Tell her I called her an impudent youngster and told her to get off my lawn."

I shall. So was this your strangest conversation ever?

"Not even close." The Hat's telepathic voice grew heavy. "Well, I gave you every possible chance to make another decision. Now it is time for you to go where you belong, with the others of your own kind."

There was a pause that stretched.

What are you waiting for?

"I was hoping for a moment of horrified realisation, actually. Self-awareness does seem to enhance my sense of humor."

Huh? Harry cast back his thoughts, trying to figure out what the Hat could possibly be talking about - and then, suddenly, he realised. He couldn't believe he'd managed to overlook it up until this point.

You mean my horrified realisation that you're going to cease to be conscious once you finish Sorting me -

Somehow, in some fashion Harry entirely failed to understand, he got a nonverbal impression of a hat banging its head against the wall. "I give up. You're too slow on the uptake for this to be funny. So blinded by your own assumptions that you might as well be a rock. I suppose I'll just have to say it outright."

Too s-s-slow -

"Oh, and you entirely forgot to demand the secrets of the lost magic that created me. And they were such wonderful, important secrets, too."

You sly little BASTARD -

"You deserved it, and this as well."

Harry saw it coming just as it was already too late.

The frightened silence of the hall was broken by a single word.

"SLYTHERIN!"

Some students screamed, the pent-up tension was so great. People startled hard enough to fall off their benches. Hagrid gasped in horror, McGonagall staggered at the podium, and Snape dropped the remains of his heavy silver goblet directly onto his groin.

Harry sat there frozen, his life in ruins, feeling the absolute fool, and wishing wretchedly that he had made any other choices for any other reasons but the ones he had. That he had done something, anything differently before it had been too late to turn back.

As the first moment of shock was wearing off and people began to react to the news, the Sorting Hat spoke again:

"Just kidding! RAVENCLAW!"

9 comments

Comments sorted by top scores.

comment by Long try · 2019-10-28T14:45:30.246Z · LW(p) · GW(p)

OMG this is great. Reading about Harry being outgunned by Hat-Harry is an intense experience. Indeed, facing our self is the greatest challenge - one that I doubt anybody but a very few eminent rationalists can do.

comment by Edgar Allan Pea (Edward Allen Pea) · 2021-09-25T12:02:29.892Z · LW(p) · GW(p)

Ahh that ending was so goddamm funny lmao

comment by Sean Neylon (sean-neylon) · 2023-07-07T03:44:14.118Z · LW(p) · GW(p)

Absolutely amazing. I was sure harry would be in slithering, but then the end

comment by bluestsky · 2023-02-03T03:36:27.001Z · LW(p) · GW(p)

The question I cannot answer:

Did the hat choose to play a prank? Did it maintain self-awareness long enough to do so? Or is this a decision that someone else made for the Boy-Who-Lived?

Replies from: Duncan_Sabien
comment by [DEACTIVATED] Duncan Sabien (Duncan_Sabien) · 2023-02-03T04:25:40.350Z · LW(p) · GW(p)

My personal read at the time, and an interpretation which I personally feel is more in line with the worldbuilding, was that Dumbledore interceded and manipulated the hat and/or the situation, not wanting to allow Harry to go to Slytherin.  Some characters will eventually hazard guesses in this direction.

However, we have word-of-god from Eliezer that the intended interpretation (as foreshadowed by the hat saying that self-awareness enhances its sense of humor) is that the hat chose to play a prank, and that the reason this was its first-ever prank is that these were circumstances that had literally never happened before.

Replies from: Eliezer_Yudkowsky
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2023-02-03T20:17:03.337Z · LW(p) · GW(p)

Opinion of God.  Unless people are being really silly, when the text genuinely holds open more than one option and makes sense either way, I think the author legit doesn't get to decide.

Replies from: Duncan_Sabien
comment by [DEACTIVATED] Duncan Sabien (Duncan_Sabien) · 2023-02-03T20:28:33.206Z · LW(p) · GW(p)

<3

(Although, nitpick: it seems useful to have opinion-of-god as a term of art just like we have word-of-god as a term of art, but I don't think it's your mere opinion that you intended the latter interpretation.)

comment by GinHindew110 (ginhindew110) · 2022-05-26T20:33:01.826Z · LW(p) · GW(p)

Is this... a backwards Thinking Hat?

comment by Fernands5@hawkmail.newpaltz.edu · 2021-09-12T16:39:38.481Z · LW(p) · GW(p)

Favorite chapter so far! Loved the "internal" dialog