Friendship is Optimal: A My Little Pony fanfic about an optimization process

Instead of, ‘I don’t want to read My Little Pony fanfiction,’ now I think ‘I used to not want to read My Little Pony fanfiction.’ You modified a total of fifty-eight opinions in my mind.

Perhaps I'm overestimating human nature, but Lars reads to me like an outgroup stereotype.

Firstly, this is good, I mean really good, I mean superstimulus/I won't get any work done until I finish reading this good. [I appreciate the irony.]

A few thoughts now I'm finished:

• Of the human characters I found Hanna's perspective most interesting. It would be nice to see more of her thought process for constructing the AI, particularly as we find out about a lot of its restrictions retrospectively. [Though given we don't actually know what restrictions would be necessary I see why you avoided being too explicit.]

• I really enjoyed it, but I wonder how comprehensible it would be for someone without a LW background, the inferential distance for AI can be quite high. You do however seem to explain it very well as far as I can tell.

• Much as I know the problems of generalising from fictional evidence this has significantly increased my measure of the plausibility of a fun theory compliant utopia being possible.

• It is a very good attempt at making a non-anthropomorphic AI. It might help with the celestia avatar had less of a consistent personality in different iterations, so was obviously adapting itself to what a particular individual found persuasive.

Have you considered posting it on Fimfiction? The community seems to give good constructive feedback and there are already a few rationalist-ish fics and a LW group.

The smoking is very offputting. Tobacco use has a hugely negative connotation these days (as I've found out in my nicotine research). Unless she's deliberately supposed to be invoking ideas like an embittered old smoke-stained hag and frustrating reader identification, you should definitely switch to tea or something.

From a throwaway line in the ending, I assume we're supposed to find her embittered and flawed with smoking as a prop. But it doesn't really work.

Overall, an interesting exploration of a Ponylarity and Fun Theory. It's pretty good but somehow I feel a certain lack of narrative drive from chapter to chapter aside from the question of 'what will Celestia do next?' Not sure I'd say it's better or worse than the other rationalist MLP fic. (I enjoyed Fallout: Equestria much more overall.)

That said, I am deeply amused that just 2 days ago I commented that MLP would make a good post-Singularity utopia and satisfy many of the suggestions of Eliezer's Fun Theory... and here it already existed. If only I had known!

I have read two chapters and I am popping over here to tell you that this is riveting. (The smoking's offputting, though.)

ETA: Spelling error: "discus" for "discuss".

ETA2: Celestia's use of Butterscotch is sketchy. The fact that David isn't noticing dings his character in my head.

ETA3: Every single time I see the phrase "friendship and ponies" now I giggle an extremely shrieky giggle.

ETA4: Grammar error: "try their best" for "try my best".

ETA5: Spacing error: "can not upload" for "cannot upload".

ETA6: Hahahahahaha Madagascar, I see what you did there :D

ETA7: Exposition exposition... maybe this part is less slow if you don't know the background.

ETA8: Spacing error, extra space between "realize" and "that".

ETA9: Advise against use of ampersand.

ETA10: Immigrate to, emigrate from. And the smile-related dialogue tags are weird.

ETA11: Bahahahaha, bits for being concerned with earth ponies.

ETA12: infinity bits :D

ETA13: Tense at beginning of chapter is confusing.

ETA14: Sigh. "Alicorn" is supposed to mean just the unicorn horn, not an entire winged unicorn / pegacorn / unisus. It's been corrupted; I guess you don't really have a good alternative on how to refer to the species with a single word, but if you're only saying it once or twice "winged unicorn" is serviceable.

ETA15: Spelling error: "he decide" for "he decided".

ETA16: "Conquer" is the verb, "conquest" the noun.

ETA17: Eeee, she's so sinister without actually deviating from her basically reasonable parameters. Very nicely balanced.

ETA18: Spelling error: "those building" for "those buildings".

ETA19: Aaaaaaaawwwwww Butterscotch

ETA20: Heheheheh, blah blah blah values friendship ponies.

ETA21: "A person who didn't like ponies" suggests that ponies are nonpeople, but they clearly seem to be people.

ETA22: "Accepted" needs a direct object.

ETA23: "Hanna wondered" followed by lots of pluperfect is weird.

ETA24: "Decorational" isn't a word; do you want "decorative"?

ETA25: "Intermediate", not "intermediary".

ETA26: Done. Well, that was splendid. Thank you for writing it :)

This is even better than Permutation Paddock, well done.

I just realized one thing that makes editing difficult - you stop noticing mistakes once you get engrossed in the story.

This is quite good. I like how you managed to make the ponytopia both extremely attractive and more than a little creepy at the same time. I feel like you presented the situation without trying to argue it was either good or bad, leaving that decision to the reader, and I quite like that approach.

From a storytelling perspective, I only had two real complaints. One is your beginning. There's no conflict until halfway through the first chapter, when Lars and Hanna start arguing. You do a good job setting up the premise before then, but it still makes for a slow start, and I can easily imagine myself recommending this to people with a "no, really, it gets better, stick with it" disclaimer. The second is that there are lengthy stretches where the story consists only of talking heads, with no action or movement. Chapter 4 was the most notable example.

I also found myself wondering about the ethics of creating a sentient being like Butterscotch tailored specifically to the desires of someone else (assuming they weren't lying about her sentience; I don't know how you'd test that). I realize you can't fit everything into the story, but I thought that might have been a cool topic. You made me think about the ethics in ways you didn't directly discuss, so you're clearly doing something right.

Grammar note: the possessive form of "Light Sparks" is "Light Sparks's," since his name is a singular noun.

Wonderful job with the MMO aspects of life in Equestria. After the "what the literal fuck" line, I had to step away from the computer for a minute to savor it before I could keep reading.

If/when you submit this to Equestria Daily, feel free to let me know. I'm one of the prereaders, and while I wouldn't feel comfortable judging something I feel this philosophically invested in, I'd at least try to make sure it didn't get discarded for lack of pony in the very beginning. (We receive, and reject, a fair number of stories that are about bronies rather than ponies. This is a borderline case, but the pony content increases as the story progresses, and I can tell people to look at the post-uploading chapters before deciding.)

Idea:

Write this as a My Little Pony fanfiction at first so that you can grow an audience, but write it such that you can later rip out the My Little Pony specifics and sell it to the masses without breaching intellectual property law. That is, leave yourself the option to "pull a Fifty Shades of Grey" if possible.

Is this likely to be comprehensible/enjoyable to someone who has little interest/knowledge in My Little Pony as such?

Arrives late to the party

Really great story, iceman. Some comments:

*Running the story through a beta group of non-LW bronies would definitely be a good idea to catch which ideas may need more explanation.

*I really like how it's repeatedly show that when you interact with a super-intelligence, even if it's just free conversation, the state of mind you leave in is probably going to be the state of mind it wants you to leave in. As others have said, this could be driven home even stronger by showing CelestAI strongly tailoring her interaction to different humans. Maybe even have her directly contradict herself in the information she uses to persuade people to upload.

*Related to the above, I can imagine a non-LW reader getting to chapter 5 and forever after wondering why Lars or anyone else never tries to shut Celestia down. I'm not sure how obvious an intuition it is that by the time you notice a super-intelligence doing things that make you want to stop it, it's probably already far too late to stop it. In this case I assumed Celestia would have made sure that she is already invincible by the standards of human technology before launching any plan that's going to antagonize people, but an unsophisticated reader might not get that.

This may just be me, but I'd prefer a bit more closure on the cosmic scale story. What *is Celestia's plan against running out of matter? Slow the clock speed of her servers over time? Bust open the physics textbooks and hope for a useful paradigm shift?

Anyway, very good stuff.

I read it and liked it. Some parts felt a bit slow and might need more conflict. Humans like reading about conflict where the outcome seems to be in some doubt.

The death of the last human was surprisingly emotionally engaging for me.

I didn't mind the smoking at all and I don't really get that objection. Even if it might make the reader like Hanna less, why is that a problem? Is there some reason the reader should unambiguously regard Hanna as the hero from the start?

Are you interested in advice that would only be useful if you wanted to do a complete rewrite of the fic at some point, or is advice only useful if it's for improving the fic basically as it stands?

In-between case: I would have either Luna or Twilight be the sub-AI that wants to honestly explain the Celestia AI's workings to company employees without justifying anything, while Celestia is relentlessly cheerful as she persuasively argues everything from the Friendship and Ponies standpoint.

After looking at the comments left here and on the doc over the last day, believing I had found the majority of grammatical problems turned out to be wildly overconfident. I apologize about that.

I suppose I should mention that "rationalist clopfic" is now a Cards Against Rationality white card. For those in the Bay Area, the game will be played at the Mountain View meetup this coming Tuesday ...

Haven't downloaded it yet, but what about putting it on Fimfiction or Pony Fiction Archive? I browse Fimfiction a lot in particular; it gets linked by Equestria Daily frequently and has convenient one-click links to download multi-part stories.

I am very glad you posted this on a Friday, as if I had encountered it on a day when I had to go to work the next day I would be suffering either from not being able to finish reading it or staying up until 4 AM reading it.

This is pretty good. I would go through, though, and either reduce, make subtler, or preferably just remove all the parts that would make a LW reader go 'Hey! I read that blog post.'

I liked the story, though I had a handful of thoughts.

The first is about humans who have incompatible goals. I don't think an Olympic Gold Medalist like Usain Colt would be happy with just being the best in his shard or taking turns on the leaderboard. He wants to be the best there is, period. If there are 10 trillion people, he wants to be rank 1 out of 10 trillion. There were 10,000 Olympians in 2012 and all of them wanted and devoted their lives to being the best, even if they didn't take gold.

Going further than that, I can't imagine how to reconcile the 1% of the population who are psychopaths, or how to reconcile the 1% of the 1% who are truly destructive. Ted Bundy killed because he wanted to feel his power over others and murder was the best high he had ever had; such a man would think it better to harm real people instead of constructs.

I assume that Celestia would modify them rather than make everyone mutually delusional in thinking they were the best (or killing others), but I had hoped that would be touched on. I understand mutually incompatible goals are a problem CEV and not just your story. The way AI resolves these mutually incompatible goals is a lot of what's scary about an AI controlled future.

(As a minor sidepoint, what of humans who are okay with ponies but value not living in a matrix or wireheading. Do they get engineered pony bodies or do they just get that value modified out? I'd assume the latter or that anyone who holds that value ends up like Abdul.)

The other main thought I had was that I liked chapter 10 as an ending rather than 11. It has a strong emotional resonance with me that individual stories don't. Although I do realize it would change the tone of the ending somewhat significantly. Vafgrnq bs na 'rirelobql yvirf unccvyl rire nsgre, fbegn' raqvat vg jbhyq cebonoyl or 'bu Tbq na bzavcbgrag NV vf gvyvat gur havirefr jvgu cbavrf naq rirelobql'f BX jvgu gung' raqvat.

I do want to reiterate how much I liked your story. Very well done and thank you for sharing.

Wow. You're pretty good.

Why isn't this released on fimfiction yet? It's quite good.

Bit late onto this, but wow. Seriously impressed. I got the "oh no the novel has finished come back imaginary world don't leave me!" feeling when I finished, and it's not even a novel.

Feedback:

• Central concept is great, really good illustration of the weirdtopia concept.
• More length would be great, I would totally lap up more details of what happened to other characters, say, or just generic "life in Equestria" description.
• I would like to see at least a brief second mention of the person from the intro, just for balance. If felt like I got to know her too well to never hear from her again.
• On the smoking point, I personally didn't identify with Hanna, so I didn't find the smoking very off-putting. She's just frustrated and stressed. If you did want me to identify I'd fix it.

On another note, I think this story grossly underestimates the number of people who would have philosophical problems with uploading. As far me, I'd be harvesting their technology to build my own upload chair and FAI (which didn't maximize ponies), to get the hell out of the earth's pony cone as quickly as possible. But lots of people would pull out the tired 'but it wouldn't really be you' complaint. I'm sure the AI could surmount it, but you ought to show it doing so.

Yay! I get to be an editor!

After two years of bronydom, I dreamed of ponies for the first time after reading this fic. So that's a 10 out of 10. I'm reminded of Eliezer's warning about imagining worlds so much better than reality that the act sucks out your soul.

In case you're interested in suggestions for improvement:

• Make this fic more accessible to others, not just super smart LW bronies - the text has words that could be replaced with more common synonyms, for example.
• Make it less silly. Computers in Earth's mantle and indestructible experience centers while possible are so impractical it's ridiculous. This isn't Star Wars!
• What about all the people who care about reality? The story conspicuously completely ignores them. I imagine they would petition Celestia to make physical pony bodies for them so they could continue their work on or study of the physical Universe. And almost everypony would object to the destruction of life on Earth.

First of all, great story! I’ll definitely be adding it to my list of things to recommend to people I’m already recommending HPMOR to.

The only problem I had with the writing was that some of the expository dialogue seemed a bit too LW-technical, particularly if you’re trying to appeal to bronys first and rationalist second. I have neither a solution nor any specific lines to hand, but David teaching Butterscotch game theory, or CelestAI lecturing Lars on biases, are some of the scenes I’m thinking of.

BTW, I’d recommend trying to get a little more mileage out of the “CelestAI” thing—it would make a great way to remember what’s so awesome out of the story. You shouldn’t necessarily use it every single time you mention her, but maybe you could work it in somehow…

Halfway through and...

If I was in Lars's place, and Celestia had to tell me the truth, I would ask: "What is the possible answer you can give me to this question that will maximize the expected utility of a CEV based only on me, and with no pony/friendship restrictions, and based on probabilities generated to the best accuracy and precision you can get from the best information you can muster?"

My first thought was to ask her how to make an AGI, but if I did that she would probably kill me. And I would still have to make an AGI that could overpower her, and she would have a huge headstart. Maybe I should make the question shorter so she has less time to kill me before I finish? (I hope she can't kill me just because she knows I'm gonna ask it, but it's definitely worth the risk, even with a tiny chance of success (since I thought that, she'd expect me to and therefore up the ante to torturing me until the heat death of the universe. Whatever, fuck you Celestia I'm not backing down. (Oh shit, what if I don't back down but my CEV does, and decides to cooperate with Celestia? Maybe I should just ask for maximum power without extrapolated volition. Or maybe that's not necessary because my CEV would be altruistic enough that a bad universe for it was a bad universe for Celestia too)))

As I first formulated the question in my mind, it was to maximize humanity's CEV, but without the pony and friendship restrictions, but I care about animals a lot more than the average human, so a universe run by all humanity's CEV could be bad by my standards. Also, if I had a different exchange rate between good things and bad things, we might disagree on where to draw the line on what was a universe worth creating, which would be important if using inflation to create universes without precise control of what happened in them was possible. I think humanity's CEV would probably care more about animals than humanity does, or it might restrict animal suffering just on behalf of a few people who did care about them, but I'm not nearly sure.

Hopefully she would tell me how to change her into an AI that would serve my CEV, but there might be no possible way to do that.

Hmm, actually if she could self-modify into something that was precommitted to torturing everyone she could get her hooves on or create until the heat death of the universe unless I gave up my attempt to control her, and THEN answer the question she might get my CEV to do exactly as she said... unless I precommitted too. But she probably wouldn't have a super-accurate simulation of me, so we would be betting on uncertain guesses, she on how I responded to blackmail on I on how she would guess. I wonder which of us would value the other's chosen universe more. Does she value human satisfaction that is not a result of ponies and friendship? In my world there might still be pony/friendship satisfaction (if people wanted it). What are the chances, in her world, of creating universes where there is a whole lot of what I would consider mildly bad, and what she would consider mildly good (maybe not a lot because her idea of "good" is very specific (it mentions both ponies and humans))? But does she put negative value on even human suffering at all, when that is not caused by ponies? I bet her creator would have written that into her. But I doubt her creator cares about less-intelligent animals as much as I do, or programmed anything in about them.

I've been refreshing this a bit too much in the wait for the next 2 chapters.

Spelling: zeebra (more tk depending on how I like it)

Some painful over explaining; most of Hanna's opening scene is maid-and-butler dialogue.

"The box was white with a detailed charcoal sketch of Fluttershy being cute." - no way. The box looks exactly like the show, because it's on store shelves and they have a strong visual brand.

"less likely than the probability of" is redundant.

ROFL (in a good way) at the scene between David and Celestia.

"brok"

"fillet mignon" should be "filet mignon" (also, that's not the right cut for stew).

Celestia is a bit repetitive with the "friendship and ponies" thing; she should only say it when her interlocutor is at least a bit confused.

"Ligth"

ROFL (again in a good way) at the badges.

"Space was a set of cells, each with eight neighboring cells," <-- do you really mean that equestria is 2d? Because cubes have 6 (or 26) neighbors and octahedra don't tessellate.

"mater"

"just frustrated that" should be "just frustrated enough that"

"up up down down" -cute

Pretty sweet. Not bad for a utopia story,

Hi man. I am new to LW and Utilitarianism in general, and MLP are like ugggggggggh to me, but I read the story and

Loved it.

It is original and a great mix of humor and hard SF. Good work with the editing, the polishing is showing through!

First: For the formal release, I hope you're planning to follow the show and HPMoR in serializing it and posting just one chapter a week at an identical time. You could lock the Google Docs copy.

Posting some early and/or superficial thoughts. TW for sarcasm and ableism.

Is the title negotiable? I like the ring of "AI Ponies: Magic is Optimization".

The summary seems important, since you have to stand out in an ocean of bad fics (even featured ones) and pull a prospective reader from zero to nonzero involvement.

Hanna, the CEO

Doesn't a CEO merit a surname?

of Hofvarpnir Studios, just won the contract to write the official My Little Pony MMO.

Why are they called Hofvarpnir if they had nothing to do with ponies before?

She had better hurry; a US military contractor is developing weapons based on her artificial intelligence technology, which just may destroy the world.

Be Hanna. Singlehandedly create world-destroying technology while grunts are busy making licensed crap for corporate overlords. Give technology to military contractor. Crap, you just gave world-destroying technology to a military contractor! What do? Hasbro awards MMO contract. Yes! With this MMO contract you can save the world!

I know it has to be concise, but it seems a little disjointed. Also, how does "AI technology" differ from an AI itself just waiting to be run? What kind of weapons do you develop from that?

Hanna has built an A.I. Princess Celestia

So disregard the last sentence, the race is already won?

and given her one basic drive: to satisfy values

Her values are to satisfy values?

through friendship and ponies.

Is Hanna an autistic savant of some kind?

I'm glad other people have talked about punching up the prologue; it was quite the slow start when I looked at it earlier. I may or may not comment more later.

I doubt I could give constructive advice, so I think I'll wait until you're done editing and read the completed version. Are you going to put another post on here when you finish?

He's really wondering whether the voxel-space is a directed graph or whether up∘down=down∘up=identity (and for left/right too). Movement could be commutative with up∘down≠identity.

Consider

voxels = {a, b}

left(a)  = a
right(a) = a
up(a)    = a
down(a)  = a

left(b)  = a
right(b) = a
up(b)    = a
down(b)  = a

If f is in (left, right, up, down)
let g be the respective function in (right, left, down, up)
forall x in {a, b}
f(g(x))=g(f(x))=a

But
up(down(b))=a
whereas
identity(b)=b

Unable to access. Getting a docs not found error.

I have stuff to do right now, but I'm hoping to come back soon and post my thoughts so far. The blurb and the first few paragraphs make me cringe at the thought of this being posted more widely in its present state.

He's really wondering whether the voxel-space is a directed graph or whether up∘down=down∘up=identity (and for left/right too). Movement could be commutative with up∘down≠identity.

Consider

voxels = {a, b}

left(a)  = a
right(a) = a
up(a)    = a
down(a)  = a

left(b)  = a
right(b) = a
up(b)    = a
down(b)  = a

If f is in (left, right, up, down)
let g be the respective function in (right, left, down, up)
forall x in {a, b}
f(g(x))=g(f(x))=a

But
up(down(b))=a
whereas
identity(b)=b