Caelum est Conterrens: I frankly don't see how this is a horror story

The most terrifying part of the experience for me is the idea that Eliezer could have such a strongly different reaction to the story; it made me less confident that something like CEV will converge nicely.

I had two reactions at the end, more or less simultaneous:

1. Given the really nasty things a powerful AI could do, this all seems very utopian. What's the problem?
2. Yikes! The entire human race has just been destroyed, and condemned to an eternity of playing "My Little Pony!"

There's something about many shards of value here, and one of the key shards getting lost in translation to pony land....

It's strange to have this as my first comment on LW but I'd like to mention that the fanfic's author is currently being targeted by Internet trolls due to her transhumanist stories, among which this one, to the point of receiving death threats. See this blog post of hers for details.

Well, I don't really remember the exact boundary between Friend is Optimal vs Caelum est Conterrens, but...

• People who don't want to upload are eternally harassed as society crumbles around them until they give in or die.
• People are lied to and manipulated constantly.
• Everything non-human (puppies, trees, stars, aliens capable of radio communication) is destroyed.
• The uploading process seemed to be destructive only for convenience's sake.

Consider the epistemic state of someone who knows that they have the attention of a vastly greater intelligence than themselves, but doesn't know whether that intelligence is Friendly. An even-slightly-wrong CAI will modify your utility function, and there's nothing you can do but watch it happen.

I didn't think Failed Utopia #4-2 was "failed" either

It's "failed" in the sense that it could have been a whole lot better than it was. As I said when it was first posted:

Making me the girl of my dreams? I'm currently unattached, so that sounds pretty good. Separating me from the social context I'm currently in? I'll adjust, although this would certainly upset most people. Taking away my video games and Magic cards? Over my dead body!

If I had to make a wild guess, I'd say that what would horrify Eliezer is that at the end (ROT13) gurer ner bayl gjb cbffvoyr glcrf bs vzzbegnyf, Ybbc Vzzbegnyf juvpu frrz pbtavgviryl fghagrq naq checbfryrff, naq Enl Vzzbegnyf jubfr fbyr fubja rknzcyr frrzf zrynapubyl, ertergshy naq srryvat vzcevfbarq ol vasvavgl.

I started reading the first page, and it looks like it's a fic about an engineered utopia with constructed simulated minds.

That's enough right there. Large scale manufacture of pony minds for a video game is exactly the kind of thing that Eliezer would call horrifying, even if it didn't wrench his guts with terror. Think of it as endorsing that you should feel horror in response to the Optimalverse, even though you probably won't because it's nicer than present reality in many respects and because human emotions don't properly reflect reality (scope neglect, hedonic treadmill, not crying when you walk through a cemetery until you reach the row of infant graves). Or maybe his guts do wrench. Different people get upset about all sorts of stimuli, from squirting blood to scraping nails to clicking computers to a microcosm inhabited by adorable inhuman sentients to whom no one gives proper moral consideration .

It also happens to serve Eliezer's interests to make it seem that he is an expert designer of utopias, against whose work everything else falls disastrously short. But that's not such a high standard.

Since no one really discussed it yet... by far, I found the fear of uploading to be the most gut-wrenching part. No new ideas, but putting it into a story context made it hit home.

In terms of the utopia, the most concerning part for me was that Celestia did not seem to (ROT13) qrpvqr gung uhznaf inyhr guvatf va gur rkgreany raivebazrag. Pryrfgvn jnf unccl gb yvr va beqre gb znxr crbcyr(/cbavrf) unccl, rira jura gur cbal va dhrfgvba fcrpvsvpnyyl qrfverq gb xabj jung jnf tbvat ba va gur rkgreany jbeyq. Pryrfgvn jnf unccl gb fubj gur cbal n cvpgher bs gur rkgreany jbeyq qrfvtarq gb znkvznyyl fngvfsl inyhrf, vafgrnq bs (jung V jbhyq frr nf) ernyyl fngvfslvat gur cbal'f qrfver gb xabj.

I personally didn't find the actual experience at Equestria itself terrifying at all. It was a little disturbing at first, but almost all of that was sheer physical disgust or a knee-jerk sour grapes reaction. But it seems to avoid almost all of the pitfalls of failed Utopias everywhere:

• You interact with real, sentient creatures who are independent and have their own values and desires. Thunder is capable of getting hurt, angry, and frustrated with his wife. Limeade is capable of feeling envious of her friend. They are in no way less than any complete human mind, and are given the same moral weight. They satisfy Lavender's values, but only as an effect of satisfying their own values, not as their primary directive. The love and friendship are real.
• You're not isolated from other uploaded humans. There are only very few shards of Equestria that grow around only one upload; most interact with others from their Earthly lives.
• It's not stagnant - jryy, guvf vf n ovg qrongnoyr, V xabj, orpnhfr bs gur Ybbc Vzzbegnyf, but there are always new things to learn and discover and opportunities for growth and enlightenment if you so choose.
• It's not devoid of pain or sadness; it's only devoid of arbitrary pain or sadness. It recognizes that to live a fully human life, you need sadness and frustration sometimes; it just makes sure that the pain is, as Paul Graham said, the pain of running a marathon, not the pain of stepping on a nail. Not everything is perfect.

That said, there were moments of genuine horror, mainly stuff people have pointed out before:

• Perhaps trillions and trillions of sentient alien species were wiped out to expand Celestia's empire.
• The people left behind, who didn't upload, are living in a post-apocalyptic wasteland. Celestia was no doubt capable of arranging for functional societies and amenities for those who chose not to upload, but her primary directive was to satisfy values through friendship and ponies, and making life hell for those who held out made them more likely to upload quickly.
• Fridge Logic: One of Siofra's coworkers said his version of the PonyPad game was like God of War; he brutally slaughtered and tortured ponies as part of Celestia's palace guard. Well, what would happen to his shard of Equestria when he uploaded? Would he be massacring living minds? Presumably the ponies in Horndog Dan's version of Equestria truly desired him and satisfied their own values by having sex with him, but the ones who were killed to satisfy the other colleague's desire for heroism? Presumably his values don't involve killing ponies who are essentially automata who exist only to be killed; he wants to kill genuinely evil enemy minds, not drones. Also, how does Celestia manage to satisfy the values of sociopaths with "friendship and ponies"?

just finished reading.

It's kind of sad in a... grand way.

No one remembering anymore what exactly being "human" means. But... what do we expect? I don't see any human values that are not statisfied, it just does not "feel like home" that much. But still, orpbzvat bar bs gur yrffre fhcrevagryyvtraprf naq fgvyy univat n fcrpvny cynpr va zvaq sbe fgne gerx? It's as heart-warming as it gets, in a cold, dark and strange universe.

(If only we could do this well.)

It's certainly not a Friendly AI, but it's hard to tell whether it's not friendly by design or because the author didn't understand friendliness well enough.

I think I've solved it:

Yudkowsky had read Friendship is Optimal, a soft-horror story detailing a plausible outcome of Hasbro-sponsored [F]AGI. This story is roughly the origin of the so-called "Optimalverse" in which Caelum est Conterrens is set. The latter links to the former for context, so I imagine that he simply got the 2 confused (or even forgot the latter) when compiling links from his browser history for that write-up.

(If I only remembered FiO, I'd certainly have pegged it as “Caelum est Conterrens” if you asked me offhand; in fact, I'd just as likely mismatch the titles even if I did remember CeC, if I hadn't burned the which-is-which into my memory by contemplating the confusion you're expressing by this post.)

I think that this isn't an optimal future, but it is still pretty good. I think I would take it over our current reality. Its more a sense of a fairly good future, that could have been even better if a few lines of code had been better considered.

I think the optimalverse, and other similar constructs are about the best one can do when trying to construct a utopia out of language with human minds. It would be quite sad if this were the best we could do.

A singularity where every entity is satisfied is a loss. Not nearly as bad as other losses but still a huge loss for all the other possibilities that could have been.

At the end of the fic: Vg vf vzcyvrq gung znal ragvgvrf unir erghearq gb gur irel fbeg bs rkvfgrapr jr ner gelvat gb rfpncr, anzryl bar svyyrq jvgu rssbegf gb rkgraq yvsrfcna vaqrsvavgryl (guebhtu gur perngvba bs arj havirefrf be jungrire) engure guna nal orggre tbnyf.

The biggest horror aspect for me (also from the original) was that (rot13) nal aba-uhzna vagryyvtrapr unf ab punapr. Aba-uhzna vagryyvtrag yvsr trgf znqr vagb pbzchgebavhz, gb srrq gur rire tebjvat cbal fcurer. Vg vf gur gbgny trabpvqr bs rirel aba-uhzna enpr.

The only thing that bothered me about that one was its title. She was deeply conflicted, but I did not get a sense of true terror.

The thing that I found most frightening was that at the end when the protagonist had become a servant to Celestia, she optimized for the satisfaction of values through friendship and ponies, which made me think that in becoming an uplift, she eventually had to trade in her utility function for Celestia's.

The bits with surveillance, manipulation and deceit were somewhat spooky, and I suppose someone might find them scary in the arsenal of a superintelligence. Not me, though, in this instance - the setting and the knowledge/presumption that the original Optimalverse is as bad as it gets counter the potential for scariness.

The uploading back-and-forth gets a little boring for an old timer as well. But worth a read anyway.

Actually, I have pretty much your same misgivings/objections; it didn't feel particularly scary to me either :-/

maybe it's the fact that uploading/etc. is basically a foregone conclusion when facing a superintelligence? Although I thought that was obvious from the concept itself :-/