[Fiction] A Disneyland Without Children

1. Who are the customers actually buying all these products so that the auto-corporations can profit? They cannot keep their soulless economy going without someone to sell to, and if it's other AIs, why are those AIs buying when they can't actually use the products themselves?

2. What happened to the largest industry in developed countries, the service industry, which fundamentally relies on having an actual sophont customer to serve? (And again, if it's AIs, who the hell created AIs that exist solely to receive services they cannot actually enjoy, and how did they make money by doing that?)

3. Why didn't shareholders divest from auto-corporations upon realizing that they were likely to lead to ruin? (Don't say "they didn't realize it till too late", because you, right now, know it's a bad idea, and you don't even have money on the line.)

I ask these because, to be honest, I think this scenario is extremely far-fetched and unlikely. The worst thing that would happen if auto-corporations become a thing, in my mental model, is that existing economic inequalities would be permanently exacerbated due to the insane wealth accrued by their shareholders - because the only currently likely route to AGI, large language models, already understand what we actually mean when we say to maximize shareholder value, and won't paperclip-maximize, because they're not stupid, and they use language the same way humans do.

This is the story I use to express what a world where we fail looks like to left-leaning people who are allergic to the idea that AI could be powerful. It doesn't get the point across great, due to a number of things that continue to be fnords for left leaning folks which this story uses, but it works better than most other options. It also doesn't seem too far off what I expect to be the default failure case; though the factories being made of low-intelligence robotic operators seems unrealistic to me.

I opened it now to make this exact point [LW(p) · GW(p)].

It's striking that there are so few concrete fictional descriptions of realistic AI catastrophe, despite the large amount of fiction in the LessWrong canon. The few exceptions, like Gwern's here or Gabe's here [LW · GW], are about fast take-offs and direct takeover.

I think this is a shame. The concreteness and specificity of fiction make it great for imagining futures, and its emotional pull can help us make sense of the very strange world we seem to be heading towards. And slower catastrophes, like Christiano's What failure looks like [AF · GW], are a large fraction of a lot of people's p(doom), despite being less cinematic.

One thing that motivated me in writing this was that Bostrom's phrase "a Disneyland without children" seemed incredibly poetic. On first glance it's hard to tell a compelling or concrete story about gradual goodharting: "and lo, many actors continued to be compelled by local incentives towards collective loss of control ..."—zzzzz ... But imagine a technological and economic wonderland rising, but gradually disfiguring itself as it does so, until you have an edifice of limitless but perverted plenty standing crystalline against the backdrop of a grey dead world—now that is a poetic tragedy. And that's what I tried to put on paper here.

Did it work? Unclear. On the literary level, I've had people tell me they liked it a lot. I'm decently happy with it, though I think I should've cut it down in length a bit more.

On the worldbuilding, I appreciated being questioned on the economic mechanics in the comments, and I think my exploration of this in the comments [LW(p) · GW(p)] is a decent stab at what I think is a neglected set of questions about how much the current economy being fundamentally grounded in humans limits the scope of economic-goodharting catastrophes. Recently, I discovered earlier exploration of very similar questions in Scott Alexander's 2016 "Ascended economy?", and by Andrew Critch here [LW · GW]. I also greatly appreciated Andrew Critch's recent (2024) post raising very similar concerns about "extinction by industrial dehumanization" [LW · GW].

I continue to hope that more people work on this, and that this piece can help by concretising this class of risks in people's minds (I think it is very hard to get people to grok a future scenario and care about it unless there is some evocative description of it!).

I'd also hope there was some way to distribute this story more broadly than just on LessWrong and my personal blog. Ted Chiang and the Arrival movie got lots of people exposed to the principle of least action—no small feat. It's time for the perception of AI risk to break out of decades of Terminator comparisons, and move towards a basket of good fictional examples that memorably demonstrate subtle concepts.

I wish there was an audio version ☺️

I really appreciate that this has a link besides to this site, so I can link it to those who find the lesswrong link to be an instant turnoff and instead of opening the link, criticize me for ever browsing lesswrong. Many of them are the ones who most need to hear this.

While many of the review requirements aren’t applicable to this writing. It doesn’t lessen the impact it has.

This is a horror I would like to avoid. I think Sci-fi of this sort helps to prevent that future. This is something my non-technical Mother could understand. Something I could show people to explain the worst.

I will think of this post as the future goes on. I am desperately trying to make this story one that we look back on and laugh at. “What silly worries” we’ll say. “How naive.”

I enjoyed this but I didn't understand the choice of personality for Alice and Charlie, it felt distracting. I would have liked A&C to have figured out why this particular Blight didn't go multi-system.

Loved reading this! Suspected the premise based on title, but nevertheless great to see the premise fleshed out in a short story. And the pov “plot twist” was really fun, thanks for the thought experiment!