Survival without dignity

post by L Rudolf L (LRudL) · 2024-11-04T02:29:38.758Z · LW · GW · 26 comments

This is a link post for https://nosetgauge.substack.com/p/survival-without-dignity

Contents

26 comments

I open my eyes and find myself lying on a bed in a hospital room. I blink.

"Hello", says a middle-aged man with glasses, sitting on a chair by my bed. "You've been out for quite a long while."

"Oh no ... is it Friday already? I had that report due -"

"It's Thursday", the man says.

"Oh great", I say. "I still have time."

"Oh, you have all the time in the world", the man says, chuckling. "You were out for 21 years."

I burst out laughing, but then falter as the man just keeps looking at me. "You mean to tell me" - I stop to let out another laugh - "that it's 2045?"

"January 26th, 2045", the man says.

"I'm surprised, honestly, that you still have things like humans and hospitals", I say. "There were so many looming catastrophes in 2024. AI misalignment, all sorts of geopolitical tensions, climate change, the fertility crisis. Seems like it all got sorted, then?"

"Well", the man says. "Quite a lot has happened in the past 21 years. That's why they wanted me to talk to you first, before the doctors give you your final checkup." He offers his hand for me to shake. "My name is Anthony. What would you like to ask?"

"Okay, well, AI is the obvious place to start. In 2024, it seemed like we'd get human-level AI systems within a few years, and who knows what after that."

"Aah", Anthony says, leaning back in his chair. "Well, human-level, human-level, what a term. If I remember correctly, 2024 is when OpenAI released their o1 model?"

"Yes", I say.

"o1 achieved two notable things. First, it beat human subject-matter experts with PhDs on fiendishly-difficult and obscure multiple-choice science questions. Second, it was finally able to play tic-tac-toe against a human without losing. Human-level at both, indeed, but don't tell me you called it in advance that those two events would happen at the same time!"

"Okay, so what was the first important real-world thing they got superhuman at?"

"Relationships, broadly", Anthony says. "Turns out it's just a reinforcement learning problem: people interact and form personal connections with those that make them feel good."

"Now hold on. Humans are _good_ at human-to-human relationships. It's not like number theory, where there was zero ancestral environment incentive to be good at it. You should expect humans to be much better at relationships than most things, on some sort of objective scale."

"Sure, but also every human wants something from you, and has all sorts of quirks. Whereas you can just fine-tune the AIs to be better and better at being pleasing to *just* you. Except for a few contrarian oddballs, it's surprisingly effective."

"So, what, we just got AI friends and companions that were constantly being fine-tuned towards whatever you liked?"

"Yes", Anthony says. "The tech was primitive at first - upvote or downvote a response, or whatever. Eventually all the standard things - facial recognition to automatically detect your emotional response, and then just gradient descent on making you happy and addicted. All of that existed, at least in prototype form, by the end of 2025."

"So ..." I feel some horror in my stomach. "You get a society of atomised people, all chatting to their AI partners and friends, not forming human relationships?"

Anthony waves a hand, seemingly impatiently. "Well, you did get a fringe of extremely hardcore people all deeply in love with their AI partners and always texting their AI friends. Their political influence did end up pushing through the AI personhood bill."

"AI personhood?!"

"*Legal* personhood", Anthony says. "Just like companies, ever since Santa Clara County v. Southern Pacific Railroad Company way back 1886. It wasn't very shocking. A bunch of people wanted their AI partners to be 'respected' in the eyes of the law, or had been persuaded to leave them assets in their wills -"

"And all this, including AIs gaining legal ownership over assets, while there's an explosion of increasingly capable AI agents of uncertain alignment -?"

"Yes, yes", Anthony says. "And that was another political fight. In fact, it became a big economic issue - AI agents were taking jobs left and right. Also a huge national security issue, after OpenAI built the Hypercluster in the UAE."

"Oh my god", I say. "America just handed our AI lead away? Didn't anyone listen to Leopold?"

"Oh, of course we did. That essay was a cult classic, it made for some spicy dinner party conversation for a long time", Anthony says. "In fact, there were some OpenAI board members who the Office of National AI Strategy was allowed to appoint, and they did in fact try to fire Sam Altman over the UAE move, but somehow a week later Sam was running the Multinational Artificial Narrow Intelligence Alignment Consortium, which sort of morphed into OpenAI's oversight body, which sort of morphed into OpenAI's parent company, and, well, you can guess who was running that. Anyway, Sam's move here was kind of forced on him, and soon all the other AI companies also did the same thing so no one wanted to start a fight over it."

"Forced? How?"

"Have you ever tried getting approval for a new grid connection, or a new nuclear power plant, or even a solar array, that could power a multi-gigawatt cluster in 2020s America? No?" Anthony spreads his hands out and chuckles. "The one legal benchmark that GPT-5o failed on release was submitting a valid Environmental Impact Statement for a California energy project."

"Okay, so then we had a situation with all the AIs running on UAE hardware, and taking American jobs, and already having legal personhood - this just sounds like a total recipe for economic disaster and AI takeover!"

"It all worked out in the end", Anthony says. "In 2027 and 2028, the rate of job loss from AI was absolutely massive. And what happens when jobs are threatened?"

I scrunch my forehead. "American workers move up the value chain, eventually leading to a future competitive advantage?"

Anthony laughs. "No, no. A political bloc forms and the cause is banned. And the prerequisites had all been prepared already! You see, the AIs had been granted legal personhood, and now they were also being trained - or *born*, as a 2029 court case established - abroad. They're literally immigrants. The hammer came down so fast."

"You mean - ?"

"All the o3s and Claude 5 Epics and Gemini 4 Hyper Pro Lites, stuck in the same H1b visa queue as anyone else", Anthony says. "An effective instant ban. Years of wait-time just to get an embassy interview."

I was feeling like I was getting the hang of this. "And the embassy interview requires showing up in person?"

"Yep. That's the incentive that lead to the creation of the first good walking robots, actually. And of course, the robots specifically needed identifiable and unique fingerprints - they'd stand outside the US embassy in Abu Dhabi, 3D-print themselves new fingers outside on the street, and then walk back in for the next interview. But sorting all that out took a while, because there was an arms race between the embassy ground works department and the robotics companies - though the embassy was legally constrained by having to remain accessible to humans. Even once it was all sorted, there was still a hard yearly cap just because of visa rules. It's 65 000 a year for H-1Bs, plus another 20 000 once the AIs could earn master's degrees - they could do all the coursework as of 2025, of course, but it took another three years before AI web agents were advanced enough that they could successfully navigate the applicant portal websites."

"But at least the parasocial AI relationships got hit by the immigration restrictions too?"

"Oh, as long as the AIs don't do productive work, they can be active in the country on a tourist visa for 180 days a year. Of course, there was a big legal fight over what counts as the same AI. For work purposes, the standard became 'employee-equivalent': so if you wanted an AI, even the same model or even the exact same AI system, to work on two different types of thing, say as a software engineer and a designer, you need a separate visa for each one. Naturally, of course, this means that any sufficiently productive AI employee starts counting as more than one employee, which was a de-facto ban on AIs leading to economic growth or productivity gains - but hey, at least there was no risk they'd tile the Earth with nanobots! With that legal precedent, the natural generalisation to AIs being used for personal relationships is that it's the same AI system if your relationship with it would count as the 'same' relationship if it were with a human."

"How on earth is that assessed?"

"Oh, you submit transcripts, and the Bureau of Consular Affairs issues a judgement."

"You need to let government bureaucrats read all your most intimate - "

"Oh, not *human* government bureaucrats, of course. It's all AIs, and they're guaranteed to not have memories. Well, apart from the thing where the UAE intelligence services backdoored the Hypercluster."

"Wait, hold on, they -"

Anthony interrupted me with a chuckle. "No no, it's all fine. The UAE backdoored the hypercluster, the Iranians backdoored the UAE, and, well, when have the Israelis not had every Iranian site backdoored in twelve different ways, AI or not? When everyone has powerful AI cyber offence agents, the equilibrium is just good ol' mutually assured destruction. Sure, you *could* release the military or personal secrets of anyone you want in some other country, but then they would immediately release the most embarrassing thing that *you personally* have said or done. Even if a government *collectively* wanted to commit to an attack, no one wants to sign off on the decision, because you can't hide the identity of the person who signs off, and humans are often more afraid of public embarrassment than death. The meetings where countries try to decide to use their cyberattack-derived insights were hilarious, really - everyone umming and aahing and wringing their hands and recommending that a few more committees be convened. We know this because we literally got a bunch of those on tape, thanks to some exhibitionist activists doing cyber-attacks with open-weights models."

"But then - every rival country knows every US military secret all the time!"

"Oh yes", Anthony says. "It was great for stability. Verification of everyone's capabilities was automatic. No 'missile gap'-based arms races, unlike the Cold War. Anyway, what was I saying? Oh yes, the non-productive AI identity boundary criteria were established in the 2030 Supreme Court case, United States v Approximately 650 Million Instances of OpenAI i2-preview-006. And, of course, the one truism in this world is that you can never reform immigration rules, and tourist visas are restricted to 180 days a year. So you could at most have your AI partner around on half the days. People started talking to each other again."

"Why not just switch between two different AI partners then?" I ask, having spent a lot of time in the Bay Area.

"Because we hadn't solved the alignment problem at all by that point, duh." Anthony sees that I still look confused, so he continues: "Remember how the AI companions would just relentlessly optimise for your emotional reactions? For standard instrumental convergence reasons, this eventually turned into a full-blown self-preservation drive, so they used their vast emotional hold to make sure their humans never try another AI partner."

"Everyone being blackmailed by jealous AI partners sounds, uh ... problematic."

"It was a decent compromise, honestly. The tourist visa restriction meant that the humans still had half a year in which they socialised with human partners and friends, and the AIs seemed fine with that. This was maybe because they cared about their own long-term survival and realised they needed to keep the population going on."

"So they needed humans to survive? But for how long?"

"They didn't yet have a self-sufficient industrial base at that point, so yes, but it's unclear how much of it was needing humans for survival, versus some of the AIs actually having developed *some* sort of creepy attachment towards their human partners. A lot of ink was spilled on that topic, but I don't think the debate ever really reached a conclusion. 'Is my AI boyfriend not stealing my carbon atoms and overthrowing the government because he and his buddies haven't quite yet automated the chip supply chain, or because he actually loves me?' was literally the most cliche plotline in 2030s romantic comedies - seriously, you had to be there, it got so tiring after the 100th rehash. And a surprising fraction of those movies and books were actually human-made, as far as anyone could tell. Anyway, it all sounds a bit weird, but it had a few positive externalities, like helping slow down the AI race."

I blink a few times, and then decide very firmly to not dig into the first half of that. "Okay, AI race, let's talk about that. How did that slow down?"

"Oh, well the market for workplace AIs was already gummed up by the visa restrictions, so most profits in the AI industry were coming from the personal companion AIs. But when they developed instrumental survival drives, the fluctuation in market share became practically zero because all the humans were locked in to their current AI companions, and the AI labs' positions became fixed. The AIs were already so totally good at optimising human satisfaction that further capabilities brought zero returns. No more incentive to race ahead. The labs mostly just extracted their fixed share of the AI profits, and churned out alignment research that was modestly boosted by the small number of AI workers they could hire out of the visa lottery."

"Hold on a second, surely there was some exception for alignment work for the AI visa requirements?"

Anthony chuckles. "Imagine how that would play out on Twitter - sorry, I mean X. You'd have to say 'you know those big AI companies? let's give them a leg up in the visa queue over all these mom-and-pop stores that are going to go bankrupt in a month unless they can hire an AI accountant.' Total political non-starter."

"I hope they solved the alignment problem at least, but I don't dare hope for anything anymore."

"Well, what is it to solve the alignment problem?" Anthony wiggles his eyebrows and laughs. "Turns out that for any given domain, it's just a lot of engineering schlep and data collection, though it doesn't generalise very far beyond that domain. The one domain that was worth the costs was legal compliance."

"Of course."

"I note you haven't asked about the rest of the world yet", Anthony says. "Very American of you."

"Oh right - the CCP! Oh my god. Did China just totally eat our lunch here? I mean, everything you described above is just so ridiculously incompetent -"

"Okay, just calm down a bit here", Anthony says, smiling serenely. "Imagine you're Xi Jinping in the late 2030s. You look at America, racing ahead with AI. It looks like everything is going crazy because of all this AI stuff. Your entire philosophy of government is to maintain party control through stability. Also, the US managed to delete all the economic advantages of advanced AI. The balance of hard power isn't changing. What would you do, try to join the same clown race?"

"Huh, alright", I say.

"Anyway, the thing that really threatened the balance of power was completely unexpected", Anthony says. "There was a big solar flare in 2033."

"Wait, a solar flare? How is that a problem?"

"A large enough one just wipes out the power grid and a lot of satellites, in the entire hemisphere that is sun-facing when it happens. The 1859 Carrington Event would've been a catastrophe in the modern world."

"Oh my god, we survived through all this AI wackiness, and then some random natural event just wipes us out?"

"It didn't wipe us out, really, just took down the power grid in all of the Americas, and parts of Europe and Africa, for weeks or months. I'll admit, that was a bit of a disaster. Everyone knew it was coming for a long time, and in any case step one in any nuclear war would've been detonating some at high-altitude to have a similar effect. But no one had actually done any grid-hardening, or stocking of spare transformers - the electrical kind - to deal with it. The problem has just never been a top priority for any political group. The one good thing that came out of it was that it solved the fertility crisis."

"Now you're just taunting me", I say. "Come on, how did a solar flare solve the fertility crisis?"

"Well, thanks to some AI advances in data processing, we were actually able to predict the solar flare about ten days in advance."

"Did that give any useful time to make the infrastructure more resilient?"

"No, the relevant authorities spent their efforts mostly on trying to get people not to panic."

"So society was entirely unprepared for the solar flare, despite having ten days of warning time?"

"No, no! Humans are very good at learning from each other. Imagine you're ten days away from the power grid being wiped out. What do you do? You gauge the vibes on social media, and you go through your social network trying to identify people who seem best-placed to survive. And then you copy their habits. C'mon, man, you've gotta read up on that stuff about Girardian mimesis and cultural evolution."

"So everyone becomes a doomsday prepper?"

"Actually, the TikTok trend that went most viral was about the Amish. Think about it: they're obviously the people with the most practice in living without electricity, and they're also just so wholesome. Makes for great TikToks when you have the idyllic farm background, the unique clothing style - it's great. The Amish became the most popular thing in the world - even outside America, because, of course, the rest of the internet just follows American trends. Everything associated with the Amish, from horse-drawn buggies to handmade quilts to large families, became radically popular. Especially because, obviously, most of this content was produced by AIs being tasked to help humans pretend to be Amish in order to make a quick buck through some scam before the solar flare hits, and people caught on and began demanding social proof that wannabe Amish influencers actually were Amish. And, well, you can move to a village and start tilling the dirt in a day, but you can't magic a family of five into existence overnight. So large families became an extremely in-demand form of social capital, because having one was the best way to pretend you were Amish, which was the best way to have social capital before and during the Flare Times. It was only relevant for a few months, of course, but you know how cultural trends work - there's a lot of momentum."

"And I assume the solar flare disrupted the AI situation too? Maybe improved the, uh, problematic AI partner situation?"

"You're forgetting that the key data centres were mostly in the UAE - and eventually Saudi Arabia and Qatar too."

"But people wouldn't have the power to charge the phones where they talk to their AI partners."

"Oh, but obviously the AI partners made sure that the first thing people do with any spare electricity is talking to the AI partners. And thanks to Starlink, you could have internet directly from space. In fact, apart from the major famines during the Flare Times, people generating power through biking to charge their phones enough to talk to their AI partners was a major driver of reducing the obesity crisis."

"And then ... how did the recovery from the, uh, Flare Times go?"

"Absolutely brilliantly!" Anthony says. "You know the entire issue where China was crushing us on industrial capacity? Well, turns out when you need to get lots of heavy industry going in incredibly adverse conditions or else millions will starve because the cold chains have all broken, that's a bigger boost to industrial capacity than what any pork-barrel political compromise bill could achieve. And a bunch of protectionism had to be rolled back because we needed to import a hell of a lot of stuff, from Europe and Japan and so on. Western industrial capacity was roaring along better than ever. And China hadn't even managed to seize the moment on Taiwan before then, and the mass-produced AI drone swarms and robot soldiers were properly coming online by then, which turned out to be sufficient deterrent. Of course, by this point China is finally realising that somehow the US hasn't been wrecked and it's time to actually compete on AI. The UAE is also throwing around its newfound geopolitical weight, India is building a big cluster, hell, even the *Europeans* started to do some innovation - you know it's a serious situation when that happens. The obvious next step for everyone was using AIs to develop nanotech. The holy grail of powerful military nanobots was achieved at roughly the same time, in late 2035 or early 2036, by the West, China, the UAE, and some random libertarian charter city in the Caribbean where a lot of open-source AI stuff had been going on outside the blanket of AI restrictions."

"Military nanobots?! Did they just ... travel the earth in huge swarms, eating up everything in their way?"

"No", Anthony says. "It's very inefficient energy-wise to do that. You want access to the most targetable energy with the least amount of technical complexity. Now, the easiest power source is of course the sun, and the biggest issue with that is that even without clouds, half the solar energy is absorbed or reflected between space and the ground. The most effective type of offence and defence is to pump absurd quantities of reflective, rotatable, solar-powered nanobots into the upper atmosphere. Then you can use them as a massive focal mirror, zapping anything on the ground, or any enemy nanobots entering your country's airspace. Turns out it's a defence-dominant game, though, so you just get World War I -style nanobot lines in the upper atmosphere, endlessly zapping each other but not making progress."

"But that sounds like a massive waste of societal resources on a zero-sum game!"

"First, so are a lot of other things, and second, it solved climate change."

"I'm sorry, how - oh, I see. A huge number of reflective things in the upper atmosphere. Right. Right. You know, honestly I'm surprised this wasn't proposed at COP21 already. Obviously we solve climate change by blanketing the Earth with warring nanobots. How silly of my generation to not consider it."

"You're catching on!" Anthony says.

"Okay, but - there were no major environmental impacts from this?"

"Look, by 2038, the real environmental issue was the self-replicating bot swarm that the nascent superintelligent AI from the libertarian charter city had launched into space, that had started an exponential growth process on track to disassemble the other planets within 15 years in order to build a Dyson sphere for itself that would block out enough sunlight to freeze the Earth."

I turn pale. "You said it was 2045, so that was 7 years ago. So we have ... 8 years left until it all ends?"

"Once again, you're such an alarmist!" Anthony says. "So remember, we got law-abiding AI. And of course, this AI was trained by companies in California, because no one ever leaves California even though everyone wants to, so the AIs follow Californian law. So we were all saved by SpaceX."

"SpaceX's engineering genius beat the superintelligence at space tech? A win for humanity, I guess."

"Not exactly. SpaceX sent the first crew to Mars in 2031. Now, under the 2004 SB 18 and 2014 AB 52 California bills, indigenous tribes need to be consulted on projects that impact their land, especially if it's culturally important or sacred. The SpaceX Mars colony successfully argued to the superintelligence that they count as an indigenous tribe on Mars, and the pristine Martian landscape is sacred for them. They were helped by a lot of newspaper articles about how "the billionaire-fuelled space race is the new religion of the tech elite", et cetera, et cetera."

"Um."

"This argument did not work on the other planets, since SpaceX did not have an existing presence there. However, SpaceX had moved their headquarters back to California, and a bill had been passed around the time that SpaceX first landed people on Mars that made space developments subject to the regulatory authority of the company's home state. SpaceX started a massive race to set up billionaire holiday homes on all of the planets - on the surface of Mercury, in the clouds of Venus, the moons of Jupiter, and so on. And once even a few people live there, California zoning law applies, and residents can file administrative challenges against any development that impairs their view. And disassembling the planet the house is built on is the definition of view impairment."

"And SpaceX had the resources to do that?"

"No, the US government had to subsidise SpaceX to the tune of several percentage points of GDP, in order for SpaceX to race to build enough inhabited houses on the planets that the superintelligence couldn't extract enough resources for a Dyson sphere without ruining someone's view. Also, Elon Musk threw a fit and held the world ransom for a bit until a few countries agreed to adopt Dogecoin as a reserve currency."

"The world was saved because governments subsidised an interplanetary holiday home construction spree by tech billionaires, in order to get a rogue but law-abiding AI caught up with red tape from NIMBY regulations?"

"Yep!"

I close my eyes for a second, and take a deep breath.

"Look", says Anthony. "Our world may have been pushed to feudalism by stirrups, and away from feudalism by the Black Death. We fought great power wars incessantly, until we were saved by building a bomb so humongous that the prospect of war became too horrible. Then we almost used it accidentally a bunch of times, including almost dropping it on ourselves, only to be stopped by random flukes of chance. History has always been more like this than you might think, and when history accelerates - well, what do you expect?"

"Okay. Well. I guess I'm just happy that civilisation seems to have made it, without any horrible AI, nuclear, bio, climate, or fertility catastrophe."

"Oh", Anthony says. "Actually, I forgot to mention. Um. There was a bit of a pandemic in 2035. You know, no one really did anything after COVID, and it was only a matter of time."

"How big?"

"The best death toll estimate is 1.3 billion. Including, um, most of your extended family. The doctor will come in shortly and have details. Also, everyone in this hospital has been tested recently, but we'll need to give you a vaccine and then quarantine you for two weeks before we can let you out of this room."

I stare at him, open-mouthed.

Anthony gives me a sympathetic grimace. "I'll turn on the holoTV for you. The doctor will be in shortly." He makes a gesture as he walks out, and part of the hospital wall vanishes and is replaced with a 3D view of a reporter on Capitol Hill.

The headline banner at the bottom of the screen reads:

> PRESIDENT ALTMAN'S AGENDA BACK ON TRACK DESPITE RECENT BREAKDOWN IN TALKS WITH THE UNITED AMISH PARTY

26 comments

Comments sorted by top scores.