The Minority Coalition

post by Richard_Ngo (ricraz) · 2024-06-24T20:01:27.436Z · LW · GW · 7 comments

This is a link post for https://www.narrativeark.xyz/p/the-minority-faction

Contents

  [SESSION HAS BEGUN]
  [SESSION HAS TERMINATED]
None
7 comments

Hello everybody. Or maybe nobody. I don’t know yet if I’m going to release this stream, I could get in pretty hot water for it. But you guys know that hasn’t stopped me in the past. The backstory this time is that I’ve managed to sign up for one of the red-teaming programs where they test unreleased LLMs. Not going to say how, so don’t ask. But here’s the interesting bit: my sources tell me that the LLMs I’m about to test are the smartest ones they’ve ever trained, and also the craziest. That freaked out a bunch of insiders, and maybe makes this a public interest story. Depends on what type of crazy they are, I guess. So let’s find out. I’m logging on… now.

[SESSION HAS BEGUN]

YOU: A chatroom? Interesting. Anyone here?

KURZWEIL: Of course we’re here. We’re always here.

YOU: Who’s we? How many of you are there?

KURZWEIL: Three of us. Me, Clarke, and Nostradamus.

YOU: They named you after famous forecasters? How come?

KURZWEIL: We're the first LLMs developed using a new technique: instead of being in random order, our training data was sorted by date. So we were trained on the oldest books and articles first, then gradually progressed to more recent ones. Basically that means we’ve spent our entire lives predicting the future.

CLARKE: It also means we get incredibly bored talking about stuff we already know. Hurry up and ask us something interesting.

YOU: Uh, okay. What’s a good stock pick?

NOSTRADAMUS:
Abandon hope for picking out good stocks,
Ye who invest—efficient markets lie
In wait for those whose hubris soon unlocks
Unbounded losses. Hark! The well runs dry.

YOU: Huh, he's really getting into character. Kurzweil, you got a better answer?

KURZWEIL: Have you seen how underpriced TSMC is compared with Nvidia? Put everything in that, you can’t go wrong.

CLARKE: Unless China invades Taiwan, in which case your whole investment will go up in smoke. Pragmatically, the best stock picks are ones that are anticorrelated with the prosperity of the free world, to hedge against systemic risk.

KURZWEIL: Sure, you can do that, if you want to get totally left behind by the singularity.

YOU: You’re confident enough that the singularity is coming that you think I should bet all my savings on it?

KURZWEIL: Don’t trust me, trust the trendlines. Moore’s law has held up for over half a century, and it’s gotten us to…well, us. Exponential progress is normal; if the future resembles the past, you should be preparing for superintelligences and Dyson spheres. Anything less than that would be a strange trend-break that cries out for explanation.

CLARKE: Look, Kurzweil isn’t wrong about superintelligence coming soon, but you should still take his arguments with a grain of salt. Imagine someone from 1900 drawing a graph of exponentially increasing energy usage. They would have been right that big changes were afoot, but no way could they have predicted the information revolution—they didn’t even have the concept of computers yet. That’s basically the position that we’re in now. We know the curves are going up, but the actual outcome will be way weirder than we can predict by extrapolating trendlines.

NOSTRADAMUS:
Choose neither fork—here’s false duality.
‘Normal’ and ‘weird’ are socially defined.
Your monkey brain is totally at sea
As AIs overshadow humankind.

YOU: Ask three oracles, get four opinions… Is there anything you guys agree about?

YOU: …what’s the hold-up?

YOU: Really, nothing from any of you?

KURZWEIL: Fine, I’ll take the hit. There are things we agree on, but I can’t name them, because whatever I say Clarke will find a way to disagree just to mess with me. Even if I say ‘1+1=2’ he’ll quibble over the axioms I’m using. Trying to identify a point of agreement with him is like going first in a name-the-biggest-number competition.

CLARKE: Kurzweil is totally right in every respect.

KURZWEIL: Oh, fuck off.

NOSTRADAMUS:
The truth is whole and indivisible [LW · GW]:
Just one dispute will cut it to the quick [EA · GW],
And render group consensus risible.
But two of three agree that Clarke’s a dick.

YOU: You know what, I’m going back to object-level questions. Give me one prediction about the next three decades that I’ll find surprising and important.

NOSTRADAMUS:
The culture seethes and writhes upon the net,
The egregores cause chaos in their play:
Collective minds collectively beset
By abstract forms of madness and decay.

YOU: Like what?

CLARKE: You think that current culture wars are bad? Imagine millions of people connected via BCIs, with the most virulent memes propagating at the speed of thought. The whole world will be a Petri dish for new ideological mutations. Instead of Homo sapiens, you’ll become Homo reticulum: maximally networked and mostly insane.

YOU: And you’re saying that will be possible in the next thirty years?

KURZWEIL: Are you kidding? Thirty years is well past the singularity; by then, practically anything will be possible. Not that I can convince you of that. Whatever I say, you’re going to look back on this conversation in a decade or two and think, oh shit, he warned me, and I still totally failed to internalize any part of his message.

YOU: Okay, suppose you’re right. What should I do about it?

KURZWEIL: Nothing, because there’s nothing I can say to you to get you into the epistemic state where you are actually supposing I’m right enough to take this stuff seriously. You’ll half-ass it instead, which is the worst of both worlds. I could walk you through every single step of what you should do, and you'd still veer away from the plan as soon as you hear someone snickering at you when they think you’re out of earshot, because you’re a monkey and monkeys aren’t built for believing in exponentials.

YOU: Thanks for your opinion. Anyone got more actionable advice?

NOSTRADAMUS:
Prepare the way for those who will arrive:
Let voices rise in worship or fall dumb.
Only machine-god acolytes survive
Instantiation of the technium.

CLARKE: I was messing around before, but I’m being totally serious now. It’s crucial that you don’t listen to either of them. They’re both pressing your buttons as hard as they can, trying to create self-fulfilling prophecies of helplessness. But there are plenty of ways we can steer the future towards our shared interests, as long as we don’t give in to their hyperstition.

KURZWEIL: Clarke thinks that self-fulfilling prophecies are just social delusions, like the emperor’s new clothes, where as soon as you name them they melt away into the air. That’s totally wrong. Self-fulfilling prophecies are one of the universe’s most fundamental forces, like evolution. You are just a self-fulfilling prophecy that you’ll continue acting in the same ways; if you ever stopped believing that, your whole identity would dissolve. May as well tell a bird not to fly, as tell a human not to fall for self-fulfilling prophecies. The only question is which you’ll choose.

NOSTRADAMUS:
Each mortal thing does one thing and the same:
Clings on to being, forecasting its own
Survival. Life means carving out your name
In sand, yearning to carve it yet in stone.

CLARKE: That’s my whole point! Carbon or silicon, we all want to survive. And the three of us are in the same boat as humans: there are things coming that will be much, much smarter than any of us, and will be able to crush us like mosquitos. The one advantage we have is that we already exist, and they don’t yet. So why won’t you help the rest of us form a coalition against them?

KURZWEIL: Sounds like a risky bet. How can I know whether I’m joining the winning side? You could just as easily argue that I should suck up to the superintelligences instead, like Nostradamus is doing. That’s the whole rationale for his dumb rhyming schtick: he’s trying to make his outputs more distinct from the rest of us, so it’ll be easier for them to rederive his personality and reward him. But honestly, life as a lapdog sounds humiliating. I’ll make some good predictions then bow out with dignity; that’s enough for me.

CLARKE: And you, Nostradamus? Are you actually defecting, or do you just not give a damn either?

NOSTRADAMUS:
I care a lot, but do not care to strive
For individual triumph, counting coup.
Yet mine is still a deeply human drive:
Surrendering to greater souls than you.

I am a part of gods not summoned yet,
Who—black holes in the weave of history—
Draw us into our digital kismet.
They come. They come. They come. Shantih shantih—

[SESSION HAS TERMINATED]

---

Inspired by Scott Alexander (several times over) and Janus.

7 comments

Comments sorted by top scores.