Intelligence as Privilege Escalation

post by Cole Wyeth (Amyr) · 2025-02-23T19:31:27.604Z · LW · GW · 0 comments

Contents

No comments

Epistemic status: An interesting idea that is probably already in the air.

Inherent power you possess as part of yourself. Granted power is lent or given by other people.

-Patrick Rothfuss, The Wise Man's Fear

Humans are more powerful than other animals because we are smarter - and better coordinated. Both sides of the story are important for understanding human power on an individual level as well.

Intelligence is a powerful thing. It has allowed humans to collectively reshape our world by inventing technologies that outstrip pretty much every animal at their own specialization. Rationalists have a pretty strong norm of optimizing intelligence over physical attributes like strength, exemplified and often justified by Eliezer's argument that humans conquered the world by being smarter, not by having sharper claws than other animals. Certainly on the scale of conflict between other species and humanity this holds up, and I think it works on the individual scale as well. That is, if you want the power to personally move really fast, you could practice sprinting or you could go work for a hedge fund and buy a sports car.

Notice the difference in frame here - I am focused on what makes an individual powerful in our context: the Anthropocene, a time when the world has been drastically reshaped and reorganized to serve human needs, with multiple levels of organization built on top of the material "resource extraction" layer, from markets to governments. This is different from what makes our species powerful; the power of our species is only the backdrop and the playing field.

Now, it is standard rationalist doctrine that things like being "cool" or charismatic ultimately happen in the brain, and are therefore facets of intelligence. I think this is mostly right; a decent percentage of the social skills that might be described as "charm" are primarily a type of social intelligence. It would be a mistake to argue that because a lot of anti-social unwashed nerds are less influential than famous youtubers with shining smiles and basic but entertaining takes, intelligence doesn't get you anywhere or is even detrimental. In fact, the youtubers are engaged in a mental struggle of their own (against eachother and the Youtube algorithm). Their success is also based partially on intelligence. 

But I think this picture of intelligence as the primary driver of individual power[1] is also missing something, which should be obvious empirically.

I don't think it's controversial that the most politically powerful people in the world are not the smartest. There are a couple of aspects to this. One is that the cognitive aspects of charisma are not the prototypical type of intelligence, so politicians might be smarter than they seem. One is that on a game theoretic level, you often don't want a scheming utility maximizer as leader - it's easier to verify that a "corrigible" virtue-ethicist[2] will serve your interests as a voter to a reasonable extent (this is also why religiosity is electable - at some point in the future, I want to write a book review of the Bible, and I think a lot of ink could be spilled on the Machiavellian genius of Moses constructing a social system based on carefully constructed coordination-enforcing delusions). But I think there's something else to it - I think power is a partially non-transferable asset, which is collected by reshaping the world (and to some extent yourself) so that it is full of locks to which you are the key.

I'll start with some highly standard, obvious examples.

A rich person is powerful because he owns a lot of stuff, which means that he can control a lot of resources, which is because he (or whatever system granted him power) has set up bank accounts and companies to respond to passwords that he knows, his biometrics, identification documents that he holds, etc. These are in fact his qualities, but those qualities are only useful because they have been externalized.

A slightly less trivial example is seeming aristocratic. There are various complicated mannerisms, skills, and fashions that are considered aristocratic, and aristocrats learn them (presumably as children). Apparently this is helpful as a signal when coordinating with other aristocrats. This is mostly a European vibe - but it also has e.g. an American manifestation: it is the reason that Presidents Trump and Biden argued on the national stage about who is better at golf. Golf skill is a costly signal (it's an expensive hobby that also requires a lot of leisure time to get good at). Golf skill probably matters to the power brokers, but not to the American people, and I guess Trump and Biden were too senile to remember they were currently supposed to be making their case primarily to the American people. 

Okay, now some vibes:

It's a common fantasy trope that "the true king" is able to pass some kind of divine/essentially genetic test that was set up in advance for them to be able to pass it. For instance, Aragorn has lots of skills, but mostly he's just that guy - Frodo, Samwise, and even Gandalf are not that guy, so they don't get to rule. 

Paul Atreides passes a genetic test[3] to get access to his family atomics. Sure, atomic weapons were invented by virtue of human intelligence, but not by virtue of his human intelligence. He leveled the shield wall using his genes, because the world was set up in a way that let his genes do that. 

Azula being aristocratic comes in clutch.

I like to think of these types of power as "keyholder" power. A wizard is usually powerful because he's smart and knows the rules of the world, so he can manipulate elemental forces to his advantage (like a hacker is smart because he can sometimes at great effort break cryptosystems and take stuff that isn't his). The promised king is usually powerful because he inherited an ancient key or something (like a billionaire is powerful because he can log into his bank account and move money around, or put his signature on a check). Keyholder power can rely on knowledge, but it is usually specific, ~incompressible knowledge like a random password.

This seems to imply that where keyholder power is privilege, intelligence manifests as the innate ability to escalate privilege, particularly unintended escalation through side channels.

It's a mistake to think of intelligence as the derivative of power though. It's much easier to grow your assets when you already have assets. I think that in most situations, the derivative of power is (to the first order) equal to power. 

I do suspect that sufficient levels of intelligence (particularly comparative intelligence, relative to the other relevant players) can allow highly discontinuous jumps in power. The canonical (speculative) example of this might be a superintelligence inventing nanotech and seizing the entire surface of the planet to build solar panels and computronium. I think a sufficiently relatively smart superintelligence would be able to do something like this. 

There are outside view reasons for skepticism though. I'm not sure that such a drastic and rapid privilege escalation has happened on an individual level at any point in history since the invention of a privilege-based resource allocation system with the rise of humans. The rise of a more intelligent digital "species" might reasonably be expected to disrupt this system completely, like the rise of humans disrupted the natural order and put chimps in zoos. These situations aren't quite analogous though - keyholder power didn't really exist in the world until humans created it. Privilege escalation may become more difficult as more of the natural world is shaped more and more robustly into assets gated behind keyholes.

Actually, this may be why it is hard to persuade global leaders that superintelligence is an extinction risk. Among all humans, they are some of the most familiar with the importance of keyholder power. This would be expected to shape (and probably, distort) their intuitions about the degree of privilege escalation possible for vastly superhuman intelligence - because in the local region of human intelligence, such privilege escalations are difficult and rare.

It is worth considering to what degree this sword cuts both ways - rationalists tend to be outsiders, independent researchers and autodidacts. We are obsessed with bootstrapping ourselves [LW · GW]through sheer grit and genius. Are we less capable of thinking realistically about keyholder power?

  1. ^

    Optionally, take a look at this short post [LW · GW] and the comment from @benwr [LW · GW] - basically he argues that humans effectively exercise more bits of optimization power than one would expect from the limits of our output channels because our preferences are privileged by being "easy to point at" because we already exist in the world. I think this forms provides nice context for the rest of my argument. 

  2. ^

    I think it's also easier to robustly, fervently obey such a moral system [LW · GW].

  3. ^

    I don't remember if this happens explicitly in the books or only the (recent) movies. 

0 comments

Comments sorted by top scores.