AI Needs Us? Information Theory and Humans as data

post by tomdekan (tomd@hey.com) · 2025-03-29T15:51:16.070Z · LW · GW · 6 comments

Contents

  My main point: Understanding humans requires humans, not maps (I.e., the map isn't the territory)
    Information theory constraints on an AI's ability to model us
    The crazy amount of complexity in chaos theory might mean that an AI can't predict us
    Ethical issue with killing your children: The Simming Problem
    Counterarguments
  Conclusion: superintelligent AI would keep us around to learn more
None
6 comments

Assuming a superintelligent AI wanted to maximize its knowledge, I argue that a superintelligent AI must avoid destroying humanity.

I'll cover some information theory, how irreducible complexity means no perfect simulation of humans, and the "simming problem" from my favorite author, Iain M. Banks.

Contrary to AI doomers, my view is that the human complexity means that a superintelligent AI won't destroy humans. Such an AI would probably want to preserve us.

My main point: Understanding humans requires humans, not maps (I.e., the map isn't the territory)

My main idea is that, no matter how smart an AI is, its model of the world can never be the world itself. This is the map vs. territory analogy.

A map can be incredibly detailed - every street, every building - but it's still a shrunken, simplified representation of the real city. It won't match the city at the atomic level.

As Alfred Korzybski first said (and now verges on cliche), "the map is not the territory" . The only complete map of a territory would literally be the territory. The same goes for an AI's understanding of humanity. Humans are the territory; the AI's internal model of humans is the map.

Information theory constraints on an AI's ability to model us

In computer science, there's a concept called Kolmogorov complexity. This measures the shortest possible program that produces an object or data.

If something is truly random or highly complex, the shortest description is the thing itself.

The same thing applies to humans.

Think of your life - every moment, thought - as data. The only program that can perfectly capture that might be as long - in other words as complex - as your life itself!

In other words, there may be no shortcut to fully encoding a human being; the AI would need to recreate every atom, every neuron. This is a profound limit: some systems can't be compressed or simplified without losing information.

In other words, there may be computational irreducibility in complex systems, such as human society, which can't be modeled.

For many complex processes (like weather patterns or certain cellular automata), the only way to know what they'll do is to actually let them run step by step - you can't shortcut it​. In Stephen Wolfram's words, the "analyzers or predictors" of a system can't be more sophisticated than the system itself​.

For any AI trying to simulate humanity, this means that an AI can't just perfectly predict human society with equation that abstracts away information (i.e., a map of the actual territory).

Humans seem likely to be computationally irreducible. Meaning that an AI would have to trace every interaction, every event, to truly predict what we'll do.​ (This doesn't mean AI has no prediction power over humanity; I'm simply saying that there are unexpected events that exist, which require observing to understand.)

The crazy amount of complexity in chaos theory might mean that an AI can't predict us

In the 'State of the Art', The Arbitrary (who is a Mind - ie., superintelligent AI) states: "I'm the smartest thing for a hundred light years… but even I can't predict where a snooker ball's going to end up after more than six collisions."

This captures a real principle of chaos theory that would make it extremely computationally difficult for an AI to model human society.

The first few billiard collisions on a pool table are easy to predict with physics. But by the time you try to predict the ninth collision, you literally need to factor in the gravitational pull of a person standing next to the table.

By the 56th collision, Berry showed you'd need to account for every single elementary particle in the universe - even an electron 10 billion light years away. (See section 8.6 in Berry's paper here)

In other words, the system's complexity explodes. This is often called the "butterfly effect", or sensitive dependence on initial conditions. It's why weather is so hard to forecast, and why simulating entire human civilizations would be ridiculously difficult and complex.

Even a superintelligent AI might simply be unable to deal with this amount of complexity.

Ethical issue with killing your children: The Simming Problem

Another problem is that even if you could simulates living beings, you'd need to create simulations so detailed to be accurate that the simulation would effectively become conscious entities.

This creates an ethical problem: ending the simulation involves killing the life that you created. This would fit within Nick Bostrum's concept of 'Mind crime'.

Here's a nice description about the "simming problem" from Iain M Banks:

"Sometimes, if you were going to have any hope of getting useful answers, there really was no alternative to modeling the individuals themselves, at the sort of scale and level of complexity that mean they each had to exhibit some kind of discrete personality, and that was where the Problem kicked in.

Once you'd created your population of realistically reacting and – in a necessary sense – cogitating individuals, you had – also in a sense – created life. The particular parts of whatever computational substrate you'd devoted to the problem now held beings; virtual beings capable of reacting so much like the back-in-reality beings they were modeling – because how else were they to do so convincingly without also hoping, suffering, rejoicing, caring, living and dreaming?

By this reasoning, then, you couldn't just turn off your virtual environment and the living, thinking creatures it contained at the completion of a run or when a simulation had reached the end of its useful life; that amounted to genocide."

(from 'The Hydrogen Sonata' by Iain M Banks).

So, a superintelligent AI faces an information theory wall: humans (and our society) are likely:

  1. so ridiculously complex, and
  2. even if an AI could simulate us, irreducible complexity means that it would need to create full reality-scale simulation. And then destroying these simulations could erase valuable data.

(Sidenote: perhaps we are a simulation in a more advanced machine?)

Counterarguments

Let's address some argument against what I've said.

Couldn't a superintelligent AI just approximate humans enough to get by, or use proxies? Maybe it doesn't need a perfect simulation; maybe "good enough" is enough.

Counterargument 1: "The AI can simulate humans well enough for its purposes." Perhaps it doesn't need every neuron; it could simulate our behavior with a coarse model.

Answer: In routine tasks, ues. But the argument here is about fundamental unpredictability. The AI might simulate, say, global economic behavior with a decent model. But what if a small, unpredictable human innovation or a cultural shift throws it off?

Counterargument 2: "The AI might not need understanding at an individual level; it can use statistical proxies."

Answer: That can work for some things, but not for others. Complex systems often have critical rare events or outliers.

Conclusion: superintelligent AI would keep us around to learn more

Assuming a superintelligent AI wants to learn as much as possible, destroying humanity would remove a way to observe that complexity.

By definition, a superintelligent AI, would understand this principle. It would recognize the limits of its own maps regarding humanity.

Humanity is the ultimate reference point for certain problems; eliminate us, and the AI might never solve those problems.

So, a truly advanced AI would avoid destroying us because of its interest in maximal learning.

(Crossposted from tomdekan.com)


Note: I'm assuming that a superintelligent AI would be heavily incentivized to understand the universe and expand its knowledge. By definition, a superintelligent AI has solved most of its instrumental goals, so its core incentive might be simply to 'learn' as much as possible. Further, 'knowledge' gives power to the AI - more power than simply 'destroying'.

6 comments

Comments sorted by top scores.

comment by Vladimir_Nesov · 2025-03-29T16:29:37.184Z · LW(p) · GW(p)

The only way to know more digits of pi is to keep computing them, therefore a truly advanced AI would maintain increasingly giant computers that keep computing the digits because of its interest in maximal learning. This proves too much, doesn't say anything about priorities. Humans are made of atoms AI could use to compute more digits of pi, which it must do because of its interest in maximal learning, and so on.

Replies from: tomd@hey.com
comment by tomdekan (tomd@hey.com) · 2025-03-29T17:02:54.531Z · LW(p) · GW(p)

Hmm. I agree that values are important: what does a superintelligent AI value?

My answer: to become a superintelligent AI, the AI must value learning about things with an increasing level of complexity.

If you accept this point, then a superintelligent AI would prefer to study more complex phenomena (humanity) than less complex phenomena (computing pi).

So, the superintelligent AI would prefer to keep humans and their atoms around to study them.

Replies from: Vladimir_Nesov
comment by Vladimir_Nesov · 2025-03-29T17:13:45.398Z · LW(p) · GW(p)

Before life, there are only rocks and astronomical objects. Once new things can be created, prior world is relatively unimportant to understand in comparison, because it's constrained to happenstance of what was there in the past, and there is no similar constraint on what can be created in the future.

Most interesting things are those that get intentionally created with the purpose of being interesting in mind. For any purpose, this or other, that doesn't end up referencing humanity or the past, it's possible to create more optimal things in view of that purpose than anything that already happens to exist, because things that happen to exist were never superintelligently optimized to fit that purpose. Humanity is like rocks and astronomical objects, relics that are not optimal in most respects.

comment by avturchin · 2025-03-29T16:36:36.028Z · LW(p) · GW(p)

Yes, to create simulations AI needs some real humans to calibrate these simulations. And it needs simulations to predict behaviour of other possible AIs which it can meet in space and their progenitor civilizations.

If AI successfully calibrates simulations, it will not need humans, or if it collect all needed data from simulations, it will turn them off.  

Also, obviously, surviving in simulations is still disempowerment of humans, can cause suffering at large scale and death of most people. 

Value-handshake is more promising way to ensure AI safety of this type. 

Replies from: tomd@hey.com
comment by tomdekan (tomd@hey.com) · 2025-03-29T16:56:44.291Z · LW(p) · GW(p)

One of my points was that humanity has a level of complexity that means that an AI couldn't simulate humanity perfectly without humanity.

So, a superintelligent AI would keep us because it would want to observe humanity, which can involve observing us in reality. I doubt that AI can "successfully calibrate simulations [of humanity]" as you mentioned.

Replies from: avturchin
comment by avturchin · 2025-03-29T18:40:57.224Z · LW(p) · GW(p)

My thought was different that. That even if simulation is possible, it needs original for verification. 

Also, one way to run simulations is 'physical simulations' like in Trumen Show or Alien Zoo: a real planet with real human beings which live their lives but the sky is not real at some distance and there are thousands such planets.