Humans can communicate with and productively use many animals (some not extinct*), some of whom even understand concepts like payment and exchange. (Animal psychology has advanced a lot since Adam Smith gave hostage to fortune by saying no one had ever seen a dog or other animal truck, barter, or exchange.) We don't 'trade' them with them. A few are fortunate enough to interest humans in preserving and even propagating them. We don't 'trade' with those either. At the end of the day, no matter how many millions her trainer earns, Lassie just gets a biscuit & ear scritches for being such a good girl. And if she isn't a good girl, we genetically engineer and manufacture (ie. breed) an ex-wolf who is a good girl.

I'd also highlight the lack of trade with many humans, as well as primates. (Consider the cost of crime and how easily one can create millions of dollars in externalities; consider the ever skyrocketing cost of maintaining research primates, especially the chimpanzees - there is nothing that a chimpanzee can do as a tradeable service which is worth >$20k/year and the costs of dealing with it being able to at any moment decide to rip off your face.)

* yet - growth mindset!

Trade with ant colonies would work iff:

• We could cheaply communicate with ant colonies;
• Ant colonies kept bargains;
• We could find some useful class of tasks that ant colonies would do reliably (the ant colonies themselves being unlikely to figure out what they can do reliably);
• And, most importantly:  We could not make a better technology that did what the ant colonies would do at a lower resource cost, including by such means as eg genetically engineering ant colonies that ate less and demanded a lower share of gains from trade.

The premise that fails and prevents superintelligences from being instrumentally incentivized to trade with humans as a matter of mere self-interest and efficiency is point 4.  Anything that can be done by a human can be done by a technology that uses less resources than a human.

The reason why it doesn't work to have an alternate Matrix movie in which the humans are paid to generate electrical power is not that the Matrix AIs can't talk to the humans, it's not that no humans will promise to pedal a generator bike if you pay them, it's not even that every kind of human gets bored and wanders away from the bike and flakes out on the job, it's that this is not the most efficient way to generate electrical power.

I agree the ant analogy is flawed. But I don't think it's as flawed as you do.

• In this scenario, the 'trade' we would make would plausibly be "do this stuff or we kill you", which is not amazing for the ants.
• I think another disanalogy is that humans can't re-arrange ants to turn them into better trading partners (or just raw materials), but AI could do that to us. (h/t to Dustin Crummett for reminding me of this). And the fact that we might not be able to understand fancy AI concepts could make this option more appealing.

Putting the entire failure to trade on the ability to communicate seems to understate the issue. Most if not all of the things listed that they 'could' do, are things which they could theoretically do with their physical capacities, but not with their cognitive abilities or ability to coordinate within themselves to accomplish a task.

In general, they aren't able to act with the level of intentionality required to be helpful to us except in cases where those things we want are almost exactly the things they have evolved to do (like bees making honey, as mentioned in another comment).

The 'failure to communicate' is therefore in fact a failure to be able to think and act at the required level of flexibility and abstraction, and that seems more likely to carry over to our relations with some theoretical, super advanced AI or civilisation.

The analogy fails for me because while "we don't trade with ants" is true, the very similar "we don't trade with bees" is not so true, for some definition of "trade" that seems at least somewhat appropriate.

But also: ants can actually do heaps of things we can’t, whereas (arguably) at some point that won’t be true for us relative to AI systems.

Devil's advocate: by comparative advantage, even if the AI was strictly superior to humans at all tasks, it might still make sense for it to trade with humans.

Get out of our houses before we are driven to expend effort killing them, and similarly for all the other places ants conflict with humans (stinging, eating crops, ..)

Ant mafia: "Lovely house you've got there, wouldn't it be a shame if it got all filled up with ants?"

I just want to note that I personally do in fact trade with ants. I really enjoy watching them carry a pile of sugar to their nest, so sometimes when I go for walks I bring a baggie of sugar, then I offer it to the ants and they carry it around for my entertainment. They don't know that's what's happening, but it works out the same: I give them something they want, they do something they wouldn't otherwise have done, and we both benefit.

By the way, you do know that ants already do service for people by harvesting seeds for rooibos tea?

https://wildaboutants.com/tag/rooibos-seeds-harvested-by-ants/

Curated.

On one hand, I think I still disagree with the thrust of this post. I think the way we might trade with ants (or bees or dogs or horses, etc), is still just really different from what people typically have in mind when they're asking why AI might keep us alive, and the prospects discussed here are not reassuring to me. (And I have model-driven guesses of why superintelligences could build replacements for whatever humans are comparatively good at)

But, this post and the comments still prompted a lot of interesting thoughts. I appreciate posts that do a kind of "original seeing" on longstanding common arguments. I think I learned some things that are at least plausibly relevant to some kinds of AI takeoff here, and I also just learned or reconceptualized a lot of interesting stuff about how humans and animals interact.

I love the genre of "Katja takes an AI risk analogy way more seriously than other people and makes long lists of ways the analogous thing could work." (the previous post in the genre being the classic "Beyond fire alarms: freeing the groupstuck.")

Digging into the implications of this post: 

In sum, for AI systems to be to humans as we are to ants, would be for us to be able to do many tasks better than AI, and for the AI systems to be willing to pay us grandly for them, but for them to be unable to tell us this, or even to warn us to get out of the way. Is this what AI will be like? No. AI will be able to communicate with us, though at some point we will be less useful to AI systems than ants could be to us if they could communicate.

I'm curious how much you think the arguments in this post should affect our expectations of AI-human relations overall? At its core, my concern is:

• sure, the AI will definitely trade with useful human organizations / institutions when it's weak (~human-level), 
• and it might trade with them a decent amount when it's strong but not world-defeating (~human-organization-level)
• eventually AI will be human-civilization-level, and probably soon after that it's building Dyson spheres and stuff. Why trade with humanity then? Do we have a comparative advantage, or are we just a waste of atoms?

I can think of a few reasons that human-AI trade might matter for the end-state:

1. We can bargain for the future while AIs are relatively weak. i.e., when humans have stuff AI wants, they can trade the stuff for an assurance that when the AI is strong, it'll give us .000001% of the universe. 
   1. This requires both leverage (to increase the share we get) and verification / trust (so the AI keeps its promise). If we have a lot of verification ability, though, we could also just try to build safe AIs? 
   2. (related: this Nate post [LW · GW] saying we can't assume unaligned AIs will cooperate / trade with us, unless we can model them well enough to distinguish a true commitment from a lie. See "Objection: But what if we have something to bargain with?") 
2. It seems possible that an AI built by humans and trained on human-ish content ends up with some sentimental desire for "authentic" human goods & services. 
   1. In order for this to end up good for the humans, we'd want the AI to value this pretty highly (so we get more stuff), and have a concept of "authenticity" that means that it doesn't torture / lobotomize us to get what it wants.  
   2. This is mostly by analogy to humans buying "authentic" products of other, poorer humans, but there's a spectrum between goods/services, and something more like "the pleasure of seeing something exist" a la zoo animals. 
      1. (goofy attempt at illustrating the spectrum: a woven shawl vs a performed monologue vs reality tv vs a zoo.)
      2. So a simpler, perhaps more likely, version of the 'desirable authentic labor' possibility is a 'human zoo', where the AI just likes having "authentic" humans around. which is not very tradelike. But maybe the best bad AI case we could hope for is something like this -- Earth left as a 'human zoo' while the AI takes over the rest of the lightcone.

I think "trade" and "communication" are linked, and seem to exist on a spectrum that correlates to creatures' ability to predict the future. At the one extreme, we have gardeners who get plants to do what they wish by shaping the environment that the plants grow in. Near the middle, we have our interactions with domestic animals. At the other extreme, we have modern capitalism, where people exchange money for time spent on tasks they often wouldn't consider doing without the pay.

I suspect that where an interaction falls on that spectrum has a lot to do with the creature's ability to suppress its "nature", desires etc, in pursuit of longer-term goals. The better we communicate with an organism, the more we can promise it some future reward to change its present actions. The better at planning you get, the more you can start reasoning about risks, future rewards, etc.

I cannot communicate with the plants in my garden, so I have to actually provide them with the light and water and nutrients and protection that they need in order for them to grow. I cannot communicate with bees* , so to keep a hive of them, I need to make the hive more appealing than the other locations that the colony might consider swarming to.

Chickens can predict the future better than plants. Often when I open the door of the house nearest their run, it's because I'm about to throw them some table scraps. Thus, they come running when they hear the door open or see me outside, because they assess that there's a high probability of being the first to the snacks if they hurry.

Dogs predict the future actions of humans quite well, and behave in ways that they anticipate will get the responses they want. We're capable of relatively nuanced communication with dogs: "if you sit and then offer me your paw and then shake my hand, you'll get a treat!" or "Don't eat my livestock and I will feed you what I want you to eat". Wolves aren't as good at predicting humans' future behavior, and thus we have a harder time coexisting closely with them, because we can't strike the "don't eat what I don't want you to" kinds of bargains.

People can (usually) predict the future better than chickens or dogs. I do my job, work on multi-month or multi-year initiatives, and attend a bunch of meetings not for the promise of immediate reward, but because I anticipate a paycheque in a few weeks and better financial health long-term due to good benefits such as 401k matching.

These interactions all happen on the level of the worse communicator. Trading is only as good as the least trustworthy party in the deal, etc. I suspect that an AI might find it most effective to get along with people by treating people like people treat people -- I get along with my chickens by understanding the flock hierarchy social rules, one gets along with dogs by understanding dog social rules, etc.

• Technically, I can tell certain kinds of lies to a colony of bees in order to influence their behavior. Classic example is that I can lie to the bees by claiming there's a forest fire and they should get ready for their hive to be destroyed, which is what I do if I blow smoke into the hive. When they think there's a fire coming, they all eat as much honey as they can, because if they have to fly away then they only get to take the food they're carrying. When a bee has its stomach full of honey, she cannot bend over in the gesture necessary to sting, and she probably doesn't want to anyways, like how you probably don't want to get in a fight right after a huge meal. (only she-bees can sting, he-bees develop reproductive anatomy instead with the parts that become the she-bees' stingers)

When a person says “we don’t trade with ants”, I think the implicit explanation is that humans are so big, powerful and smart compared to ants that we don’t need to trade with them because they have nothing of value and if they did we could just take it; anything they can do we can do better, and we can just walk all over them. Why negotiate when you can steal?

I think this is an overly narrow definition of trading for this context. If an AGI wants something from humans it needs to leave us alive and happy enough to produce it. It might be nonconsensual, but dead people can't produce anything to steal, so the problem has become "avoid being tortured" rather than "avoid being killed altogether."

A lot of people do trade spiders warmth and shelter in exchange for keeping the insect population low. The spider does not know that this is happening, and it can't negotiate, but my impression is this is working out pretty well for them. Locally at least- maybe sufficiently intelligent house spiders would regret evolving to only live in houses and die outside, or would resent our reworking of the ecostructure- but they are alive, and selection to live in houses probably doesn't make them miserable in houses, even if it does limit their options. 

Much like "Let's think about slowing down AI" (Also by KatjaGrace, ranked #4 from 2022), this post finds a seemly "obviously wrong" idea and takes it completely seriously on its own terms. I worry that this post won't get as much love, because the conclusions don't feel as obvious in hindsight, and the topic is much more whimsical. 

I personally find these posts extremely refreshing, and they inspire me to try to question my own assumptions/reasoning more deeply. I really hope to see more posts like this. 

While I can imagine very simple forms of trade with human-level intelligent ants (e.g. you provide X units of wood, we will give you Y units of sugar), I do not expect a good outcome if I try to hire "army of ants" as an employee in my organization. I do not expect they would be able to join meetings, contribute points, understand other humans' illegible desires for a project, understand our vague preferences, etc. What I'm saying is I only think this works for very well-defined trades, and not for a lot of other trades.

With ants, ‘go over there and we won’t kill you’ would do a lot, and it doesn’t involve concepts at the foggy pinnacle of human meaning-construction.

I agree that with a human-ant language, I could tell ants to leave my house. But then they'd probably come back in a week? I don't think ants can reason about the future. 

Likewise, humans might lack some concepts that are necessary for making meaningful trades with advanced AI agents.

the potentially enormous speed difference (https://www.lesswrong.com/posts/Ccsx339LE9Jhoii9K/slow-motion-videos-as-ai-risk-intuition-pumps [LW · GW]) will almost certainly be an effective communications barrier between humans and AI. there’s a wonderful scene of AIs vs humans negotiation in william hertling’s “A.I. apocalypse” that highlights this.

Anecdotal example of trade with ants (from a house in Bali, as described by David Abrams):

The daily gifts of rice kept the ant colonies occupied–and, presumably, satisfied. Placed in regular, repeated locations at the corners of various structures around the compound, the offerings seemed to establish certain boundaries between the human and ant communities; by honoring this boundary with gifts, the humans apparently hoped to persuade the insects to respect the boundary and not enter the buildings.

When I tried to answer why we don't trade with ants myself, communication was one of the first things (I can't remember what was actually first) I considered. But I worry it may be more analogous to AI than argued here.

We sort of can communicate with ants. We know to some degree what makes them tick, it's just we mostly use that communication to lie to them and tell them this poison is actually really tasty. The issue may be less that communication is impossible, and more that it's too costly to figure out, and so no one tries to become Antman even if they could cut their janatorial costs by a factor of 7.

The next thought I had was that, if I were to try to get ants to clean my room, I think it's likely that the easiest line towards that is not figuring out how to communicate, but breeding some ants with different behavior (e.g. search for small bits of food, instead of large bits. This seems harder than that sentence suggests, but still probably easier than learning to speak ant). I don't like what that would be analogous to in human-AI interactions.

I think it's possible that an AI could fruitfully trade with humans. While it lacks a body, posting an ad on Craigslist to get someone to move something heavy is probably easier than figuring out how to hijack a wifi-enabled crane or something. 

But I don't know how quickly that changes. If the AI is trying to build a sci-fi gadget, it's possible that an instruction set to build it is long or complicated enough that a human has trouble following it accurately. The costs of writing intuitive instructions, and also designing the gadget such that idiot-proof construction is possible could be high enough that it's better to do it itself. 

Great post.

I don't think communicating trades is the only issue. Even if we could communicate with ants, e.g. "Please clean this cafeteria floor and we'll give you 5 kg of sugar" "Sure thing, human", I think there are still barriers.

• Can the ants formulate a good plan for cleaning the floor?
• Can the ants tell when the floor is clean enough?
• Can the ants motivate their team?
• Can the ants figure out where to deposit debris, and figure this out if a human janitor accidentally leaves the bin in a different place than yesterday

There's a lot to the task of cleaning the cafeteria floor beyond is it mechanically possible for the worker and can the worker speak English well enough to articulate a trade.

 Another problem is trust. In order for trade to work, the AI has to trust that the human will follow the deal. If 1% of humans decide to smash the robots instead, humans could be totally useless. Sure, there are some things ants could do, but if the ants sometimes caused a problem, they would be much less useful. 

Humans are the strongest source of potentially adversarial optimization. The cost of defending against an enemy is huge. Hiring someone who has even a 1% chance of actively trying to harm you is probably a bad move in expectation. 

Also, all the useful stuff ants could do is because ants are better on many metrics than human made robots. Evolution still holds tricks that human engineers don't yet wield. 

The other advantage of communicating-ants would be avoiding the human cost of labor. 

An AI that can easily duplicate itself has a very low cost of labor. And if the AI's robotics beats human bodies on most metrics, we have no remaining advantages.

And trade isn't the only option. Trade is one way to persuade another mind to work for you, often an expensive one as you must trade something away. The ideal state for the AI is to give humans nothing the AI minds loosing. For example, the AI could trade us virtual loot in a hyperaddictive computer game for whatever it wanted from us. Or otherwise mind-hack us. 

Should we humans broadcast more explicitly to future AGIs that we greatly prefer the future where we engage in mutually beneficial trade with them to the future where we are destroyed?

(I am making an assumption here that most, if not all, people would agree with this preference. It seems fairly overdetermined to me. But if I'm missing something where this could somehow lead to unintended consequences, please feel free to point that out.)

Humans are unlikely to be the most efficient configuration of matter to carry out any particular task the AI wants to get done - so if the power imbalance is sufficiently large the AI will be better off wiping us out to configure the matter in a more efficient way.

Human brains are easily hackable (a good book does it, or a good brainwasher), we change our views easily given the right "argument", logical and/or emotional. Anything smarter than us can figure out a way to get us what it wants us to do without anything like a "trade", but because we think it's the right thing to do. If you doubt it, note that EA did a good job of convincing people to donate to charity. The military is an even more extreme example. The best con does not feel like a con at all. (Not saying that either of the two examples is a con, just that we cannot tell a good cause from a bad cause apart when we are sufficiently brainwashed.)

For an ant analogy, spray some pheromones to where you need them to go, without any reward.

Ants that have intelligence anywhere near the level needed for meaningful trade with us would likely cause our extinction. We don't trade with ants mainly because they're too stupid to do anything we want, despite being physically capable of doing plenty of things we might want.

If you scale up their intelligence to the level where they're worthwhile trading with, there are stories you can tell about how beneficial trade would be between our species, but frankly I don't think that we would be of much benefit to them^[1].

1. ^{^}

   Them! was a 1950's movie about giant mutated ants. Even as a child I thought that quadrillions of tiny intelligent ants would be a far worse threat.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

Size circumscribes – it has no room For petty furniture – The Giant tolerates no Gnat For Ease of Gianture –

Repudiates it, all the more – Because intrinsic size Ignores the possibility Of Calumnies – or Flies.

~ Dickinson

The following issue seems fundamental and related (though i am not sure how exactly :-) ): There is a difference between things ants could physically do and what they are smart enough to do / what we can cheaply enough explain to them. Similarly for humans: delegating takes work. For example, hiring an IQ 80 cleaner might only be worth it for routine tasks, not for "clean up after this large event and just tell me when it's done, bye". Similarly, for some reason I am not supervising 10 master students, even if they were all smarter than me.

End of the day, it's about power.

Sufficiently advanced AI could create bots to do the things they need done. We cannot create an ant equivalent bot (yet). Messengers on horses don't exist alongside the car, plane, or internet. Bots created by AI will likely fit the needs they have much more neatly and for lower maintenance costs than paying humans.

We are everyday finding new ways to automate human labor, from mental to physical to creative. Why would AI suddenly stop that effort in order to trade with us?

Good post, but there is a big disbalance in human-ants relationships. 

If people could communicate with ants, nothing would stop humans to make ants suffer if it made the deal better for humans because of a power disbalance. 

For example, domesticated chickens live in very crowded and stinky conditions, and their average lifespan is a month after which they are killed. Not a particularly good living conditions.

People just care about profitability do it just because they can.

Directionally agree, but: A) Short period of trade before we become utterly useless is not much comfort. B) Trade is a particular case of bootstrapping influence on what an agent value to influence on their behaviour. The other major way of doing that is blackmail - which is much more effective in many circumstances, and would have been far more common if the State didn't blackmail us to not blackmail each other, to honour contacts, etc.

BTW those two points are basically how many people afraid that capitalism (i.e. our trade with super human organisations) may go wrong: A) Automaton may make us less and less economically useful. B) enough money may give an organisation the ability to blackmail - private army, or more likely influence on governmental power.

Assuming that automation here mean AI, this is basically hypothesising a phase in which the two kinds of super human agents (AI and companies) are still incentivized to cooperate with each other but not with us.

Assuming AI doesn't care about acting ethically, and even assuming AI can communicate and find useful things for us to do, there's no reason why AI wouldn't just manipulate and coerce humans rather than trading with them. 

Even in real-life we have plenty of examples of humans enslaving each other, when you get sci-fi possibilities like an AI just implanting mind control devices in human heads, then why would an AI waste resources and probably sacrifice efficiency just to trade evenly with humans?

Human can manipulate animals and make them do what they want. So could AI

AI is dependent on humans.  It gets power and data from humans and it cannot go on without humans.   We don't trade with it, we dictate terms. 

Do we fear a world where we have turned over mining, production and powering everything to the AI. Getting there would take a lot more than self amplifying feedback loop of a machine rewriting its own code.