Against most, but not all, AI risk analogies

My point is that we should stop relying on analogies in the first place. Use detailed object-level arguments instead!

And yet you immediately use an analogy to make your model of AI progress more intuitively digestible and convincing:

I expect AIs will be born directly into our society, deliberately shaped by us, for the purpose of filling largely human-shaped holes in our world

That evokes the image of entities not unlike human children. The language following this line only reinforces that image, and thereby sneaks in an entire cluster of children-based associations. Of course the progress will be incremental! It'll be like the change of human generations. And they will be "socially integrated with us", so of course they won't grow up to be alien and omnicidal! Just like our children don't all grow up to be omnicidal. Plus, they...

... will be numerous and everywhere, interacting with us constantly, assisting us, working with us, and even providing friendship to hundreds of millions of people.

That sentence only sounds reassuring because the reader is primed with the model of AIs-as-children. Having lots of social-bonding time with your child, and having them interact with the community, is good for raising happy children who grow up how you want them to. The text already implicitly establishes that AIs are going to be just like human children. Thus, having lots of social-bonding time with AIs and integrating them into the community is going to lead to aligned AIs. QED.

Stripped of this analogizing, none of what this sentence says is a technical argument for why AIs will be safe or controllable or steerable. Nay, the opposite: if the paragraph I'm quoting from started by talking about incomprehensible alien intelligences with opaque goals tenuously inspired by a snapshot of the Internet containing lots of data on manipulating humans, the idea that they'd be "numerous" and "everywhere" and "interacting with us constantly" and "providing friendship" (something notably distinct from "being friends", eh?) would have sounded starkly worrying.

The way the argument is shaped here is subtler than most cases of argument-by-analogy, in that you don't literally say "AIs will be like human children". But the association is very much invoked, and has a strong effect on your message.

And I would argue this is actually worse than if you came out and made a direct argument-by-analogy, because it might fool somebody into thinking you're actually making an object-level technical argument. At least if the analogizing is direct and overt, someone can quickly see what your model is based on, and swiftly move onto picking at the ways in which the analogy may be invalid.

The alternative being demonstrated here is that we essentially have to have all the same debates, but through a secondary layer of metaphor, at which we're pretending that these analogy-rooted arguments are actually Respectably Technical, meaning we're only allowed to refute them by (likely much more verbose and hard-to-parse) Respectably Technical counter-arguments.

And I think AI Risk debates are already as tedious as they need to be.

The broader point I'm making here is that, unless you can communicate purely via strict provable mathematical expressions, you ain't getting rid of analogies.

I do very much agree that there are some issues with the way analogies are used in the AI-risk discourse. But I don't think "minimize the use of analogies" is good advice. If anything, I think analogies improve the clarify and the bandwidth of communication, by letting people more easily understand each others' positions and what reference classes others are drawing on when making their points.

You're talking about sneaking-in assumptions – well, as I'd outlined above, analogies are actually relatively good about that. When you're directly invoking an analogy, you come right out and say what assumptions you're invoking!

They establish almost nothing of importance about the behavior and workings of real AIs, but nonetheless give the impression of a model for how we should think about AIs. 

How do you know that they establish nothing of importance? 

Many proponents of AI risk seem happy to critique analogies when they don't support the desired conclusion, such as the anthropomorphic analogy. 

At the very least, this seems to go both ways. Like, afaict, one of Quintin and Nora’s main points in “AI is Easy to Control” is that aligning AI is pretty much just like aligning humans, with the exception that we (i.e., backpropagation) have full access to the weights which makes aligning AI easier. But is aligning a human pretty much like aligning an AI? Can we count on the AI to internalize our concepts in the same way? Do humans come with different priors that make them much easier to “align”? Is the dissimilarity “AI might be vastly more intelligent and powerful than us” not relevant at all, on this question? Etc. But I don’t see them putting much rigor into that analogy—it’s just something that they assume and then move on. 

My point is that we should stop relying on analogies in the first place. Use detailed object-level arguments instead!

It seems reasonable, to me, to request more rigor when using analogies. It seems pretty wild to request that we stop relying on them altogether, almost as if you were asking us to stop thinking. Analogies seem so core to me when developing thought in novel domains, that it’s hard to imagine life without them. Yes, there are many ways AI might be. That doesn’t mean that our present world has nothing to say about it. E.g., I agree that evolution differs from ML in some meaningful ways. But it also seems like a mistake to completely throw out a major source of evidence we have about how intelligence was produced. Of course there will be differences. But no similarities? And do those similarities tell us nothing about the intelligences we might create? That seems like an exceedingly strong claim. 

I would ask the question:

• Is the analogy being used to make a specific point, rather than trying to set up vague vibes-based associations?

If yes, good for you. If no, well there’s your problem.^[1]

In particular,

• future AI is like aliens in some ways but very unlike aliens in other ways;
• future AI is like domesticated animals in some ways but very unlike them in other ways;
• future AI is like today’s LLMs in some ways but very unlike them in other ways;

etc. All these analogies can be helpful or misleading, depending on what is being argued.

So anyway, if I were writing a post like this:

• I would say that analogy usage can be bad, rather than saying that the analogies themselves are bad.
• I would find examples of analogy-usage that are in fact actually bad, and then argue that they are bad. Instead, here you offer a list of 10 examples, but they’re all stripped from their context, so I can’t tell whether I endorse them or not.
• I would suggest that optimists and pessimists, and more generally people on all sides of every issue, sometimes use analogies appropriately and sometimes use them inappropriately, and thus I would try to pick a more even-handed set of examples. (For example, you seem to be suggesting that nobody would really be so crazy as to use a google maps analogy, except in the strange imaginings of doomers, but in fact this is a real example—Ajeya almost certainly had this 2012 post by her colleague Holden [LW · GW] at the back of her mind.)

1. ^{^}

   It’s a bit more subtle than that, because when someone is using an analogy in a good way to make a valid point, they are usually simultaneously choosing words and metaphors that set up vibes in a way that supports their point. It’s fine to be sad that this happens—so am I [LW · GW]—but people on all sides of every issue do it constantly.

A general point unrelated to use of analogies for AI risk specifically is that a demand for only using particular forms of argument [LW · GW] damages ability to think or communicate, by making that form of argument less available, even in cases where it would turn out to be useful.

An observation that certain forms of argument tend to be useless or misleading should take great care to guard against turning into a norm. If using a form of argument requires justification or paying a social cost of doing something that's normally just not done, that makes it at least slightly inconvenient [LW · GW], and therefore it mostly stops happening in practice.

I just want to clarify my general view on analogies here, because I'd prefer not to be interpreted as saying something like "you should never use analogies in arguments". In short:

I think that analogies can be good if they are used well in context. More specifically, analogies generally serve one of three purposes:

1. Explaining a novel concept to someone
2. Illustrating, or evoking a picture of a thing in someone's head
3. An example in a reference class, to establish a base rate, or otherwise form the basis of a model

I think that in cases (1) and (2), analogies are generally bad as arguments, even if they might be good for explaining something. They're certainly not bad if you're merely trying to tell a story, or convey how you feel about a problem, or convey how you personally view a particular thing in your own head.

In case (3), I think analogies are generally weak arguments, until they are made more rigorous. Moreover, when the analogy is used selectively, it is generally misleading. The rigorous way of setting up this type of argument is to deliberately try to search for all relevant examples in the reference class, without discriminating in favor of ones that merely evoke your preferred image, to determine the base rate.

My point is that we should stop relying on analogies in the first place. Use detailed object-level arguments instead!

Yeah, seriously. As a field, why haven't we outgrown analogies? It drives me crazy, how many loose and unsupported analogies get thrown around. To a first approximation: Please stop using analogies as arguments.

I sure don't think I rely much on analogies when I reason or argue about AI risk. I don't think I need them. I encourage others to use them less as well. It brings clarity of thought and makes it easier to respond to evidence (in my experience).

Part of this is that I don't share other people's picture about what AIs will actually look like in the future. This is only a small part of my argument, because my main point is that that we should use analogies much less frequently, rather than switch to different analogies that convey different pictures.

You say it's only a small part of your argument, but to me this difference in outlook feels like a crux. I don't share your views of what the "default picture" probably looks like, but if I did, I would feel somewhat differently about the use of analogies.

For example, I think your "straightforward extrapolation of current trends" is based on observations of current AIs (which are still below human-level in many practical senses), extrapolated to AI systems that are actually smarter and more capable than most or all humans in full generality.

On my own views, the question of what the future looks like is primarily about what the transition looks like between the current state of affairs, in which the state and behavior of most nearby matter and energy is not intelligently controlled or directed, to one in which it is. I don't think extrapolations of current trends are much use in answering such questions, in part because they don't actually make concrete predictions far enough into the future.

For example, you write:

They will be numerous and everywhere, interacting with us constantly, assisting us, working with us, and even providing friendship to hundreds of millions of people. AIs will be evaluated, inspected, and selected by us, and their behavior will be determined directly by our engineering.

I find this sorta-plausible as a very near-term prediction about the next few years, but I think what happens after that is a far more important question. And I can't tell from your description / prediction about the future here which of the following things you believe, if any:

• No intelligent system (or collection of such systems) will ever have truly large-scale effects on the world (e.g. re-arranging most of the matter and energy in the universe into computronium or hedonium, to whatever extent that is physically possible).
• Large-scale effects that are orders of magnitude larger or faster than humanity can currently collectively exert are physically impossible or implausible (e.g. that there are diminishing returns to intelligence past human-level, in terms of the ability it confers to manipulate matter and energy quickly and precisely and on large scales).
• Such effects, if they are physically possible, are likely to be near-universally directed ultimately by a human or group of humans deliberately choosing them.
• The answer to these kinds of questions is currently too uncertain or unknowable to be worth having a concrete prediction about.

My own view is that you don't need to bring in results or observations of current AIs to take a stab at answering these kinds of questions, and that doing so can often be misleading, by giving a false impression that such answers are backed by empiricism or straightforwardly-valid extrapolation.

My guess is that close examination of disagreements on such topics would be more fruitful for identifying key cruxes likely to be relevant to questions about actually-transformative smarter-than-human AGI, compared to discussions centered around results and observations of current AIs.

I admit that a basic survey of public discourse seems to demonstrate that my own favored approach hasn't actually worked out very well as a mechanism for building shared understanding, and moreover is often frustrating and demoralizing for participants and observers on all sides. But I still think such approaches are better than the alternative of a more narrow focus on current AIs, or on adding "rigor" to analogies that were meant to be more explanatory / pedagogical than argumentative in the first place. In my experience, the end-to-end arguments and worldviews that are built on top of more narrowly-focused / empirical observations and more surface-level "rigorous" theories, are prone to relatively severe streetlight effects, and often lack local validity, precision, and predictive usefulness, just as much or more so than many of the arguments-by-analogy they attempt to refute.

Here are my thoughts about this post:

First, 'well played, my friend' re: your interaction with Thane Ruthenis. Perhaps like Thane, I saw red when reading your analogy, but then I remembered that this was sorta exactly your point: analogies are weak evidence, you can use them fairly easily to argue for all sorts of things, and so the various pro-AI-risk-concern analogies you dislike can be mirrored by anti-AI-risk-concern analogies as well. (Here I'll also remind the reader of Yann LeCun's analogy of AI to ballpoint pens...)

Here's a suggestion for rule of thumb about how to use analogies; would you agree with this?

(1) People are free to make them but they are generally considered low-tier evidence, trumped by models and reference classes. 

(2) NOTE: If you find yourself playing reference class tennis, what you are doing is analogy, not reference classes. Or rather the difference between analogy and reference class is a spectrum, and it's the spectrum of how hard it is to play tennis. Something is a good reference class insofar as it's uncontroversial.

(3) It is valid to reply to analogies with counter-analogies. "An eye for an eye." However, it is usually best to do so briefly and then transition the dialogue into evaluating the analogies, i.e. pointing out ways in which they are relevantly similar or dissimilar to the case in question.

(4) And as said in point #1, even better is having actual models. But bandying about analogies and analyzing them is often a good first step towards having actual models!

I think some of the quotes you put forward are defensible, even though I disagree with their conclusions. 

Like, Stuart Russell was writing an opinion piece in a newspaper for the general public. Saying AGI is  "sort of like" meeting an alien species seems like a reasonable way to communicate his views, while making it clear that the analogy should not be treated as 1 to 1. 

Similarly, with Rob wilbin, he's using the analogy to get across one specific point, that future AI may be very different from current AI. He also disclaims with the phrase "a little bit like" so people don't take it too seriously. I don't think people would come away from reading this thinking that AI is directly analogous to an octopus.

Now, compare these with Yudkowsky's terrible analogy. He states outright "The AI is an unseen actress who, for now, is playing this character.". No disclaimers, no specifying which part of the analogy is important. It directly leads people into a false impression about how current day AI works, based on an incredibly weak comparison. 

Even if there are risks to using analogies with persuasion, we need analogies in order to persuade people. While a lot of people here are strong abstract thinkers, this is really rare. Most people need something more concrete to latch onto. Uniform disarmament here is a losing strategy; and not justified here as I don't think the analogies are as weak as you think. If you tell me what you consider to be the two weakest analogies above, I'm sure I'd be pretty to steelman at least one of them.

If we want to improve epistemics, a better strategy would probably be to always try to pair analogies (at least for longer texts/within reason). So identify an analogy to describe how you think about AI, identify an alternate plausible analogy for how you should think about it and then explain why your analogy is better/whereabouts you believe AI lies between the two.

Many proponents of AI risk seem happy to critique analogies when they don't support the desired conclusion, such as the anthropomorphic analogy [LW · GW].

Of course! Has there ever been a single person in the entire world who has embraced all analogies instead of useful and relevant analogies?

Maybe you're claiming that AI risk proponents reject analogies in general when someone is using an analogy that supports the opposite conclusion, but accepting the validity of analogies when it supports their conclusion. If this were the case, it would be bad, but I don't actually think this is what is happening. My guess would be that you've seen situations where someone has used an analogy to critique AI safety and then the AI safety person said something along the lines, "Analogies are often misleading" and you took this as a rejection of analogies in general as opposed to a reminder to check whether the analogy actually applies.

In contrast to the AIs-will-be-alien model, I expect AIs will be born directly into our society, deliberately shaped by us, for the purpose of filling largely human-shaped holes in our world. They will be socially integrated with us and will likely substantially share our concepts about the social and physical world, having been trained on our data and being fluent in our languages. They will be numerous and everywhere, interacting with us constantly, assisting us, working with us, and even providing friendship to hundreds of millions of people. AIs will be evaluated, inspected, and selected by us,

I think the AIs-will-be-alien model is great & yet every sentence here is also part of my model, so I think you are being a bit unfair to me & my position by making it sound otherwise.

I cut off the bit about "behavior will be determined directly by our engineering" because that is false and will remain false until we make a lot more progress in mechinterp and related fields.

I think this post misses one of the points of analogies. Often, if someone thinks "AI is like human", you can't say "no, AI is totally not like human" and expect your point to be understood. Often the point of analogy is to defamiliarize the subject.

But if you are attempting to convey the plausibility of inner misalignment, or a mental model of inner misalignment, why not choose instead to analogize the situation to within-lifetime learning among humans [LW · GW]?

This prompted me to think about this analogy for a few hours, and writing down my thoughts here [LW(p) · GW(p)]. Would be interested to know if you have any comments on my comments.

Also, I think this serves as a positive example for arguing by analogy, showing that it's possible to make intellectual progress this way, going from evolution-as-alignment to within-lifetime-learning-as-alignment to my latest analogy (in the above comment) of evolution-as-alignment-researcher, each perhaps contributing to a better overall understanding.

I think this post shifts the burden to risk-concerned folks and only justifies that risk with its own poor analogies. 7/10 of the analogy-makers you cite here have pdooms <=50%, so they are only aiming for plausibility. You admit that the analogies point to the "logical possibility" of stark misalignment, but one person's logical possibility is another person's plausibility.  

To give an example, Golden Retrievers are much more cherry-picked than Cotra's lion/chimpanzee examples. Of all the species on Earth, the ones we've successfully domesticated are a tiny, tiny minority. Maybe you'd say we have a high success rate when we try to domesticate a species, but that took a long time in each case and is still meaningfully incomplete. I think 7/10 are advocating for taking the time with AIs and would be right to say we shouldn't expect e.g. lion domestication to happen overnight, even thought we're likely to succeed eventually. 

The presentation of Quentin's alternate evolution argument seems plausible, but not clearly more convincing than the more common version one might hear from risk-concerned folks. Training fixes model weights to some degree, after which you can do some weaker adjustments with things like fine-tuning, RLHF, and (in the behavioral sense, maybe) prompt-engineering. Our genes seem like the most meaningfully fixed thing about us, and those are ~entirely a product of our ancestors' performance, which is heavily weighted toward the more stable pre-agricultural and pre-industrial human environments.  

I don't know how I hadn't seen this post before now! A couple weeks after you published this, I put out my own post [LW · GW] arguing against most applications of analogies in explanations of AI risk. I've added a couple references to your post in mine. 

If someone wants to establish probabilities, they should be more systematic, and, for example, use reference classes. It seems to me that there's been little of this for AI risk arguments in the community, but more in the past few years.

Maybe reference classes are kinds of analogies, but more systematic and so less prone to motivated selection? If so, then it seems hard to forecast without "analogies" of some kind. Still, reference classes are better. On the other hand, even with reference classes, we have the problem of deciding which reference class to use or how to weigh them or make other adjustments, and that can still be subject to motivated reasoning in the same way.

We can try to be systematic about our search and consideration of reference classes, and make estimates across a range of reference classes or weights to them. Do sensitivity analysis. Zach Freitas-Groff seems to have done something like this in AGI Catastrophe and Takeover: Some Reference Class-Based Priors, for which he won a prize from Open Phil's AI Worldviews Contest.

Of course, we don't need to use direct reference classes for AI risk or AI misalignment. We can break the problem down.

People aren't blank slates though.

If you talk to a very uninformed person about AI you'll find they have all sorts of broken assumptions. I think I was once in a twitter discussion with someone about AI and they said that chatGPT doesn't hallucinate. Their evidence for this was that they'd asked chatGPT whether it hallucinates and it said that it didn't.

I think your beliefs about the future of AI could fit into an analogy too: AI will be like immigrants. They'll take our jobs and change the character of society.

So, perhaps analogies are a crude tool that is sometimes helpful.