Will AI See Sudden Progress?

tldr: I think it matters differently when something is discontinuitous with linear vs exponential vs hyperbolic curves.

Some thoughts copied over from an earlier conversation about this.

Reading this and some of Paul's related thoughts, I experienced some confusion about why we care about discontinuity in the first place. Here are my thoughts on that.

Caring about discontinuity is similar but not identical to "why do we care about takeoff speed?". ESRogs mentions in this comment in the related Paul thread that we might care about:

How much of a lead will one team need to have over others in order to have a decisive strategic advantage...

in wall clock time? in economic doublings time?

How much time do we have to solve hard alignment problems...

in wall clock time? in economic doublings time?

A related issue is how surprising an advance will be, in a fashion that disrupts our ability to manage it.

A problem the AI safety crowd has faced is that people don't intuitively understand exponentials. In most of our lives, things move fairly linearly. The "one grain of rice, doubled per day" parable works because the King has no conception of how bad things are going to get and how fast.

So if there's a discontinuity in a linear curve, the average human will be surprised, but only once, and then the curve goes back to it's usual progress and things are mostly okay.

If a curve that looked linear-ish turns out to be exponential, that human may find themselves surprised if they're not paying attention, once the curve moves past an inflection point. And more surprised if the curve was hyperbolic. (If you are paying attention I'm not sure whether it matters whether the curve is exponential or hyperbolic)

If an exponential curve has a discontinuity, it's even more surprising. The average human is hit both with a curve arcing upward faster than they expected, and with that curve suddenly jumping upward.

Now, I'm not sure how relevant this is to Leaders of Industry in the AI world – presumably at least some are paying attention. I'm not actually sure how much attention I expect them to be paying (there's a lot of things to pay attention to, and if we're at the early stage of a hyperbolic curve it's a plausible mistake to assume it's linear and/or "only" exponential).

(I think it matters a lot more whether industry leaders are paying attention than whether Joe/Jane Public is. It seems like a major reason discontinuity matters (or whether DeepMind et all think it matters, or think other organizations think it matters) has to do with both arms races, and whether or not it's plausible that we end up in a multipolar scenario vs a singleton)

A lot of great points!

I think we can separate the arguments into about three camps, based on their purpose (though they (EDIT: whoops, forgot a don't) don't all cleanly sit in one camp):

• Arguments why progress might be generally fast: Hominid variation, Brain scaling.
• Arguments why a local advantage in AI might develop: Intelligence explosion, One algorithm, Starting high, Awesome AlphaZero.
• Arguments why a local advantage in AI could cause a global discontinuity: Deployment scaling, Train vs. test, Payoff thresholds, Human-competition threshold, Uneven skills.

These facts need to work together to get the thesis of a single disruptive actor to go through: you need there to be jumps in AI intelligence, you need them to be fairly large even near human intelligence, and you need those increases to translate into a discontinuous impact on the world. This framework helps me evaluate arguments and counterarguments - for example, you don't just argue against Hominid variation as showing that there will be a singularity, you argue against its more limited implications as well.

Bits I didn't agree with, and therefore have lots to say about:

Intelligence Explosion:

The counterargument seems pretty wishy-washy. You say: "Positive feedback loops are common in the world, and very rarely move fast enough and far enough to become a dominant dynamic in the world." How common? How rare? How dominant? Is global warming a dominant positive feedback loop because warming leads to increased water in the atmosphere which leads to more warming, and it's going to have a big effect on the world? Or is it none of those, because Earth won't get all that much warmer, because there are other well-understood effects keeping it in homeostasis?

More precisely, I think the argument from reference class that a positive feedback loop (or rather, the behavior that we approximate as a positive feedback loop) will be limited in time and space is hardly an argument at all - it practically concedes that the feedback loop argument works for the middle of the three camps above, but merely points out that it's not also an argument that intelligence will be important. A strong argument against the intelligence feedback hypothesis has to argue that a positive feedback loop is unlikely.

One can obviously respond by emphasizing that objects in the reference class you've chosen (e.g. tipping back too far in your chair and falling) don't generally impact the world, and therefore this is a reference class argument against AI impacting the world. But AI is not drawn uniformly from this reference class - the only reason we're talking about it is because it's been selected for the possibility of impacting the world. Failure to account for this selection pressure is why the strength of the argument seemed to change upon breaking it into parts vs. keeping it as a whole.

Deployment scaling:

We agree that slow deployment speed can "smooth out" a discontinuous jump in the state of the art into a continuous change in what people actually experience. You present each section as a standalone argument, and so we also agree that fast deployment speed alone does not imply discontinuous jumps.

But I think keeping things so separate misses the point that fast deployment is among the necessary conditions for a discontinuous impact. There's also risk, if we think of things separately, of not remembering these necessary conditions when thinking about historical examples. Like, we might look at the history of drug development, where drug deployment and adoption takes a few years, and costs falling to allow more people to access the treatment takes more years, and notice that even though there's an a priori argument for a discontinuous jump in best practices, peoples' outcomes are continuous on the scale of several years. And then, if we've forgotten about other necessary factors, we might just attribute this to some mysterious low base rate of discontinuous jumps.

Payoff thresholds:

The counterargument doesn't really hold together. We start ex hypothesi with some threshold effect in usefulness (e.g. good enough boats let you reach another island). Then you say that it won't cause a discontinuity in things we care about directly; people might buy better boats, but because of this producers will spend more effort making better boats and sell them more expensively, so the "value per dollar" doesn't jump. But this just assumes without justification that the production eats up all the value - why can't the buyer and the producer both capture part of the increase in value? The only way the theoretical argument seems to work is in equilibrium - which isn't what we care about.

Nuclear weapons are a neat example, but may be a misleading one. Nuclear weapons could have had half the yield, or twice the yield, without altering much about when they were built - although if you'd disagree with this, I'd be interested in in hearing about it. (Looking at your link, it seems like nuclear weapons were in fact more expensive per ton of TNT when they were first built - and yet they were built, which suggests there's something fishy about their fit to this argument).

Awesome AlphaZero:

I think we can turn this into a more general thesis: Research is often local, and often discontinuous, and that's important in AI. Fields whose advance seems continuous on the several-year scale may look jumpy on the six-month scale, and those jumps are usually localized to one research team rather than distributed. You can draw a straight line through a plot of e.g. performance of image-recognition AI, but that doesn't mean that at the times in between the points there was a program with that intermediate skill at image-recognition. This is important to AI if the scale of the jumps, and the time between them, allows one team to jump through some region (not necessarily a discontinuity) of large gain in effect and gain a global advantage.

The missing argument about strategy:

There's one possible factor contributing to the likelihood of discontinuity that I didn't see, and that's the strategic one. If people think that there is some level of advantage in AI that will allow them to have an important global impact, then they might not release their intermediate work to the public (so that other groups don't know their status, and so their work can't be copied), creating an apparent discontinuity when they decide to go public, even if 90% of their AI research would have gotten them 90% of the taking-over-the-world power.

This project (best read in the bolded link, not just in this post) seemed and still seems really valuable to me. My intuitions around "Might AI have discontinuous progress?" become a lot clearer once I see Katja framing them in terms of concrete questions like "How many past technologies had discontinuities equal to ten years of past progress?". I understand AI Impacts is working on an updated version of this, which I'm looking forward to.

Updated me quite strongly towards continuous takeoff (from a position of ignorance)

Seconding Rohin. 

I think this is basically the same nomination as the post "Arguments Against Fast Takeoff", it's all one conversation, but just wanted to nominate it to be clear.

Promoted to frontpage.