Arguments about fast takeoff

I believe that the disagreement is mostly about what happens before we build powerful AGI. I think that weaker AI systems will already have radically transformed the world, while I believe fast takeoff proponents think there are factors that makes weak AI systems radically less useful. This is strategically relevant because I'm imagining AGI strategies playing out in a world where everything is already going crazy, while other people are imagining AGI strategies playing out in a world that looks kind of like 2018 except that someone is about to get a decisive strategic advantage.

Huh, I think this is most useful framing of the Slow/Fast takeoff paradigm I've found. Normally when I hear about the slow takeoff strategy I think it comes with a vague sense of "and therefore, it's not worth worrying out because our usual institutions will be able to handle it."

I'm not sure if I'm convinced about the continuous takeoff, but it at least seems plausible. It still seems to me that recursive self improvement should be qualitatively different - not necessarily resulting in a "discontinuitous jump" but changing the shape of the curve.

The curve below is a bit sharper than I meant it to come out, I had some trouble getting photoshop to cooperate.

More thoughts to come. I agree with Qiaochu that this is a good "actually think for N minutes" kinda post.

↑ comment by paulfchristiano · 2018-02-26T17:11:15.453Z · LW(p) · GW(p)

The curve is hyperbolic (rather than exponential) because it already includes the loop (output --> better tech --> higher productivity --> more output), and RSI just seems like an important kind of better tech. So I'm not convinced RSI changes the shape of the curve (though it might be a more "local" form of improvement).

Replies from: Raemon

↑ comment by Raemon · 2018-02-26T20:16:56.373Z · LW(p) · GW(p)

This highlights perhaps a different thing, which is that... it'd seem very strange to me if the benefits from RSI exactly mapped onto an existing curve.

(This makes me think of an old Scott Alexander post reviewing the Great Stagnation, about how weird it is that all these economic curves seem pretty straight forwardly curvy. So this may be a domain where reality is normal and my intuitions are weird.)

There's another thing that bothers me too, which is the sheer damnable linearity of the economic laws. The growth of science as measured in papers/year; the growth of innovation as measured in patents; the growth of computation as measured by Moore's Law; the growth of the economy as measured in GDP. All so straight you could use them to hang up a picture. These lines don't care what we humans do, who we vote for, what laws we pass, what we invent, what geese that lay long-hanging fruit have or have not been killed. The Gods of the Straight Lines seem right up there with the Gods of the Copybook Headings in the category of things that tend to return no matter how hard you try to kick them away. They were saying back a few decades ago Moore's Law would stop because of basic physical restrictions on possible transistor size, and we worked around those, which means the Gods of the Straight Lines are more powerful than physics.

Replies from: habryka4, paulfchristiano

↑ comment by habryka (habryka4) · 2018-02-26T20:36:23.295Z · LW(p) · GW(p)

This is one of my major go-to examples of this really weird linear phenomenon:

150 years of a completely straight line! There were two world wars in there, the development of artificial fertilizer, the broad industrialization of society, the invention of the car. And all throughout the line just carries one, with no significant perturbations.

Replies from: jacobjacob, snowde, Alex_Altair

↑ comment by jacobjacob · 2018-03-07T21:09:34.491Z · LW(p) · GW(p)

I'm confused. Moore's law, GDP growth etc. are linear in log-space. That graph isn't. Why are these spaces treated as identical for the purposes of carrying the same important and confusing intuition? (E.g. I imagine one could find lots of weird growth functions that are linear in some space. Why are they all important?)

Replies from: habryka4

↑ comment by habryka (habryka4) · 2018-03-07T21:17:26.131Z · LW(p) · GW(p)

Hmm, in either case a function being linear in a very easy to describe space (i.e. log, log-log or linear) highlights that the relationship the function describes is extremely simple, and seems to be independent of most factors that vary drastically over time. I expect the world to be stochastic and messy, with things going up and down for random reasons and with things over time going up and down quite a bit for very local reasons, but these graphs do not seem to conform with that intuition in a way that I don't easily know how to reconcile.

↑ comment by snowde · 2018-03-31T01:06:16.522Z · LW(p) · GW(p)

It is remarkable how certain industries are protected against shocks, this reminds me of countries who easily bare financial crises as a result of being primary food exporters (NZ). What is even more interesting is the difficulting of fully getting rid of humans in an econmic endevour (latter part of the chart).

Regarding your point on linearity, this chart does show a linear decrease as of 1800. However lets say farming activity halved as a portion of all jobs in the first 100 years, in the latter hundred years it decreased with a multiple of 25. So in the first half it was transformed by X*1/2 and in the second half X*1/25. Is this still linear ? I was wondering about this phenomena, is there something like absolute and relative linearity or exponentiality ?

It seems that the percentage change then is exponential

↑ comment by Alex_Altair · 2021-12-18T05:03:07.083Z · LW(p) · GW(p)

I think this is just a sigmoid function, but mirrored over the y-axis. If you extended it farther into the past, it would certainly flatten out just below 100%. So I think it's just another example of how specific technologies are adopted in sigmoid curves, except in reverse, because people are dis-adopting manual farming.

(And I think the question of why tech grows in sigmoid curves is because that's the solution to the differential equation that models the fundamental dynamics of "grows proportional to position, up to a carrying capacity".)

↑ comment by paulfchristiano · 2018-02-27T04:56:55.341Z · LW(p) · GW(p)

I don't think that would be strange. It seems like a priori RSI should result in hyperbolic growth for the same reason that technology should be hyperbolic, since the dynamic is basically identical. You could argue that the relationship between human cognitive effort and machine cognitive effort is different than the relationship between human labor and machine labor, but that seems like it's lost in the noise. You could make an argument about discontinuities near 100% automation, but as far as I can currently tell those arguments fall apart on inspection.

Thanks for writing this! Your point about chimps vs. humans was new to me and I'll chew on it.

Because I think this post is important enough I'm gonna really go above and beyond here: I'm gonna set a timer and think for 5 minutes, with paper, before writing the rest of my comment.

Okay, that was fun. What I got out of reading your post twice (once before writing this and once during the 5-minute timer) was basically the following. Let me know if this is an accurate summary of your position:

AI researchers will try to optimize AI to be as useful as possible. There are many tradeoffs that need to be navigated in order to do this, e.g. between universality and speed, and the Pareto boundary describing all of these tradeoffs is likely to grow slowly / continuously. So we probably won't see fast / discontinuous Pareto improvements in available AIs.

I find this reasonably persuasive, but I'm not convinced the positions you've described as slow takeoff vs. fast takeoff are what other people mean when they talk about slow vs. fast takeoff.

I'd be interested to see you elaborate on this:

There is another reason I’m skeptical about hard takeoff from universality secret sauce: I think we already could make universal AIs if we tried (that would, given enough time, learn on their own and converge to arbitrarily high capability levels), and the reason we don’t is because it’s just not important to performance and the resulting systems would be really slow. This inside view argument is too complicated to make here and I don’t think my case rests on it, but it is relevant to understanding my view.

(Also, meta: I notice that setting the 5-minute timer had the effect of shifting me out of "look for something to disagree with" and into "try to understand what is even being claimed in the first place." Food for thought!)

Since there are some aspects of your version of "slow" takeoff that have been traditionally associated with the "fast" view, (e.g. rapid transformation of society), I'm tempted to try to tease apart what we care about in the slow vs fast discussion into multiple questions:

1) How much of a lead will one team need to have over others in order to have a decisive strategic advantage...

a) in wall clock time?

b) in economic doublings time?

2) How much time do we have to solve hard alignment problems...

a) in wall clock time?

b) in economic doublings time?

It seems that one could plausibly hold a "slow" view for some of these questions and a "fast" view for others.

For example, maybe Alice thinks that GDP growth will accelerate, and that there are gains to scale/centralization for AI projects, but that it will be relatively easy for human operators to keep AI systems under control. We could characterize her view as "fast" on 1a and 2a (because everything is happening so fast in wall clock time), "fast" on 1b (because of first-mover advantage), and "slow" on 2b (because we'll have lots of time to figure things out).

In contrast, maybe Bob thinks GDP growth won't accelerate that much until the time that the first AI system crosses some universality threshold, at which point it will rapidly spiral out of control, and that this all will take place many decades from now. We might call his view "fast" on 1a and 1b, and "slow" on 2a and 2b.

Curious to hear thoughts on this framework and whether there's a better one.

Just want to mention that this post caused me to update towards a continuous take-off. The humans and chimps section the most, as I hadn't noticed how much chimps weren't being optimised for general intelligence. And then the point that slow take-off means more change, sooner, was also key. I had been rounding slow take-off to a position which didn't really appreciate the true potential for capability gain, but your post showed me how to square that circle.

Fast take-off seems to me at the very least like a simplifying assumption, because I feel pretty unequipped to predict what will happen in the space before 'strongly superintelligent AGI' but after 'radical change due to AGI'. Would be interested to hear anyone's suggestions for reasoning about what that world will look like.

This is tangential, but I wanted to note that, a slow takeoff is, at least in some ways, a worse scenario than fast takeoff in terms of AI risk.

Designing an aligned AI (probably) involves trade-offs between safety and capability. In a fast takeoff scenario, the FAI "only" has to be capable enough to prevent UFAI from ever coming into being, in order for AI risk to be successfully mitigated. The most powerful potentially adversarial agents in its environment would be mere humans (and, yes, humans should be counted as somewhat adversarial since they are the source of the UFAI risk). In a (sufficiently) slow takeoff scenario, there will be a number of competing superintelligences, and it seems likely that at least some of them will not be aligned. This means that the FAIs have to be capable enough to effectively compete with UFAIs that are already deployed (while remaining safe).

Epistemics: Yes, it is sound. Not because of claims (they seem more like opinions to me), but because it is appropriately charitable to those that disagree with Paul, and tries hard to open up avenues of mutual understanding.

Valuable: Yes. It provides new third paradigms that bring clarity to people with different views. Very creative, good suggestions.

Should it be in the Best list?: No. It is from the middle of a conversation, and would be difficult to understand if you haven't read a lot about the 'Foom debate'.

Improved: The same concepts rewritten for a less-familiar audience would be valuable. Or at least with links to some of the background (definitions of AGI, detailed examples of what fast takeoff might look like and arguments for its plausibility).

Followup: More posts thoughtfully describing positions for and against, etc. Presumably these exist, but i personally have not read much of this discussion in the 2018-2019 era.

I imagine the "secret sauce" line of thinking as "we are solving certain problems in the wrong complexity class". Changing complexity class of an algorithm introduces a discontinuity; when near a take-off, then this discontinuity can get amplified into a fast take-off. The take-off can be especially fast if the compute hardware is already sufficient at the time of the break-through.

In other words: In order to expect a fast take-off, you only need to assume that the last crucial sub-problem for recursive self-improvement / explosion is done in the wrong complexity class prior to the discovery of a good algorithm.

For strong historical precedents, I would look for algorithmic advances that improved empirical average complexity class, and at the same time got a speed-up of e.g. 100 x on problem instances that were typical prior to the algorithmic discovery (so Strassen matrix-multiply is out).

For weaker historical precedent, I would look for advances that single-handedly made the entire field viable -- that is, prior to the advance one had a tiny community caring about the problem; post the advance, the field (e.g. type of data analysis) became viable at all (hence, very limited commercial / academic interest in the subfield prior to its breakthrough). I think that this is meaningful precedent because people optimize for expected pay-off, and it is sometimes surprising that some small-but-crucial-if-possible subproblem can be solved at all (reasonable quickly)!

And I do believe that there are many parts of modern ML that are in the wrong complexity class (this does not mean that I could do better, nor that I necessarily expect an improvement or even discontinuous jump in usefulness).

There seems to be a stronger version of the (pro-fast-takeoff) chimp vs human argument, namely average-IQ humans vs von Neumann: "IQ-100 humans have brains roughly the same size as von Neumann, but are much worse at making technology." Does your counterargument work against that version too? (I'm not sure I fully understand it.)

I like this post, but I think it's somewhat misleading to call your scenario a "slow takeoff". To my mind, a "slow takeoff" evokes an image from non-singularitarian science fiction where you have human-level robots running around and they've been running around for decades if not centuries: that is, a very slow and gradual development that gives society and institutions plenty of time to adapt. But your version is clearly not this, since you are talking on the timescale of a few years, and note yourself that time will be of the essence even with a "slow" takeoff:

If takeoff is slow: it will become quite obvious that AI is going to transform the world well before we kill ourselves, we will have some time to experiment with different approaches to safety, policy-makers will have time to understand and respond to AI, etc. But this process will take place over only a few years, and the world will be changing very quickly, so we could easily drop the ball unless we prepare in advance.

You also note that much of the safety community seems to believe in a fast takeoff, in disagreement with you. I don't know whether you're including me there, but I've previously talked about a takeoff on the scale of a few years being a fast one, since to me a "fast takeoff" is one where there's little time for existing institutions to prepare and respond adequately, and a few years still seems short enough to meet that criteria.

I'd prefer to use some term like "moderate takeoff" for the scenario that you're talking about.

[edit: sorry if I seem like I'm piling on with the terminology, I first wrote this comment and only then read the other comments and saw that they ~all brought up the same thing.]

[Note that I am in no way an expert on strategy, probably not up to date with the discourse, and haven't thought this through. I also don't disagree with your conclusions much.]

[Also note that I have a mild feeling that you engage with a somewhat strawman version of the fast-takeoff line of reasoning, but have trouble articulating why that is the case. I'm not satisfied with what I write below either.]

These possible arguments seem not included in your list. (I don't necessarily think they are good arguments. Just mentioning whatever intuitively seems like it could come into play.)

Idiosyncrasy of recursion. There might be a qualitative difference between universality across economically-incentivized human-like domains, and universality extended to self-improvement from the point of view of a self-improving AI, rather than human-like work on AI. In this case recursive self-improvement looks more like a side effect than mainstream linear progress.

Actual secrecy. Some group might actually pull off being significantly ahead and protecting their information from leaking. There are incentives to do this. Related: Returns to non-scale. Some technologies might be easier to develop by a small or medium sized well-coordinated group, rather than a global/national ecosystem. This means there's a selection effect for groups which stay somewhat isolated from the broader economy, until significantly ahead.

Non-technological cruxes. The ability to extract high quality AI research from humans is upstream of technological development, and an early foom loop might route through a particular configuration of researcher brains and workflow. However, humans are not fungible and there might be strange non-linear progress achieved by this. This consideration seems historically more important for projects that really push the limits of human capability, and an AGI seems like such a project.

Nash equilibria. The broader economy might random-walk itself into a balance of AI technologies which actively hinders optimizing for universality, e.g. by producing only certain kinds of hardware. This means it's not enough to argue that at some point researchers will realize the importance of AGI, but you have to argue they will realize this before the technological/economic lock-in occurs.

This post lead me to visualise the leadup to AGI more clearly, and changed how I talk about it in conversations. I think that while Paul's arguments were key, Ray's graph in the comments [LW(p) · GW(p)] is the best summary of my main update. Also I think that the brief comments from Eliezer and Nate [LW · GW] would be helpful if included.

I first interpreted your operationalization of slow take off to mean something that is true by definition (assuming the economy is strictly increasing).

I assume how you wanted me to interpret it is that the first 4 year doubling interval is disjoint from the first 1 year doubling interval. (the 4 year one ends before the 1 year one starts.)

I really liked this; despite its length, it's very readable, and helped move forward the discussion on this important topic with some simple ideas (to pick two: the graph was great, and the chimp argument seems to have surprised many people including me). For these reasons, I've curated this post.

(In future I would be happy to copy over the whole content of such a linked post, to encourage users to read the full arguments in detail. Let me know if there's any way we can help with that.)

I think this post is about an important point, but I notice myself wanting to nominate it more for Raemon's diagram [LW(p) · GW(p)] that it sparked than Paul's post itself. 

I think the arguments in this post have been one of the most important pieces of conceptual progress made in safety within the last few years, and have shifted a lot of people's opinions significantly.

I just came here to point out that even nuclear weapons were a slow takeoff in terms of their impact on geopolitics and specific wars. American nuclear attacks on Hiroshima and Nagasaki were useful but not necessarily decisive in ending the war on Japan; some historians argue that the Russian invasion of Japanese-occupied Manchuria, the firebombing of Japanese cities with massive conventional bombers, and the ongoing starvation of the Japanese population due to an increasingly successful blockade were at least as influential in the Japanese decision to surrender.

After 1945, the American public had no stomach for nuclear attacks on enough 'enemy' civilians to actually cause large countries like the USSR or China to surrender, and nuclear weapons were too expensive and too rare to use them to wipe out large enemy armies -- the 300 nukes America had stockpiled at the start of the Korean War in 1950 would not necessarily have been enough to kill the 3 million dispersed Chinese soldiers who actually fought in Korea, let alone the millions more who would likely have volunteered to retaliate against a nuclear attack.

The Soviet Union had a similarly-sized nuclear stockpile and no way to deliver it to the United States or even to the territory of key US allies; the only practical delivery vehicle at that time was via heavy bomber, and the Soviet Union had no heavy bomber force that could realistically penetrate western air defense systems -- hence the Nike anti-aircraft missiles rusting along ridgelines near the California coast and the early warning stations dotting the Canadian wilderness. If you can shoot their bombers down before they can reach your cities, then they can't actually win a nuclear war against you.

Nukes didn't become a complete gamechanger until the late 1950s, when the increased yields from hydrogen bombs and the increased range from ICBMs created a truly credible threat of annihilation.

My thoughts on the "Humans vs. chimps" section (which I found confusing/unconvincing):

Chimpanzees have brains only ~3x smaller than humans, but are much worse at making technology (or doing science, or accumulating culture…). If evolution were selecting primarily or in large part for technological aptitude, then the difference between chimps and humans would suggest that tripling compute and doing a tiny bit of additional fine-tuning can radically expand power, undermining the continuous change story.

But chimp evolution is not primarily selecting for making and using technology, for doing science, or for facilitating cultural accumulation.

For me the main takeaway of the human vs. chimp story to be information about the structure of mind space, namely that there are discontinuities in terms of real world consequences. 

Evolution changes continuously on the narrow metric it is optimizing, but can change extremely rapidly on other metrics. For human technology, features of the technology that aren’t being optimized change rapidly all the time. When humans build AI, they will be optimizing for usefulness, and so progress in usefulness is much more likely to be linear.

I don't see how "humans are optimizing AI systems for usefulness" undermines the point about mind space - if there are discontinuities in capabilities / resulting consequences, I don't see how optimizing for capabilities / consequences makes things any more continuous. 

Also, there is a difference between "usefulness" and (say) "capability of causing human extinction", just as there is a difference between "inclusive genetic fitness" and "intelligence". Cf. it being hard to get LLMs do what you want them to do, and the difference between the publicity* of ChatGPT and other GPT-3 models is more about usability and UI instead of the underlying capabilities.

*Publicity is a different thing from usefulness. Lacking a more narrow definition of usefulness, I still would argue that to many people ChatGPT is more useful than other GPT models.

A few thoughts:

• Even if we could theoretically double output for a product, it doesn't mean that there will be sufficient demand for it to be doubled. This potential depends on how much of the population already has thing X
• Even if we could effectively double our workforce, if we are mostly replacing low-value jobs, then our economy wouldn't double
• Even if we could say halve the cost of producing robot workers, that might simply result in extra profits for a company instead of increasing the size of the economy
• Even if we have a technology that could double global output, it doesn't mean that we could or would deploy it in that time, especially given that companies are likely to be somewhat risk adverse and not scale up as fast as possible as they might be worried about demand. This is the weakest of the four arguments in my opinion, which is why it is last.

So economic progress may not accurately represent technological progress, meaning that if we use this framing we may get caught up in a bunch of economic debates instead of debates about capacity.

Updated me quite strongly towards continuous takeoff (from a position of ignorance). (I would also nominate the AI Impacts post, but I don't think it ever got cross-posted.)

Here's an argument why (at least somewhat) sudden takeoff is (at least somewhat) plausible.

Supposing:

(1) At some point P, AI will be as good as humans at AI programming (grandprogramming, great-grandprogramming, ...) by some reasonable standard, and less than a month later, a superintelligence will exist.

(2) Getting to point P requires AI R&D effort roughly comparable to total past AI R&D effort.

(3) In an economy growing quickly because of AI, AI R&D effort increases by at least the same factor as general economic growth.

Then:

(4) Based on (3), if there's a four year period during which economic growth is ten times normal because of AI (roughly corresponding to a four year doubling period), then AI R&D effort during that period is also at least ten times normal.

(5) Because 4*10=40 and because of additional R&D effort between now and the start of the four year period, total AI R&D effort between now and the end of such a period would be at least roughly comparable to total AI R&D effort until now.

(6) Therefore, based on (2) and (1), at most a month after the end of the first four year doubling period, a superintelligence will exist.

I think (1) is probable and (2) is plausible (but plausibly false). I'm more confused about (3), but it doesn't seem wrong.

There's a lot of room to doubt as well as sharpen this argument, but I hope the intuition is clear. Something like this comes out if I introspect on why it feels easier to coherently imagine a sudden than a gradual takeoff.

If there's a hard takeoff claim I'm 90% sure of, though, the claim is more like (1) than (6); more like "superintelligence comes soon after an AI is a human-level programmer/researcher" than like "superintelligence comes soon after AI (or some other technology) causes drastic change". So as has been said, the difference of opinion isn't as big as it might at first seem.

Promoted to frontpage.