What are the strongest arguments for very short timelines?

Here's the structure of the argument that I am most compelled by (I call it the benchmarks + gaps argument), I'm uncertain about the details.

1. Focus on the endpoint of substantially speeding up AI R&D / automating research engineering. Let's define our timelines endpoint as something that ~5xs the rate of AI R&D algorithmic progress (compared to a counterfactual world with no post-2024 AIs). Then make an argument that ~fully automating research engineering (experiment implementation/monitoring) would do this, along with research taste of at least the 50th percentile AGI company researcher (experiment ideation/selection).
2. Focus on REBench since it's the most relevant benchmark. REBench is the most relevant benchmark here, for simplicity I'll focus on only this though for robustness more benchmarks should be considered.
3. Based on trend extrapolation and benchmark base rates, roughly 50% we'll saturate REBench by end of 2025.
4. Identify the most important gaps between saturating REBench and the endpoint defined in (1). The most important gaps between saturating REBench and achieving the 5xing AI R&D algorithmic progress are: (a) time horizon as measured by human time spent (b) tasks with worse feedback loops (c) tasks with large codebases (d) becoming significantly cheaper and/or faster than humans. There are some more but my best guess is that these 4 are the most important, should also take into account unknown gaps.
5. When forecasting the time to cross the gaps, it seems quite plausible that we get to the substantial AI R&D speedup within a few years after saturating REBench, so by end of 2028 (and significantly earlier doesn't seem crazy).
   1. This is the most important part of the argument, and one that I have lots of uncertainty over. We have some data regarding the "crossing speed" of some of the gaps but the data are quite limited at the moment. So there are a lot of judgment calls needed and people with strong long timelines intuitions might think the remaining gaps will take a long time to cross without this being close to falsified by our data.
   2. This is broken down into "time to cross the gaps at 2024 pace of progress" -> adjusting based on compute forecasts and intermediate AI R&D speedups before reaching 5x.
6. From substantial AI R&D speedup to AGI. Once we have the 5xing AIs, that's potentially already AGI by some definitions but if you have a stronger one, the possibility of a somewhat fast takeoff means you might get it within a year or so after.

One reason I like this argument is that it will get much stronger over time as we get more difficult benchmarks and otherwise get more data about how quickly the gaps are being crossed.

I have a longer draft which makes this argument but it's quite messy and incomplete and might not add much on top of the above summary for now. Unfortunately I'm prioritizing other workstreams over finishing this at the moment. DM me if you'd really like a link to the messy draft.

I think the gaps between where we are and human-level (and broadly but not precisely human-like) cognition are smaller than they appear. Modest improvements in to-date neglected cognitive systems can allow LLMs to apply their cognitive abilities in more ways, allowing more human-like routes to performance and learning. These strengths will build on each other nonlinearly (while likely also encountering unexpected roadblocks). 

Timelines are thus very difficult to predict, but ruling out very short timelines based on averaging predictions without gears-level models of fast routes to AGI would be a big mistake. Whether and how quickly they work is an empirical question.

One blocker to taking short timelines seriously is the belief that fast timelines mean likely human extinction. I think they're extremely dangerous but that possible routes to alignment also exist - but that's a separate question.

I also think this is the current default path, or I wouldn't describe it. 

I think my research career using deep nets and cognitive architectures to understand human cognition is pretty relevant for making good predictions on this path to AGI. But I'm biased [LW(p) · GW(p)], just like everyone else.

Anyway, here's very roughly why I think the gaps are smaller than they appear. 

Current LLMs are like humans with excellent:

• language abilities,
• semantic memory
• working memory

They can now do almost all short time-horizon tasks that are framed in language better than humans. And other networks can translate real-world systems into language and code, where humans haven't already done it.

But current LLMs/foundation models are dramatically missing some human cognitive abilities:

• Almost no episodic memory for specific important experiences
• No agency - they do only what they're told
• Poor executive function (self-management of cognitive tasks)
  • Relatedly, bad/incompetent at long time-horizon tasks.
• And zero continuous learning (and self-directed learning)
  • Crucial for human performance on complex tasks

Those lacks would appear to imply long timelines.

But both long time-horizon tasks and self-directed learning are fairly easy to reach. The gaps are not as large as they appear.

Agency is as simple as repeatedly calling a prompt of "act as an agent working toward goal X; use tools Y to gather information and take actions as appropriate". The gap between a good oracle and an effective agent is almost completely illusory.

Episodic memory is less trivial, but still relatively easy to improve from current near-zero-effort systems. Efforts from here will likely build on LLMs strengths. I'll say no more publicly; DM me for details. But it doesn't take a PhD in computational neuroscience to rederive this, which is the only reason I'm mentioning it publicly. More on infohazards later.

Now to the capabilities payoff: long time-horizon tasks and continuous, self-directed learning. 

Long time-horizon task abilities are an emergent product of episodic memory and general cognitive abilities. LLMs are "smart" enough to manage their own thinking; they  don't have instructions or skills to do it.  o1 appears to have those skills (although no episodic memory which is very helpful in managing multiple chains of thought), so similar RL training on Chains of Thought is probably one route achieving those.

Humans do not mostly perform long time-horizon tasks by trying them over and over. They either ask someone how to do it, then memorize and reference those strategies with episodic memory; or they perform self-directed learning, and pose questions and form theories to answer those same questions.

Humans do not have or need "9s of reliability" to perform long time-horizon tasks. We substitute frequent error-checking and error-correction. We then learn continuously on both strategy (largely episodic memory) and skills/habitual learning (fine-tuning LLMs already provides a form of this habitization of explicit knowledge to fast implicit skills).

Continuous, self-directed learning is a product of having any type of new learning (memory), and using some of the network/agents' cognitive abilities to decide what's worth learning. This learning could be selective fine-tuning (like o1s "deliberative alignment), episodic memory, or even very long context with good access as a first step. This is how humans master new tasks, along with taking instruction wisely. This would be very helpful for mastering economically viable tasks, so I expect real efforts put into mastering it.

Self-directed learning would also be critical for an autonomous agent to accomplish entirely novel tasks, like taking over the world.

This is why I expect "Real AGI [LW · GW]" that's agentic and learns on its own, and not just transformative tool "AGI" within the next five years (or less). It's easy and useful, and perhaps the shortest path to capabilities (as with humans teaching themselves).

If that happens, I don't think we're necessarily doomed, even without much new progress on alignment (although we would definitely improve our odds!). We are already teaching LLMs mostly to answer questions correctly and to follow instructions. As long as nobody gives their agent an open-ended top-level goal like "make me lots of money", we might be okay. Instruction-following AGI is easier and more likely than value aligned AGI [LW · GW] although I need to work through and clarify why I find this so central. I'd love help.

Convincing predictions are also blueprints for progress. Thus, I have been hesitant to say all of that clearly. 

I said some of this at more length in Capabilities and alignment of LLM cognitive architectures [LW · GW] and elsewhere. But I didn't publish it in my previous neuroscience career nor have I elaborated since then.

But I'm increasingly convinced that all of this stuff is going to quickly become obvious to any team that sits down and starts thinking seriously about how to get from where we are to really useful capabilities. And more talented teams are steadily doing just that.

I now think it's more important that the alignment community takes short timelines more seriously, rather than hiding our knowledge in hopes that it won't be quickly rederived. There are more and more smart and creative people working directly toward AGI. We should not bet on their incompetence.

There could certainly be unexpected theoretical obstacles. There will certainly be practical obstacles. But even with expected discounts for human foibles and idiocy and unexpected hurdles, timelines are not long. We should not assume that any breakthroughs are necessary, or that we have spare time to solve alignment adequately to survive.

I've been arguing for 2027-ish AGI for several years now. I do somewhat fall into the annoying category of refusing to give my full details for believing this (publicly). I've had some more in-depth discussions about this privately.

One argument I have been making publicly is that I think Ajeya's Bioanchors report greatly overestimated human brain compute. I think a more careful reading of Joe Carlsmith's report that hers was based on supports my own estimates of around 1e15 FLOPs.

Connor Leahy makes some points I agree with in his recent Future of Life interview. https://futureoflife.org/podcast/connor-leahy-on-why-humanity-risks-extinction-from-agi/

Another very relevant point is that recent research on the human connectome shows that long-range connections (particularly between regions of the cortex) are lower bandwidth than was previously thought. Examining this bandwidth in detail leads me to believe that efficient decentralized training should be possible. Even with considering that training a human brain equivalent model would require 10000x parallel brain equivalents to have a reasonable training time, the current levels of internet bandwidth between datacenters worldwide should be more than sufficient.

Thus, my beliefs are strongly pointint towards: "with the right algorithms we will have more than good enough hardware and more than sufficient data. Also, those algorithms are available to be found, and are hinted at by existing neuroscience data." Thus, with AI R&D accelerated research on algorithms, we should expect rapid progress on peak capabilities and efficiency which doesn't plateau at human-peak-capability or human-operation-speed. Super-fast and super-smart AGI within a few months of full AGI, and rapidly increasing speeds of progress leading up to AGI.

If I'm correct, then the period of time from 2026 to 2027 will contain as much progress on generally intelligent systems as all of history leading up to 2026. ASI will thus be possible before 2028.

Only social factors (e.g. massively destructive war or unprecedented international collaboration on enforcing an AI pause) will change these timelines.

Further thoughts here: A path to human autonomy [LW · GW]

An AGI broadly useful for humans needs to be good at general tasks for which currently there is no way of finding legible problem statements (where System 2 reasoning is useful) with verifiable solutions. Currently LLMs are slightly capable at such tasks, and there are two main ways in which they become more capable, scaling and RL.

Scaling is going to continue rapidly showing new results at least until 2026-2027, probably also 2028-2029 [LW · GW]. If there's no AGI or something like a $10 trillion AI company by then, there won't be a trillion dollar training system and the scaling experiments will fall back to the rate of semiconductor improvement.

Then there's RL, which as o3 demonstrates applies to LLMs as a way of making them stronger and not merely eliciting capabilities formed in pretraining. But it only works directly around problem statements with verifiable solutions, and it's unclear how to generate them for more general tasks or how far will the capabilities generalize from the training problems that are possible to construct in bulk. (Arguably self-supervised learning is good at instilling general capabilities because the task of token prediction is very general, it subsumes all sorts of things. But it's not legible.) Here too scale might help with generalization stretching further from the training problems, and with building verifiable problem statements for more general tasks, and we won't know how much it will help until the experiments are done.

So my timelines are concentrated on 2025-2029, after that the rate of change in capabilities goes down. Probably 10 more years of semiconductor and algorithmic progress after that are sufficient to wrap it up though, so 2040 without AGI seems unlikely.

I have seen a poll asking "when will indefinite lifespans be possible?", and Eric Drexler answered "1967", because that was when cryonic suspension first became available. 

Similarly, I think we've had AGI at least since 2022, because even then, ChatGPT was an intelligence, and it was general, and it was artificial. 

(To deny that the AIs we have now have general intelligence, I think one would have to deny that most humans have general intelligence, too.)

So that's my main reason for very short timelines. We already crossed the crucial AGI threshold through the stupid serendipity of scaling up autocomplete, and now it's just a matter of refining the method, and attaching a few extra specialized modules. 

Summary: Superintelligence in January-August, 2026.  Paradise or mass death, shortly thereafter.

This is the shortest timeline proposed in these answers so far.  My estimate (guess) is that there's only 20% of this coming true, but it looks feasible as of now.  I can't honestly assert it as fact, but I will say it is possible.

It's a standard intelligence explosion scenario: with only human effort, the capacities of our AIs double every two years.  Once AI gets good enough to do half the work, we double every one year.  Once we've done that for a year, our now double-smart AIs help us double in six months.  Then we double in three months, then six weeks.... to perfect ASI software, running at the the limits of our hardware, in a finite time.  Then the ASI does what it wants, and we suffer what we must.

I hear you say "Carl, this argument is as old as the hills.  It hasn't ever come true, why bring it up now?"  The answer is, I bring it up because it seems to be happening.  

• For at least six months now, we've had software assistants that can roughly double the productivity of software development.  At least at the software company where I work, people using AI (including me) are seeing huge increases in effectiveness.  I assume the same is true of AI software development, and that AI labs are using this technology as much as they can.  
• In the last few months, there's been a perceptible increase in the speed of releases of better models.  I think this is a visible (subjective, debatable) sign of an intelligence explosion starting up.  

So I think we're somewhere in the "doubling in one year" phase of the explosion.  If we're halfway through that year, the singularity is due in August 2026.  If we're near the end of that year, the date is January 2026.

There are lots of things that might go wrong with this scenario, and thereby delay the intelligence explosion.  I will mention a few, so you don't have to.

First, the government might stop the explosion, by banning AI being used for the development of AI.  Or perhaps the management of all major AI labs will spontaneously not be so foolish as to.  This will delay the problem for an unknown time.

Second, the scenario has an extremely naive model of intelligence explosion microeconomics.  It assumes that one doubling of "smartness" produces one doubling of speed.  In Yudkowsky's original scenario, AIs were doing all the work of development, and this might be a sensible assumption.  But what has actually happened is that successive generations of AI can handle larger and larger tasks, before they go off the rails.  And they can handle these tasks far faster than humans.  So the way we work now is that we ask the AI to do some small task, and bang, it's done.  It seems like testing is showing that current AIs can do things that would take a human up to an hour or two.  Perhaps the next generation will be able to do tasks up to four hours.  The model assumes that this allows a twofold speedup, then fourfold, etc.  But this assumption is unsupported.

Third, the scenario assumes that near-term hardware is sufficient for superintelligence.  There isn't time for the accelerating loop to take effect in hardware.  Even if design was instant, the physical processes of mask making, lithography, testing, yield optimization and mass production take more than a year.  The chips that the ASI will run on in mid-2026 have their design almost done now, at the end of 2024.  So we won't be able to get to ASI, if the ASI requires many orders of magnitude more FLOPs than current models. Instead, we'll have to wait until the AI designs future generations of semiconductor technology.  This will delay matters by years (if using humans to build things) or hours (if using nanotechnology.)

(I don't think the hardware limit is actually much of a problem; AIs have recently stopped scaling in numbers of parameters and size of training data.  Good engineers are constantly figuring out how to pack more intelligence into the same amount of computation.  And the human brain provides an existence proof that human-level intelligence requires much less training data.  Like Mr. Helm-Burger above, I think human-equivalent cognition is around 10^15 Flops.  But reasonable people disagree with me.)

I have a meta-view on this that you might think falls into the bucket of "feels intuitive based on the progress so far". To counter that, this isn't pure intuition. As a side note I  don't believe that intuitions should be dismissed and should be at least a part of our belief updating process. 

I can't tell you the fine details of what will happen and I'm suspicious of anyone who can because a) this is a very complex system b) no-one really knows how LLMs work, how human cognition works, or what is required for an intelligence takeoff. 

However, I can say that for the last decade or so most predictions of AI progress  have been on consistently longer timescales than what has happened. Things are happening quicker than the experts believe they will happen. Things are accelerating. 

I also believe that there are many paths to AGI, and that given the amount of resources currently being put into the search for one of those paths, they will be found sooner rather than later. 

The intelligence takeoff is already happening. 

It's worthy of a (long) post, but I'll try to summarize. For what it's worth, I'll die on this hill.

General intelligence = Broad, cross-domain ability and skills.

Narrow intelligence = Domain-specific or task-specific skills.

The first subsumes the second at some capability threshold.

My bare bones definition of intelligence: prediction. It must be able to consistently predict itself & the environment. To that end it necessarily develops/evolves abilities like learning, environment/self sensing, modeling, memory, salience, planning, heuristics, skills, etc. Roughly what Ilya says about token prediction necessitating good-enough models to actually be able to predict that next token (although we'd really differ on various details)

Firstly, it's based on my practical and theoretical knowledge of AI and insights I believe to have had into the nature of intelligence and generality for a long time. It also includes systems, cybernetics, physics, etc. I believe a holistic view helps inform best w.r.t. AGI timelines. And these are supported by many cutting edge AI/robotics results of the last 5-9 years (some old work can be seen in new light) and also especially, obviously, the last 2 or so.

Here are some points/beliefs/convictions I have for thinking AGI for even the most creative goalpost movers is basically 100% likely before 2030, and very likely much sooner. A fast takeoff also, understood as the idea that beyond a certain capability threshold for self-improvement, AI will develop faster than natural, unaugmented humans can keep up with.

It would be quite a lot of work to make this very formal, so here are some key points put informally:

- Weak generalization has been already achieved. This is something we are piggybacking off of already, and there is meaningful utility since GPT-3 or so. This is an accelerating factor.

- Underlying techniques (transformers , etc) generalize and scale. 

- Generalization and performance across unseen tasks improves with multi-modality.

- Generalist models outdo specialist ones in all sorts of scenarios and cases.

- Synthetic data doesn't necessarily lead to model collapse and can even be better than real world data.

- Intelligence can basically be brute-forced it looks like, so one should take Kurzweil *very* seriously (he tightly couples his predictions to increase in computation).

- Timelines shrunk massively across the board for virtually all top AI names/experts in the last 2 years. Top Experts were surprised by the last 2 years.

- Bitter Lesson 2.0.: there are more bitter lessons than Sutton's, which are that all sorts of old techniques can be combined for great increases in results. See the evidence in papers linked below.

- "AGI" went from a taboo "bullshit pursuit for crackpots", to a serious target of all major labs, publicly discussed. This means a massive increase in collective effort, talent, thought, etc. No more suppression of cross-pollination of ideas, collaboration, effort, funding, etc.

- The spending for AI only bolsters, extremely so, the previous point. Even if we can't speak of a Manhattan Project analogue, you can say that's pretty much what's going on. Insane concentrations of talent hyper focused on AGI. Unprecedented human cycles dedicated to AGI.

- Regular software engineers can achieve better results or utility by orchestrating current models  and augmenting them with simple techniques(RAG, etc). Meaning? Trivial augmentations to current models increase capabilities - this low hanging fruit implies medium and high hanging fruit (which we know is there, see other points). 

I'd also like to add that I think intelligence is multi-realizable, and generality will be considered much less remarkable soon after we hit it and realize this than some still think it is.

Anywhere you look: the spending, the cognitive effort, the (very recent) results, the utility, the techniques...it all points to short timelines.

In terms of AI papers, I have 50 references or so I think support the above as well. Here are a few:

SDS : See it. Do it. Sorted Quadruped Skill Synthesis from Single Video Demonstration, Jeffrey L., Maria S., et al. (2024).

DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning, Zhenyu J., Yuqi X., et in. (2024).

One-Shot Imitation Learning, Duan, Andrychowicz, et al. (2017).

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, Finn et al., (2017).

Unsupervised Learning of Semantic Representations, Mikolov et al., (2013).

A Survey on Transfer Learning, Pan and Yang, (2009).

Zero-Shot Learning - A Comprehensive Evaluation of the Good, the Bad and the Ugly, Xian et al., (2018).

Learning Transferable Visual Models From Natural Language Supervision, Radford et al., (2021).

Multimodal Machine Learning: A Survey and Taxonomy, Baltrušaitis et al., (2018).

Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine, Harsha N., Yin Tat Lee et al. (2023).

A Vision-Language-Action Flow Model for General Robot Control, Kevin B., Noah B., et al. (2024).

Open X-Embodiment: Robotic Learning Datasets and RT-X Models, Open X-Embodiment Collaboration, Abby O., et al. (2023).