Comments on "AI 2027"

post by Randaly · 2025-04-11T20:32:34.419Z · LW · GW · 10 comments

Contents

10 comments

I find the decision to brand the forecast as "AI 2027" very odd. The authors do not in fact believe this; they explicitly give 2028, 2030, or 2033 for their median dates for a superhuman coder.

The point of this project was presumably to warn about a possible outcome; by the authors' own beliefs, their warning will be falsified immediately before it is needed.

When presenting predictions, forecasters always face tradeoffs regarding how much precision to present. Precise forecasting attracts attempts and motivates action; adding many concrete details produces a compelling story, stimulating discussion; this also involves falsifiable predictions. Emphasizing uncertainty avoids losing credibility when some parts of story inevitably fail; prevents overconfidence; and encourages more robust strategies that can work across a range of outcomes. But I can't think of any reason to only consider a single high precision story that you don't think is all that likely.

I think that the excessive precision is pretty important in this case: the current pace of AI R&D spending is unsustainable, so it matters exactly how much more progress is needed for superhuman coders.

***

I don't believe that METR's time horizons forecast is sufficiently strong evidence for a precise timeline:

1. *In general* AI competence relative to humans isn't ordered by time- AI's can complete some tasks that would take humans centuries, and can't complete other than would take humans seconds.
2. METR produced their analysis by taking a subset of activities: SWAA, HCAST, and RE-BENCH. SWAA has 66 tasks from 1-30 seconds; HCAST has 97 tasks from 1 minutes to 30 hours; and RE-BENCH has 7 tasks of 8 hours. SWAA and HCAST were created by METR for this research.
3. METR did not- and cannot- use an obvious/straightforward/canonical set of human tasks: current AI's cannot accomplish most human economic tasks, because most tasks do not solely involve interacting with computers.
4. METR created SWAA and HCAST *after* having observed AI progress. Their selection of tasks may have been unconsciously biased by knowing how difficult each task is for LLM's. (Albeit they also checked SWE-Bench Verified times and found a similar result within its tasks.)
5. The publicly shared Task Summaries from HCAST do not represent tasks separated solely by time. Instead, they represent *different* tasks, which are measured as taking different amounts of time, but which also have different difficulties. This undermines METR's attempt to provide a precise future forecast: the time taken for a task can be measured precisely, but its difficulty cannot. This poses two problems:
   1. METR provides examples of future tasks (e.g. "Optimize code for custom chip"); a forecast of when they will be solvable depends on METR's having accurately evaluated their difficulty, in addition to the time it takes humans to accomplish them.

   2. Regarding task difficulty, METR writes that "If this is the case, we may be underestimating the pace of model improvement." They seem to be viewing this as models becoming capable of solving more difficult problems *in addition* to time horizons increasing exponentially. However, in their dataset increased difficulty of tasks is correlated with time horizons- see section B.1.1; so increased capabilities might contribute to increasing measured time horizons by allowing the completion of tasks which were too difficult but not too long to accomplish previously.

      Two concrete examples: humans can read ~75 tokens per minute, so any task that GPT-2 could do that filled its entire 1024-token context length^[1] would be a 15 minute task; meanwhile, a leap of mathematical intuition takes a human only a few seconds, but is arguably still impossible for LLM's. Adding many tasks like these would make the calculated task length improvement slower.

6. It is plausible that the future projection for solvable tasks is correct. However, this is not a straightforward projection from past progress; instead, it is dependent on METR researchers' intuitions about the relative difficulties of tasks being correct.
7. Unfortunately, while I don't believe in it or its results, there is no better alternative; METR's time horizons forecast is pretty clearly the best available attempt at making a precise forecast of AI improvement.

***

I don't believe that the "benchmarks-and-gaps" model is the correct way to forecast future AI development:

1. "benchmarks-and-gaps" has the form of a Fermi estimate; I'm suspicious that this is how/why it was adopted. However, Fermi estimates are typically used in areas where estimates within an order of magnitude are useful- not to give a precise estimate. On a typical Fermi estimate reasoning, the authors should have rounded their timeline to one year, and instead written "AI 2026". Of course they were right to not do this- because this is a question where precision matters. However, taking a Fermi estimate's "2 months-2 decades" estimate and and producing a timeline of 2 years conveys excessive precision.
2. There is no particular reason to endorse the particular set of gaps chosen. The most prominent gap that I've seen discussed, the ability of LLMs to come up with new ideas or paradigms, wasn't included.
3. "benchmarks-and-gaps" has historically proven to be an unreliable way of forecasting AI development. The problem is that human intuitions about what capabilities are required for specific tasks aren't very good, and so more "gaps" are discovered once the original gaps have been passed.
4. I believe that the discussion of "Feedback loops: Working without externally provided feedback" considerably underestimates the difficulty involved: it claims that removing Best-of-K represents half of the difficulty. Eli writes:

 I and others have consistently been surprised by progress on easy-to-evaluate, nicely factorable benchmark tasks, while seeing some corresponding real-world impact but less than I would have expected. Perhaps AIs will continue to get better on checkable tasks in substantial part by relying on trying a bunch of stuff and seeing what works, rather than general reasoning which applies to more vague tasks. And perhaps I’m underestimating the importance of work that is hard to even describe as “tasks”.

Narrowly, labs often are optimizing against benchmarks, meaning that non-benchmarked progress is slower than you would otherwise measure. More broadly, it's plausible that important tasks or knowledge *can't* be benchmarked, even with a benchmark that maxes out METR's messiness metric. James C. Scott uses the word "legibility" to describe the process of making it possible for outsiders to gain understanding: creating simplified, standardized categories; imposing uniform systems of measurement; and favoring formal, official knowledge over practical, local knowledge. I like that term in this context because it emphasizes that LLM's trained on text and passing objectively measured benchmarks will have trouble with some "illegible" types of tasks that humans can do: think intuitive leaps, context-dependent judgments, or embodied skills.^[3]

***

I believe that the "AI 2027" scaling forecasts are implausible.

1. "AI 2027" forecasts that OpenAI's revenue will reach 140 billion in 2027. This considerably exceeds even OpenAI's own forecast, which surpasses 125 billion in revenue in 2029. I believe that the AI 2027 forecast is implausible.^[4]
   1. FutureSearch's estimate of paid subscribers for April 2025 was 27 million; the actual figure is 20 million. They justify high expected consumer growth with reference to the month-on-month increase in unpaid users from December 2024 -> February 2025. Data from Semrush replicates that increase, but also shows that traffic has since declined rather than continuing to increase.
   2. Looking at market size estimates, FutureSearch seems to implicitly assume that OpenAI will achieve a near-monopoly on Agents, the same way they have for Consumer subscriptions. Enterprise sales are significantly different from consumer signups, and OpenAI doesn't currently have a significant technical advantage.
   3. Current alignment techniques may be insufficient, not just to align AGI, but to align the independently acting Agents which are expected to make up 60% of OpenAI's revenue.
2. "AI 2027" uses an implausible forecast of compute/algorithm improvement past 2028. It assumes that each continues exponential progress, but at half the rate (so 2.35x compute/year, and 1.5x algorithmic improvement/year).
   1. Algorithmic improvements have been driven in part by exponential increases in spending (to hire researchers), and in part by exponential increases in compute. "AI 2027" indeed implies the decreased speed of algorithmic improvement is due to "the human research population...growing at a slower rate". However, technological improvements often outright plateau. This can be due to objective limits, or due to low-hanging fruit having been picked. For a concrete example, some of the most significant algorithmic improvements have been the changes from fp32 -> fp8 training; Nvidia Blackwell also improved support for fp4 training; further progress, if any, will have natural limits.
   2. Given their separate assumption of a 1.35x annual FLOPs increase, this implies a 1.7x annual increase in AI R&D spending- down from today's 2.6x.^[5] However, it's much more likely that AI R&D spending will instead drop by an order of magnitude or more: an "AI Winter". In the next section, I'll try running through some concrete numbers about future AI R&D spending.

***

Projected single-company R&D spending, in billions:^[6]

2024	2025	2026	2027	2028	2029	2030
4	10	27	70	183	311	528

Here are a few relevant points:

1. OpenAI's R&D spending *thus far* has been primarily driven by equity sales, not revenue. This is a typical model for a Silicon Valley startup, whose growth will eventually plateau at a profitable level. But OpenAI isn't like other startups.
2. OpenAI's leadership is unlike that of other startups: they are racing to an endpoint, not trying to build a sustainable business. Even if they could slow down R&D spending in order to guarantee that they can avoid bankruptcy, they aren't likely to do so.
3. OpenAI has a very different cost structure than other software companies. Typically, R&D costs are high but ~constant, whereas marginal costs are near-zero. OpenAI (and other AI companies) have high marginal costs due to the cost of the compute to run their models. Per OpenAI's 2024 budget, they had 4 billion in revenue, and spent 2 billion on inference, 400 million on hosting, 300 million on marketing, and probably 280 million on non-R&D salaries. (If salaries and compute had the same training:inference ratio.) For comparison, here are the costs of revenue for some major companies:

Meta	Salesforce	SAP	Oracle	Microsoft	Alphabet	OpenAI	Walmart	Ford
18%	23%	27%	29%	30%	42%	75%	75%	91%

1. For OpenAI in particular, turning a profit at all, even with zero R&D spending, will likely be impossible even if Altman were to stop racing. OpenAI has *also* signed a revenue sharing agreement with Microsoft that sends them another 20% of their revenue. Even worse, OpenAI and associated Project Stargate are already planning on racking up substantial amounts of debt even before 2026, let alone 2029. Realistically, it will be impossible for OpenAI to even pay the interest on its debt, let alone spend *anything* on R&D.^[7]
2. Even OpenAI reaching 140 billion in revenue in 2027, as per "AI 2027", won't come even close to making it profitable. It's not even clear whether OpenAI (or Anthropic, or xAI) can survive increasing their R&D spending at the current rate up to 2027.
3. OpenAI, or other AI companies, will likely have trouble raising further funding. "AI 2027" implies that it will control 400 billion in capex by 2027; raising approximately that much would be the largest financial transaction in human history, for only a fraction of a company which had never been profitable. OpenAI's *existing* fundraising has been explicable as normal investments;^[8] this would not be.^[9]
4. Other pre-existing tech companies are stronger contenders for continuing AI R&D. They don't have the revenue sharing agreement, and they have tens of billions in cash-on-hand and current profits to fund R&D without taking on debt. Meta isn't directly making revenue from its models; Amazon doesn't appear to be doing frontier AI research; Microsoft is pulling back from AI; which leaves Google as the likely winner if there's no takeoff by 2029.
5. If Google has a similar cost of revenue to OpenAI, it would need to make at least 1.2 trillion in new revenue to pay for the new R&D spending in 2028, or 2 trillion in new revenue in 2029. That's assuming no increase in Google's profits from now, and no increase in General and Administrative spending. That's also assuming that SC hasn't been reached. This all seems *highly* unlikely.^[10] (Note that Google currently makes 350 billion in revenue and is spending 50 billion on R&D.)^[11] 

One way to summarize this section: any forecast of AGI arrival is *highly* sensitive to the exact point during current scaling at which AGI is achievable- because current scaling cannot continue for long.^[12] If current growth rates can deliver superhuman coding capabilities by 2027, we might actually see it happen. However, if those same capabilities would require until 2028, then on some plausible financial models we wouldn't see AGI until the mid-2030's or later. It's therefore *extremely* unfortunate that (a) per the timelines discussion, there is no way to get much confidence in a forecast of the near future, and (b) the AI 2027 team's median timeline is longer that what they shared.

1. ^{^}

   Note that from GPT-2 to today, neglecting Llama 4 Scout, the length of the context window did increase with a doubling time of 6.6 months- very similar to METR's claimed 7 month doubling time.

2. ^{^}

   This is just repeating my first point.

3. ^{^}

   This point is also an objection to using METR's time horizon forecasting, as it's also based on benchmarks.

4. ^{^}

   Note: "AI 2027" chooses to call the leading lab "OpenBrain", but FutureSearch is explicit that they're talking about OpenAI.

5. ^{^}

   These numbers are sourced from Epoch, but their estimates don't  add up. 1.35x in computational performance, and 2.6x in training costs, should yield a 3.5x increase in training compute, not the calculated 4.7x. They are using two different datasets for the calculations, presumably one is more correct.

6. ^{^}

   Taking 2024's 4 billion level from OpenAI's spending; multiplying by 2.6 until 2029, then by 1.7, i.e. using the "AI 2027" numbers. Note that actual spending will be spikier than this: some years will see higher capex spending to build datacenters, other years may see lower data center construction, and instead are amortizing that capex + spending on electricity and salaries.

7. ^{^}

   I haven't even considered that General and Administrative spending might also scale with revenue.

8. ^{^}

   Total spending of 10-40 billion on R&D projects is high but not unprecedented- although spending of 40 billion per year is completely unprecedented. (The Manhattan project cost ~35 billion in today's dollars, over ~3 years; the largest private sector R&D effort ever, the 787, cost ~45 billion in today's dollars, over ~8 years.) The current spending of ~4 billion per year is pretty normal for a major R&D project. It's not even the highest current spending relevant to AI- Nvidia is spending ~13  billion on R&D this year.

9. ^{^}

   An *additional* problem here is that 2028 is approximately when data availability and data movement may become significant issues- this will slow down the future rate at which money buys effective compute, which will in turn dissuade people from funding further development.

10. ^{^}

    There are, to be fair, also some ways in which this forecast is overly negative. An AI Winter would drive down the cost of GPU's; this is only helpful for companies which don't own their own datacenters, i.e. Anthropic. (And if all companies increasingly use their own GPU designs, they may have trouble quickly integrating different GPU's into their setup.) Also, part of companies' current inference spend is on free users:  this can be eliminated entirely, at the cost of giving up on public mindshare/raising more money. Also, especially if other AI companies are going bankrupt, prices can be raised.

11. ^{^}

    Even Google would need to either sell significant equity or take on significant debt to fund AI development until 2028- its current cash and profits aren't nearly enough. Note that this halt would probably co-occur with a recession- given that hundreds of billions of dollars of AI capex would have suddenly just vanished. AI companies would first cut new capex spending, and may try to sell their GPU's in this scenario; this also plausibly kills off Nvidia and halts semiconductor improvements.

12. ^{^}

    It's also highly sensitive to exactly how long scaling can continue, of course. Scaling might plausibly continue to 2030 or beyond if national governments take over funding, for example.

10 comments

Comments sorted by top scores.