romeo's Shortform

post by romeo · 2025-04-27T04:00:46.212Z · LW · GW · 8 comments

Contents

8 comments

8 comments

Comments sorted by top scores.

comment by romeo · 2025-04-27T04:00:46.211Z · LW(p) · GW(p)

Maybe i'm in an echo chamber or have just had my head in the sand while working on AI 2027, but now that i've been paying attention to AI safety for almost 2 years and seen my timelines gradually collapse, I really want to engage with compelling arguments that might lengthen my timelines again. 

I have feel like there are a bunch of viewpoints expressed about long timelines/slow takeoff but a lack of arguments. This is me reaching out in the hope that people might point me to the best existing write-ups or maybe make new ones!

I am tracking things like: "takeoff will be slow because of experiment compute bottlenecks," or "timelines to AIs with good research taste are very long," or even more general "look how bad AI is at all this (not-super-relevant-to-a-software-only-singularity-)stuff that is so easy for humans!" but in my opinion, these are just viewpoints (which by the way, seem to often get stated very confidently in a way that makes me not trust the epistemology behind them). So sadly these statements don't tend to lengthen my timelines.

In my view, these viewpoints would become arguments if they were more like (excuse the spitballing): 

  • "1e28 FLOPs of experiment compute is unlikely to produce much algorithmic progress + give a breakdown of why a compelling allocation of 1e28 FLOP doesn't get very far"
  • "Research taste is in a different reference class to the things AI has been making progress on recently + compelling reasoning, like, maybe:
    • 'it has O(X) more degrees of freedom,'
    • 'it has way less existing data and/or it's way harder to create data, or give a reward signal'
    • 'things are looking grim for the likelihood of generalization to these kinds of skills'
  • "there are XYZ properties intelligence needed that can't be simulated by current hardware paradigms"

Currently I feel like I have a heavy tail on my timelines and takeoff speeds as a placeholder in lieu of arguments like this, that i'm hoping exist. 

Replies from: TsviBT, Kaj_Sotala, Vladimir_Nesov, Archimedes, romeo, Jonas Hallgren
↑ comment by Kaj_Sotala · 2025-04-27T10:10:54.868Z · LW(p) · GW(p)

I have feel like there are a bunch of viewpoints expressed about long timelines/slow takeoff but a lack of arguments.

I kind of like feel the opposite way, in that a lot of people seem to think we'll have short timelines but the arguments for that seem weak! They seem to mostly be based on something like trend extrapolation and assuming that e.g. models getting improving scores on benchmarks means they're actually getting better on real-world tasks. E.g. somebody like Leopold Aschenbrenner will write that GPT-4 is "on the level of a smart high schooler" while at the same time, language models require extensive additional scaffolding [LW · GW] to even get started on a simple game like Pokemon (and none of them have managed to beat it yet). 

There seems to be a general and unjustified assumption that merely because language models perform on some specific narrow problems on the level of a "smart high schooler", you can say that they have that level of intelligence overall. But that seems clearly false, somewhat analogous to saying that a calculator is a superintelligence just because it's superhuman at quickly operating on large numbers. Rather, the AI we have so far seems to succeed at the kinds of things it's been specifically trained at, but fail to generalize [LW · GW] to more novel situations. People also haven't been able to point at much in the way of convincing novel discoveries [LW · GW] made by LLMs.

I asked for the strongest arguments in favor of short timelines [LW · GW] some time ago, and didn't feel like any of them were very compelling. By far the most top-voted answer was one arguing that we might get AI to substantially accelerate AI progress because a particular AI research engineering benchmark looks like it will get saturated within a couple of years. But this is again assuming that good performance on a benchmark for AI research engineering actually translates into significant real-world capability. o3 is said to perform "on par with elite human competitors" on CodeForces, but recent characterizations of its programming ability [LW · GW] are that while it produces code that "almost certainly works", that code is "verbose, brittle, hard to read". 

Furthermore, the most advanced reasoning models seem to be doing an increasing amount of reward hacking [LW · GW] and resorting to more cheating in order to produce the answers that humans want. Not only will this mean that some of the benchmark scores may become unreliable, it means that it will be increasingly hard to get productive work out of them as their intelligence increases and they get better at fulfilling the letter of the task in ways that don't meet the spirit of it.

The benchmarks also do not take into account the fact that the vast majority of them measure a model's performance in a situation where the model is only given one task at a time, and it can completely focus on solving that. If you want the models to act agentically in situations where they have multiple conflicting priorities and need to decide what kinds of approaches to try, then you need them to have something like executive function. Much of Claude's difficulties in playing Pokemon come from severe problems with executive function [LW · GW]. 

This is an entirely separate capability that we're only now starting to make progress on, one which is more difficult to solve because the nature of good executive function is environmentally dependent rather than something with one objectively correct solution. E.g. sometimes it's good if you're able to determinedly work on a problem without getting frustrated or bored, but some of Claude's problems in playing Pokemon are because it doesn't ever get frustrated or bored, so keeps trying the same hopeless approaches over and over again. Sometimes getting utterly obsessed with something that seems unpromising makes you into Charles Darwin, sometimes it causes you to just obsess over something nobody else cares about for your whole life. That's one huge crucial capability that any real-world agents would need to be effective, that I've seen minimal discussion over.

Does any of this make me certain about longer timelines? No. AI progress has been surprisingly fast in the past, and maybe it will be that in the future too. But I don't see any obvious reason to be very confident in short ones, either.

(Also none of this is considering the possibility of increased regulation slowing down the pace of AI - we've already seen regulators starting to get nervous about AI, and widespread anti-AI sentiment.)

Also, a conversation I had with Claude 3.6 about why people tend to overestimate the speed of AI progress

ME: After the 2007 DARPA Grand Challenge there was a lot of hype about how self-driving cars were just around the corner. But we mostly still don't have them. Why did it so much longer than expected?

Claude: [...] Several key factors contributed to this delay:

  • The structured nature of the DARPA challenge versus real-world complexity The DARPA challenges took place in relatively controlled environments - first desert terrain, then a mock urban setting. The real world presents infinitely more edge cases: unexpected construction zones, emergency vehicles, temporary traffic patterns, and regional driving cultures. A system that works 99% of the time isn't good enough when failures could be fatal.
  • The long tail of edge cases Engineers discovered that handling the last few percentage points of driving scenarios was exponentially more difficult than the first 90%. Things humans handle intuitively - like understanding that a ball rolling into the street might be followed by a child - proved extremely challenging to systematize. [...]
  • Regulatory and liability frameworks The legal and insurance frameworks for autonomous vehicles needed to be developed essentially from scratch. Questions about liability in accidents, certification standards, and testing requirements took years to work through.

This pattern of overoptimistic timelines followed by a "reality check" period has been common in AI development. We often underestimate the gap between solving a problem in a controlled environment versus deploying a robust solution in the messy real world. [...]

Me: You mention overoptimistic timelines followed by a reality check having been common in AI development. What are some other examples?

Claude: Speech recognition was predicted to be "solved" multiple times - in the 1950s, researchers at RCA and MIT suggested we'd have widespread speech-to-text within a few years. But achieving human-level performance in real-world conditions (different accents, background noise, conversational speech) took until around 2017.

Machine translation shows a similar pattern. During the Cold War, researchers were optimistic that Russian-English translation would be solved quickly. Warren Weaver wrote an influential memo in 1949 suggesting it was imminent. But achieving truly fluent translation, especially for language pairs with very different structures, took many decades longer.

Computer vision is another classic example. In 1966, Marvin Minsky assigned a summer project to undergraduate students to "connect a camera to a computer and get the computer to describe what it sees." The fundamental difficulty of the problem wasn't apparent until researchers really started working on it. Even something as seemingly simple as reliable edge detection took decades to get right.

Expert systems in the 1980s generated enormous excitement and investment. Companies built specialized hardware and software to encode expert knowledge in rules. But they proved brittle and unable to handle novel situations or acquire new knowledge efficiently. This contributed to the "AI winter" of the 1990s.

Even in recent years, we've seen this pattern repeat with technologies like fully autonomous driving (as we just discussed) and robotic manipulation. The basic demos can look very impressive, but getting to robust real-world performance proves much harder than initially expected.

I think this pattern emerges because:

  1. We tend to underestimate how much implicit knowledge and context humans bring to these tasks
  2. Early successes in controlled environments don't scale linearly to real-world complexity
  3. Edge cases multiply exponentially as systems move from demos to deployment

The key lesson seems to be that achieving narrow success in controlled conditions is often just the first small step toward practical, robust solutions.

↑ comment by Vladimir_Nesov · 2025-04-27T06:24:04.236Z · LW(p) · GW(p)

There is no way to predict the location of crucial thresholds of capabilities and the timing of when basic science comes up with new methods, so any theoretical arguments can only slightly slosh the probabilities along the timeline. Evals are getting better, but saturation of any given eval remains only a lower bound for crossing capability thresholds. We get to experimentally observe the capabilities once they have actually been achieved, but no earlier.

The most concrete consideration is how the speed of compute scaling changes somewhat predictably (fast now, ~3x slower after funding stops growing at current rates in 2027-2029), as it's the key input to any methods of creating capabilities. Natural text data will be ~completely running out around 2027-2029 as well, and pretraining from other kinds of data is plausibly much less efficient, slowing down scaling of capabilities from pretraining further.

The AI companies might have some idea about the scaling laws for long reasoning training, which inputs to training influence capabilities how much, whether there are scarce inputs or bounds on capabilities [LW(p) · GW(p)] inherited from the base models. Public knowledge on this is still at the stage of reproducing and slightly improving the methods of DeepSeek-R1, but unlike last year the research efforts in the open have a more defined target. This brings the crucial thresholds of capabilities closer than pretraining on its own, but it's not clear if they will be approaching faster as well, or if it's a one-time improvement on top of pretraining, so that scaling of pretraining will still remain a limiting factor without new training methods (whose timing of arrival is unpredictable).

↑ comment by Archimedes · 2025-04-27T04:55:41.760Z · LW(p) · GW(p)

Epoch.ai just released this today:

https://epoch.ai/gradient-updates/the-case-for-multi-decade-ai-timelines

Excerpt:

I would summarize the key cruxes that separate my views from people who have shorter timelines as follows:

  • I don’t see the trends that one would extrapolate in order to arrive at very short timelines on the order of a few years. The obvious trend extrapolations for AI’s economic impact give timelines to full remote work automation of around a decade, and I expect these trends to slow down by default.
  • I don’t buy the software-only singularity as a plausible mechanism for how existing rates of growth in AI’s real-world impact could suddenly and dramatically accelerate by an order of magnitude, mostly because I put much more weight on bottlenecks coming from experimental compute and real-world data. This kind of speedup is essential to popular accounts of why we should expect timelines much shorter than 10 years to remote work automation.
  • I think intuitions for how fast AI systems would be able to think and how many of them we would be able to deploy that come from narrow writing, coding, or reasoning tasks are very misguided due to Moravec’s paradox. In practice, I expect AI systems to become slower and more expensive as we ask them to perform agentic, multimodal, and long-context tasks. This has already been happening with the rise of AI agents, and I expect this trend to continue in the future.
Replies from: Vladimir_Nesov
↑ comment by Vladimir_Nesov · 2025-04-27T12:06:10.844Z · LW(p) · GW(p)

The extensive discussion of trends in global datacenter/Nvidia revenue shows that the framing considers human economy as a whole as the system driving eventual AI takeoff, that there are always essential complementary inputs that can't be abstracted out.

Software-only singularity is about considering scaling laws for a different system [LW(p) · GW(p)] that is not the entire economy and whose relevant inputs are specific AIs (varying in their capabilities and compute efficiency) and the novel software and cultural knowledge they are producing, rather than more material forms of capital or compute or data from the physical world. An intermediate construction is an AI/robot economy that's highly decoupled from the human economy and does its own thing at its own pace.

Early trends of an algal bloom shouldn't be about the total mass of organic matter in the ocean. The choice of the system to consider relevant carries more of the argument than detailed analyses of any given system. In the post, Ege Erdil makes a point that we know very little about the system where a possible software-only singularity takes place:

It’s just hard to be convinced in a domain where the key questions about the complexity of the job of a researcher and the complementarity between cognitive and compute/data inputs remain unanswered.

This is a reason for persistence of the disagreement about which systems are relevant, as those who feel that software-only recursive self-improvement can work and is therefore a relevant system will fail to convince those who don't, and conversely. But instead of discussing the crux of which system is relevant (which has to be about details of recursive self-improvement), only the proponents will tend to talk about software-only singularity, while the opponents will talk about different systems whose scaling they see as more relevant, such as the human economy or datacenter economy.

In the current regime, pretraining scaling laws tether AI capabilities to compute of a single training system, but not to the total amount of compute (or revenue) in datacenters worldwide. This in turn translates to relevance of finances of individual AI companies and hardware improvements, which will remain similarly crucial if long reasoning training takes over from pretraining, the difference being that AI company money will be buying inference compute for RL training from many datacenters, rather than time on a single large training system. A pivot to RL (if possible) lifts some practical constraints on the extent of scaling, and the need to coordinate construction of increasingly large and expensive training systems that are suboptimal for other purposes. This might let the current scaling regime extend for another 3-4 years, until 2030-2032, as an AI company would only need to cover a training run rather than arrange construction of a training system, a difference of 10x.

↑ comment by romeo · 2025-04-27T04:13:09.584Z · LW(p) · GW(p)

A brief history of the things that have most collapsed my timelines down since becoming aware of AI safety <2 years ago:

  • Fun with +12 OOMs of Compute [LW · GW] IMO, a pretty compelling writeup that brought my 'timelines to AGI uncertainty-over-training-compute-FLOP' down a bunch
  • Generally working on AI 2027, which has included
    • Writing and reading the capabilities progression where each step seems plausible.
    • Researching how much compute is scaling.
    • Thinking about how naive and limiting current algorithms and architectures seem, and what changes they are plausibly going to be able to implement soon.
    • The detailed benchmarks+gaps argument in the timelines forecast.
  • The recent trend in METR's time horizon data.
↑ comment by Jonas Hallgren · 2025-04-27T05:58:26.837Z · LW(p) · GW(p)

There's an entire class of problem within ML that I would see as framing problems and the one thing I think LLMs don't help that much with is framing. 

I don't believe that these will be solved within the scaling paradigms that have been hypothesised. (Related to what Archimedes linked from Epoch, not only in training data but theoretical modelling for fitting on that training data.)

There's this quote I've been seeing from Situation Awareness that all you have to do is "believe in a straight line on a curve" and when I hear that and see the general trend extrapolations my spider senses start tingling. In the frame of the model, the assumptions of shorter timelines make sense, if you reject the frame you start seeing holes. 

Those holes are more like open scientific questions that no one has answered but it raises the variance of timelines by quite a lot. 

If you want to engage more fully with skeptics, I really liked going to ICML last year so I can recommend it. Also, see this comment for some more details: https://www.lesswrong.com/posts/TpSFoqoG2M5MAAesg/#nQAXHms3JCJ9meBey