My model of what is going on with LLMs

post by Cole Wyeth (Amyr) · 2025-02-13T03:43:29.447Z · LW · GW · 35 comments

Contents

35 comments

Epistemic status: You probably already know if you want to read this kind of post, but in case you have not decided: my impression is that people are acting very confused about what we can conclude about scaling LLMs from the evidence, and I believe my mental model cuts through a lot of this confusion - I have tried to rebut what I believe to be misconceptions in a scattershot way, but will attempt to collect the whole picture here. I am a theoretical computer scientist and this is a theory. Soon I want to do some more serious empirical research around it - but be aware that most of my ideas about LLMs have not had the kind of careful, detailed contact with reality that I would like at the time of writing this post. If you're a good engineer (or just think I am dropping the ball somewhere) and are interested in helping dig into this please reach out. This post is not about timelines, though I think it has obvious implications for timelines. 

We have seen LLMs scale to impressively general performance. This does not mean they will soon reach human level because intelligence is not just a knob that needs to get turned further, it comprises qualitatively distinct functions. At this point it is not plausible that we can precisely predict how far we are from unlocking all remaining functions since it will probably require more insights. The natural guess is that the answer is on the scale of decades. 

It's important to take a step back and understand the history of how progress in A.I. takes place, following the main line of connectionist algorithms that (in hindsight, back-chaining from the frontier) are load-bearing.  This story is relatively old and well-known, but I still need to retell it because I want to make a couple of points clear. First, deep learning has made impressive steps several times over the course of decades. Second, "blind scaling" has contributed substantially but has not been the whole story, conceptual insights piled on top of (and occasionally mostly occluding/obsoleting) each other have been necessary to shift the sorts of things we knew how to train artificial neural nets to do.

Rosenblatt invented the perceptron in 1958. Initially it didn't really take off because of compute, and also ironically because the book "Perceptrons" (published ~10 years later) showed the theoretical limitations of the idea in its nascent form (there weren't enough layers, turns out adding more layers works but you have to invent backpropogation). 

 Apparently^[1] enthusiasm didn't really ramp up again until 2012, when AlexNet proved shockingly effective at image classification. AlexNet was a convolutional neural network (CNN), a somewhat more complicated idea than the perceptron but clearly a philosophical descendant. Both relied on supervised learning; a nice clear "right/wrong" signal for every prediction and no feedback loops. 

Since then there has been pretty steady progress, in the sense that deep learning occasionally shocks everyone by doing what previously seemed far out of reach for AI. Back in the 20-teens, that was mostly playing hard games like Go and Starcraft II. These relied on reinforcement learning, which is notoriously hard to get working in practice, and conceptually distinct from perceptrons and CNNs - though it still used deep learning for function approximation. My impression is that getting deep learning to work at all on new and harder problems usually required inventing new algorithms based on a combination of theory and intuition - it was not just raw scaling, that was not usually the bottleneck. Once we^[2] left the realm of supervised learning every victory was hard fought. 

In the 2020's, it has been LLMs that demonstrated the greatest generality and impressiveness. They are powered mostly by supervised learning with transformers, a new architecture that we apparently kind of just stumbled on (the intuitions for attention don't seem compelling to me). Suddenly AI systems can talk a lot like humans do, solve math problems, AND play a decent game of chess sometimes - all with the same system! There is a lot of noise about AGI being ten - no five - no three - no two years away (one of the better received and apparently higher quality examples is Aschenbrenner's "situational awareness", which ironically is also a term for something that LLMs may or may not have). Some people even seem to think it's already here. 

It is not, because there is one crucial test (yes this is a crux) that LLMs have not passed. They have never done anything important. 

They haven't proven any theorems that anyone cares about. They haven't written anything that anyone will want to read in ten years (or even one year). Despite apparently memorizing more information than any human could ever dream of, they have made precisely zero novel connections or insights in any area of science^[3].

If you model intelligence as a knob that you continuously turn up until it hits and surpasses human intelligence, then this makes no sense. 

Clearly LLMs are smarter than humans in some sense: they know more. 

But not in some other sense(s):  they have done nothing that matters. 

I do not know exactly which mental functions are missing from LLMs. I do have a suspicion that these include learning efficiently (that is, in context) and continually (between interactions) and that those two abilities are linked to fundamentally being able to restructure knowledge in a detailed inside-the-box way^[4]. Relatedly, they can't plan coherently over long timescales. 

The reason for both of these defects is that the training paradigm for LLMs is (myopic) next token prediction, which makes deliberation across tokens essentially impossible - and only a fixed number of compute cycles can be spent on each prediction (Edit: this is wrong/oversimplified in the sense that the residual streams for earlier positions in the sequence are available at all later positions, though I believe the training method does not effectively incentivize multiple steps of deliberation about the high-level query, see comment from @hmys [LW · GW] for further discussion). This is not a trivial problem. The impressive performance we have obtained is because supervised (in this case technically "self-supervised") learning is much easier than e.g. reinforcement learning and other paradigms that naturally learn planning policies. We do not actually know how to overcome this barrier. 

At this point its necessary to spend some time addressing the objections that I anticipate, starting with the most trivial. 

My position is NOT that LLMs are "stochastic parrots." I suspect they are doing something akin to Solomonoff induction with a strong inductive bias in context - basically, they interpolate, pattern match, and also (to some extent) successfully discover underlying rules in the service of generalization. My mental picture of the situation is that the knowledge we have fed into LLMs forms a fairly dense but measure ~0 "net." Nearly every query misses the net but is usually close-ish to some strand, and LLMs use the cognitive algorithms reflected in that strand to generalize. I am fascinated that this is possible. 

 I am aware that perfect sequence prediction would pretty much require solving all problems. For instance, the most likely answer to a complicated never-seen-before question is probably the right answer, so if LLMs were perfectly calibrated through their training process, they would basically be omniscient oracles, which could easily be bootstrapped into powerful agents - but actually, I would guess that other things break down far before that point. The kind of cognitive algorithm that could approach perfection on sequence prediction would have to solve lots of problems of deliberation, which would in practice require agency. However, deep learning is not perfect at optimizing its objective.  In a sense, this entire line of inquiry is misguided - in fact, it almost works in the opposite direction it is usually offered in: because perfect prediction would require highly general agency, and because we do not know how to teach AI systems highly general agency, we shouldn't expect to get anywhere close to perfect prediction. Certainly without access to a gear-level model of why deep learning works so well one could imagine transformers just getting better and better at prediction without limit, and this has continued for a surprisingly long time, but it seems to be stalling (where is the GPT-5 level series of models?) and this is exactly what we should expect from history and our (imperfect, heuristic) domain knowledge. 

Now, there is a lot of optimism about giving LLMs more compute at inference time to overcome their limitations. Sometimes the term "unhobbling" is thrown around. The name is misleading. It's not like we have hobbled LLMs by robbing them of long term memory, and if we just do the common sense thing and give them a scratchpad they'll suddenly be AGI. That is not how these things work - its the kind of naive thinking that led symbolic AI astray. The implicit assumption is that cognition is a sequence of steps of symbol manipulation, and each thought can be neatly translated into another symbol which can then be carried forward at the next step without further context. Now, in principle something like this must be theoretically possible (a Turing machine is just a symbol manipulation machine) but a level of abstraction or so is getting dropped here - usefully translating a rich cognitive process into discrete symbols is a hard bottleneck in practice. We don't actually know how to do it and there is no reason to expect it will "just work." In fact I strongly suspect that it can't.

Another objection, which I take somewhat-but-not-very seriously, is that even if LLMs remain limited they will rapidly accelerate AI research to the extent that AGI is still near. It's not clear to me that this is a reasonable expectation. Certainly LLMs should be useful tools for coding, but perhaps not in a qualitatively different way than the internet is a useful tool for coding, and the internet didn't rapidly set off a singularity in coding speed. In fact, I feel that this argument comes from a type of motivated reasoning - one feels convinced that the singularity must be near, sees some limitations with LLMs that aren't obviously tractable, and the idea of LLMs accelerating AI progress "comes to the rescue." In other words, is that your true objection?

Okay, so now I feel like I have to counter not an argument but a sort of vibe. You interact with a chatbot and you feel like "we must be close, I am talking to this intelligent thing which reminds me more of humans than computers." I get that. I think that vibe makes people invent reasons that it has to work - ah but if we just let it use a chain of thought, if we just use some RLHF, if we just etc. 

Sure, I cannot prove to you that none of those things will work. Also, no one can prove that deep learning will work, generally - its pretty much just gradient descent on an objective that is not convex (!) - and yet it does work. In other words, we don't know exactly what is going to happen next. 

However, I am asking you not to mistake that vibe for a strong argument, because it is not. On the outside view (and I believe also on the inside view) the most plausible near future looks something like the past. Some more conceptual insights will be necessary. They will accrue over the course of decades with occasional shocks, but each time that we solve something previously believed out of reach, it will turn out that human level generality is even further out of reach after all. Intelligence is not atomic. It has component functions, and we haven't got all of them yet. We don't even know what they are - at least, not the ones we need in practice for agents that can actually be built inside our universe^[5].

Also, I believe the writing is on the wall at this point. It was reasonable to think that maybe transformers would just work and soon when we were racing through GPT-2, GPT-3, to GPT-4. We just aren't in that situation anymore, and we must propagate that update fully through our models and observe that the remaining reasons to expect AGI soon (e.g. "maybe all of human intelligence is just chain of thoughts") are not strong.  

Of course, whenever we actually invent human level AGI the course of history will be disrupted drastically. I am only saying that point may well be pretty far off still, and I do not think it is reasonable to expect it inside a few years.

1. ^{^}

   I'd be interested to learn more about the trajectory of progress in connectionist methods during the intervening years?

2. ^{^}

   Below "we" always means "humanity" or "AI researchers." Usually I have nothing to do with the people directly involved.  

3. ^{^}

   Of course, narrow systems likes AlphaFold have continued to solve narrow problems - this just doesn't have much to do with AGI. Small teams of really smart people with computers can solve hard problems - it is nice and not surprising.  

4. ^{^}

   This is a phrase I have borrowed in its application as a positive ability from Carl Sturtivant. Its meant to capture about the same vibe as "gears level."

5. ^{^}

   That is, barring AIXI. 

35 comments

Comments sorted by top scores.