Davidmanheim's Shortform

Distinct views about LLM Intelligence

In discussing AI and existential risk, I’ve noticed that disagreements often hinge on views of AI capabilities—particularly whether large language models (LLMs) are "doing reasoning." This deserves clearer differentiation. Matt Yglesias once noted that “most optimists are not, fundamentally, AI optimists — they are superintelligence skeptics.” This feels deeply true, and it’s connected to the spectrum of beliefs about current AI systems’ reasoning abilities. To continue Ygelsias’ dichotomy, I’ll sketch a brief and uncharitable pastiche of each view.

The first, which I’ll (unfairly) call the stochastic parrot view, is that LLMs don’t actually understand. What models actually do is constructed based on a textual model of the world that has deeply confused relationships between different things. The successes based on various metrics like exams and reasoning tests are in large part artifacts of training on test data, i.e. leakage. Given that there is some leakage and data about all of the different types of evaluation which are performed on LLMs in the training data, along with substantive answers to similar or identical questions, statistics about the training data text, unrelated to understanding, leads to many correct answers. This success, however, is essentially entirely due to the cognitive labor of those whose input data was used, and not a successful world-model. The failures in doing slight variations of common tasks are largely an obvious result of this, since there isn’t an underlying coherent understanding.

The second, which I’ll (also unfairly) call the already AGI view, is that LLMs do have a largely coherent underlying model of the world. The textual data on which they are trained is rich enough to allow the model’s training to capture many true facts about the world, even though its implicit causal understanding is imperfect. The successes based on various metrics such as exams are similar to that of most humans who take exams, a combination of pattern matching based on training, and cached implicit models of what is expected. At least a large portion of LLM failures in doing slight variations of common tasks are because it is implicitly being asked to reproduce the typical class of answers to questions, and most humans pay only moderate attention and will make similar mistakes, especially in a single pass. The success of prompt design, such as “reason step-by-step,” is just telling the model to do what humans actually do. The abilities which are displayed are evidence that the first-pass failure is sloppiness, not inability.

I suspect that there are two stages of the crux between these views. The first is whether LLMs are doing “true” reasoning, and the second is whether humans are. I have mostly focused on the first, but to conclude, the second seems important as well - if there’s something called reasoning and humans don’t do it most of the time, and some humans don’t do at all - which some have claimed - then given how much these systems have memorized and their extant competencies, I’m not sure that this form of reasoning is needed for AGI, even if it’s needed for ASI.

Toby Ord writes that “the required resources [for LLM training] grow polynomially with the desired level of accuracy [measured by log-loss].” He then concludes that this shows “very poor returns to scale,” and christens it the "Scaling Paradox." (He continues to point out that this doesn’t imply it can’t create superintelligence, but I agree with him about that.)

But what would it look like if this were untrue? That is, what would be the conceptual alternative, where required resources grow more slowly?I think the answer is that it’s conceptually impossible.

To start, there is a fundamental bound on loss at zero, since the best possible model perfectly predicts everything - it exactly learns the distribution. This can happen when overfitting a model, but it can also happen when there is a learnable ground truth; models that are trained to learn a polynomial function can learn them exactly. 

But there is strong reason to expect the bound to be significantly above zero loss. The training data for LLMs contains lots of aleatory randomness, things that are fundamentally conceptually unpredictable. I think it’s likely that things like RAND’s random number book are in the training data, and it’s fundamentally impossible to predict randomness. I think something similar is generally true for many other things - predicting world choice for semantically equivalent words, predicting where typos occur, etc.

Aside from being bound well above zero, there's a strong reason to expect that scaling is required to reduce loss for some tasks. In fact, it’s mathematically guaranteed to require significant computation to get near that level for many tasks that are in the training data. Eliezer pointed out that GPTs are predictors [LW · GW], and gives the example of a list of numbers followed by their two prime factors. It’s easy to generate such a list by picking pairs of primes and multiplying them, the writing the answer first - but decreasing loss for generating the next token to predict the primes from the product is definitionally going to require exponentially more computation to perform better for larger primes.

And I don't think this is the exception, I think it's at least often the rule. The training data for LLMs contains lots of data where the order of the input doesn’t follow the computational order of building that input. When I write an essay, I sometimes arrive at conclusions and then edit the beginning to make sense. When I write code, the functions placed earlier often don’t make sense until you see how they get used later. Mathematical proofs are another example where this would often be true.

An obvious response is that we’ve been using exponentially more compute for better accomplishing tasks that aren’t impossible in this way - but I’m unsure if that is true. Benchmarks keep getting saturated, and there’s no natural scale for intelligence. So I’m left wondering whether there’s any actual content in the “Scaling Paradox.”

(Edit: now also posted to my substack.)