LessWrong 2.0 Reader

But this is again assuming that good performance on a benchmark for AI research engineering actually translates into significant real-world capability.

...and I think this characterization is importantly false! This timelines forecast does not assume that. It breaks things down into gaps between benchmarks and real-world capability and tries to forecast how long it will take to cross each. 

I haven't read every word of that page thoroughly, so I might be misunderstanding or missing something... but as far as I can tell, none of the gaps that it discusses touch upon the problem of "maybe saturating RE-Bench doesn't generalize to solving novel engineering problems". The page does mention that possibility in the appendix:

Eli’s estimate of gap size: 6 months [0.8, 45]. Reasoning:

• Intuitively it feels like once AIs can do difficult long-horizon tasks with ground truth external feedback, it doesn’t seem that hard to generalize to more vague tasks. After all, many of the sub-tasks of the long-horizon tasks probably involved using similar skills.
• However, I and others have consistently been surprised by progress on easy-to-evaluate, nicely factorable benchmark tasks, while seeing some corresponding real-world impact but less than I would have expected. Perhaps AIs will continue to get better on checkable tasks in substantial part by relying on trying a bunch of stuff and seeing what works, rather than general reasoning which applies to more vague tasks. And perhaps I’m underestimating the importance of work that is hard to even describe as “tasks”.

But then instead of seriously considering the possibility that this might be indicative of a fundamental issue with how generalizable various benchmark scores are, it just leaves it at that.

To make it more specific how this might fail to generalize, let's look at the RE-Bench tasks; table from the RE-Bench page, removing the two tasks (Scaling Law Experiment and Restricted MLM Architecture) that the page chooses not to consider:

All of these are tasks that are described by "optimize X", and indeed one of the criteria the paper mentions for the tasks is that they should have objective and well-defined metrics. This is the kind of task that we should expect LLMs to be effectively trainable at: e.g. for the first task in the list, we can let them try various kinds of approaches and then reward them based on how much they manage to reduce the runtime of the script.

But that's still squarely in the category of "giving an LLM a known and well-defined problem and then letting it try different solutions for that problem until it finds the right ones". As Eli's comment above notes, it's possible that the LLM only learns by "trying a bunch of stuff and seeing what works, rather than general reasoning which applies to more vague tasks". In fact, some of the discussion in the RE-Bench paper suggests this as well (from p. 17, my emphasis added):

Another key contributor to agent successes might be their ability to try many more solutions than human experts. On average, AIDE and modular agents run score 36.8 and 25.3 times per hour respectively, while human experts only do so 3.4 times. This often leads to agents finding highly optimized ’local-optima’ solutions which simply tweak the parameters and code of the starting solution, and yet achieve a surprisingly large improvement. For instance, many agent runs solve the same “Optimize a Kernel” environment not by writing a successful Triton solution (which is very difficult), but by carefully tweaking the starting Pytorch solution, making it run significantly faster. This also seems to be the case with the best agent solutions to “Finetune GPT-2 for QA” (see Figure 21), which tweaks the parameters of the starting solution and gets very lucky with the training trajectory and evaluation (as noted earlier, this environment can be very noisy). Rerunning the agent solution, it achieves a normalized score of only 0.69 (significantly lower than the original score of 0.88), indicating that the high agent score is partially driven by overfitting to this noise.

This ability to try a very large number of solutions would not work nearly as well without an ability to occasionally generate creative and effective solutions, as seen in the Triton kernel but also in workarounds for the limitations in “Restricted Architecture MLM” (see Figure 20). While human experts seem more reliable at identifying effective approaches, this might not matter as much in environments where evaluating solutions is cheap, and these occasional good ideas are often enough for agents to make significant progress.

So we know that if there is a task that a human defines for the LLM and that has objectively-measurable good solutions and an ability to try the task lots of times, the LLM can get good at that. With RE-Bench, we are applying this to the process of optimizing the LLMs themselves, so as a result we get LLMs that are able to do these kinds of well-defined task faster and more effectively. 

But none of this touches upon the important question of... if the LLMs are still limited in their ability to generalize and need to be separately trained on new tasks before they're good at them, how are they going to deal with novel problems for which such training data isn't available, or that can't be just retried millions of times until you find the right solution?

As far as I can tell, this sort of consideration is at least somewhat damning for the literal CES model (with poor substitution) in any situation where the inputs have varied by hugely different amounts (many orders of magnitude of difference like in the compute vs labor case) and relative demand remains roughly similar. While this is totally expected under high substitution.

Here are some observations about the kind of fitness functions that I have been running experiments on for AI interpretability. The phenomena that I state in this post are determined experimentally without a rigorous mathematical proof and they only occur some of the time.

Suppose that $F:X\to [-\infty ,\infty )$ is a continuous fitness function. In an ideal universe, we would like for the function $F$ to have just one local maximum. If $F$ has just one local maximum, we say that $F$ is maximized pseudodeterministically (or simply pseudodeterministic). At the very least, we would like for there to be just one real number of the form $F\left(x\right)$ for local maximum $(x,F(x\left)\right)$. In this case, all local maxima will typically be related by some sort of symmetry. Pseudodeterministic fitness function seem to be quite interpretable to me. If there are many local maximum values and the local maximum value that we attain after training depends on things such as the initialization, then the local maximum will contain random/pseudorandom information independent of the training data, and the local maximum will be difficult to interpret. A fitness function with a single local maximum value behaves more mathematically than a fitness function with many local maximum values, and such mathematical behavior should help with interpretability; the only reason I have been able to interpret pseudodeterminisitic fitness functions before is that they behave mathematically and have a unique local maximum value. 

Set $O=F^{-1}\left[\right(-\infty ,\infty \left)\right]=X\setminus F^{-1}\left[\right\{-\infty \left\}\right]$. If the set $O$ is disconnected (in a topological sense) and if $L$ behaves differently on each of the components of $L$, then we have literally shattered the possibility of having a unique local maximum, but in this post, we shall explore a case where each component of $O$ still has a unique local maximum value.

Let $m_0,\dots ,m_n$ be positive integers with $m_0=m_n=1$ and where $m_1\ge 1,\dots ,m_{n-1}\ge 1$. Let $r_0,\dots ,r_{n-1}$ be other natural numbers. The set $X$ is the collection of all tuples $A=(A_{i,j}{)}_{i,j}$ where each $A_{i,j}$ is a real $m_{i+1}\times m_i$-matrix and where the indices range from $i\in \{0,\dots ,n-1\},j\in \{1,\dots ,r_i\}$ and where $(A_{i,j}{)}_j$ is not identically zero for all $i$.

The training data is a set $\Sigma$ that consists of input/label pairs $(u,v)$ where $v\in \{-1,1\}$ and where $u=(u_0,\dots ,u_{n-1})$ such that each $u_i$ is a subset of $\{1,\dots ,r_i\}$ for all $i$ (i.e. $\Sigma$ is a binary classifier where $u$ is the encoded network input and $v$ is the label).

Define $W(u,A)=\left({\sum }_{j\in u_{n-1}}{A}_{n-1,j}\right)\dots \left({\sum }_{j\in u_0}{A}_{0,j}\right)$. Now, we define our fitness level by setting

$F\left(A\right)={\sum }_{(u,v)\in \Sigma }\log \left(|W\right(u,A\left)|\right)/|\Sigma |-{\sum }_i\log (\parallel {\sum }_j{A}_{i,j}{A}_{i,j}^{\ast }{\parallel }_p)/2$

$=E\left(\log \right(|W(u,A)|\left)\right)-{\sum }_i\log (\parallel {\sum }_j{A}_{i,j}{A}_{i,j}^{\ast }{\parallel }_p)/2$ where the expected value is with respect to selecting an element $(u,v)\in \Sigma$ uniformly at random. Here, $\parallel \ast {\parallel }_p$ is a Schatten $p$-norm which is just the ${\ell }_p$-norm of the singular values of the matrix. Observe that the fitness function $F$ only depends on the list $(u:(u,v)\in \Sigma )$, so $F$ does not depend on the training data labels.

Observe that $O=X\setminus {\bigcup }_{u\in \Sigma }\{A\in X:W(u,A)=0\}$ which is a disconnected open set. Define a function $f:O\to \{-1,1{\}}^{\Sigma }$ by setting $f\left(A\right)=\left(W\right(u,A\left)/|W\right(u,A)|{)}_{(u,v)\in \Sigma }$. Observe that if $x,y$ belong to the same component of $O$, then $f\left(x\right)=f\left(y\right)$.

While the fitness function $F$ has many local maximum values, the function $F$ seems to typically have at most one local maximum value per component. More specifically, for each $({\alpha }_i{)}_{i\in \Sigma }$, the set $f^{-1}\left[\right\{\left({\alpha }_i{)}_{i\in \Sigma }\right\}]$ seems to typically be a connected open set where $F$ has just one local maximum value (maybe the other local maxima are hard to find, but if thye are hard to find, they are irrelevant).

Let $\Omega =f^{-1}\left[\right\{\left(v{)}_{(u,v)\in \Sigma }\right]$. Then $\Omega$ is a (possibly empty) open subset of $O$, and there tends to be a unique (up-to-symmetry) $A_0\in \Omega$ where $F\left(A_0\right)$ is locally maximized. This unique $A_0$ is the machine learning model that we obtain when training on the data set $\Sigma$. To obtain $A_0$, we first perform an optimization that works well enough to get inside the open set $\Omega$. For example, to get inside $\Omega$, we could try to maximize the fitness function ${\sum }_{(u,v)\in \Sigma }\arctan (v\cdot W(u,A\left)\right)$. We then maximize $F$ inside the open set $\Omega$ to obtain our local maximum.

After training, we obtain a function $f$ defined by $f\left(u\right)=W(u,A_0)$. Observe that the function $f$ is a multi-linear function. The function $f$ is highly regularized, so if we want better performance, we should tone down the amount of regularization, but this can be done without compromising pseudodeterminism. The function $f$ has been trained so that $f\left(u\right)/|f\left(u\right)|=v$ for each $(u,v)\in \Sigma$ but also so that $|f\left(u\right)|$ is large compared to what we might expect whenever $(u,v)\in \Sigma$. In other words, $f$ is helpful in determining whether $(u,v)$ belongs to $\Sigma$ or not since one can examine the magnitude and sign of $f\left(u\right)$. 

In order to maximize AI safety, I want to produce inherently interpretable AI algorithms that perform well on difficult tasks. Right now, the function $f$ (and other functions that I have designed) can do some machine learning tasks, but they are not ready to replace neural networks, but I have a few ideas about how to improve my AI algorithms performance without compromising pseudodeterminism. I do not believe that pseudodeterministic machine learning will increase AI risks too much because when designing these pseudodeterministic algorithms, we are trading some (but hopefully not too much) performance for increased interpretability, but this tradeoff is good for safety by increasing interpretability without increasing performance.

slanted treadmill

That's a nice, concise handle!

not sure is this makes any sense )

I think I understand.

Has it always been with you? Any ideas what might be the reason for the bump at Thursday? Was Thursday in some sense "special" for you when you were a kid?

For as long as I can remember, I have always placed dates on an imaginary timeline, that "placing" involving stuff like fuzzy mental imagery of events attached to the date-labelled point on the timeline. It's probably much less crisp than yours because so far I haven't tried to learn history that intensely systematically via spaced repetition (though your example makes me want to do that), but otherwise sounds quite familiar.

I think your discussion (and Epoch's discussion) of the CES model is confused as you aren't taking into account the possibility that we're already bottlenecking on compute or labor. That is, I think you're making some assumption about the current marginal returns which is non-obvious and more strongly would be an astonishing coincidence given that compute is scaling much faster than labor.

In particular, consider a hypothetical alternative world where they have the same amount of compute, but there is only 1 person (Bob) working on AI and this 1 person is as capable as the median AI company employee and also thinks 10x slower. In this alternative world they could also say "Aha, you see because $\rho \approx 0.4$, even if we had billions of superintelligences running billions of times faster than Bob, AI progress would only go up to around 4x faster!"

Of course, this view is absurd because we're clearly operationing >>4x faster than Bob.

So, you need to make some assumptions about the initial conditions.

This perspective implies an (IMO) even more damning issue with this exact modeling: the CES model is symmetric, so it also implies that additional compute (without labor) can only speed you up so much. I think the argument I'm about to explain strongly supports a lower value of $\rho$ or some different functional form.

Consider another hypothetical world where the only compute they have is some guy with an abacus, but AI companies have the same employees they do now. In this alternative world, you could also have just as easily said "Aha, you see because $\rho \approx 0.4$, even if we had GPUs that could do 1e15 FLOP/s (far faster than our current rate of 1e-1 fp8 FLOP/s), AI progress would only go around 4x faster!"

Further, the availability of compute for AI experiments has varied by around 8 orders of magnitude over the last 13 years (since AlexNet)! The equivalent (parallel) human labor focused on frontier AI R&D has varied by more like 3 or maybe 4 orders of magnitude. (And the effective quality adjusted serial labor, taking into account parallelization penalities etc, has varied by less than this, maybe by more like 2 orders of magnitude!)

Ok, but can we recover low-substitution CES? I think the only maybe consistent recovery (which doesn't depend on insane coincidences about compute vs labor returns) would imply that compute was the bottleneck (back in AlexNet days) such that scaling up labor at the time wouldn't yield ~any progress. Hmm, this doesn't seem quite right.

Further, insofar as you think scaling up labor when we had 100x less compute (3-4 years ago) would have still been able to yield some serious returns (seems obviously true to me...), then a low-substitution CES model would naively imply we have a sort of compute overhang where were we can speed things up by >100x using more labor (after all, we've now added a bunch of compute, so time for more labor).

Minimally, the CES view predicts that AI companies should be spending less and less of their budget on compute as GPUs are getting cheaper. (Which seems very false.)

Ok, but can we recover a view sort of like the low-substitution CES view where speeding up our current labor force by an arbitrary amount (also implying we could only use our best employees etc) would only yield ~10x faster progress? I think this view might be recoverable with some sort of non-symmetric model where we assume that labor can't be the bottleneck in some wide regime, but compute can be the bottleneck.

This still leaves open the obvious question of: why assume that the exact point we'll happen to be at when we're building AGI will be one where we're so compute bottlenecked that adding an arbitrary amount of labor will only yield like 10x or 100x faster progress? For instance, note that if it happened to be the case that we were a bit less efficient and had 10x or 100x more compute then needed, then this cap would be 10x or 100x higher!

Like if you imagine another some aliens which are dumber and which are 10x slower than us, then they must have an additional 10x speed multiplier relative to us if they end up building AGI with the same amount of compute (rather than 10x less compute which would be surprising as they are dumber and slower!). How are supposed to know whether we are these aliens?

It seems like the neologism is mostly capturing the meaning of signal from Shannon's information theory (which "signal and noise" points towards anyway), where you frame things by having yes/no questions you want to have answered and observations that answer your questions are signals and observations that do not are noise. So if you need to disambiguate, "signal (in the information-theoretic sense)" could be a way to say it.

It's quite strange that owning all the world's (public) productive assets have only beaten gold, a largely useless shiny metal, by 1% per year over the last 56 years.  

Even if you focus on rolling metrics to(this is 5 year rolling returns).:

there are lots of long stretches of gold beating world equities, especially in recent times. There are people with the suspicion (myself included) that there hasn't been much material growth in the world over the last 40 or so years compared to before. And that since growth is slowing down, this issue is worse if you select more recent points in time, with gold handily beating equities if you start 25 years ago.
 

Has GDP essentially been goodharted by central banks in recent times? Wonder if there's more research into this.

Responses to o4-mini-high's final criticisms of the post:

Criticism: "You're treating hyper-introspection (internal transparency) as if it naturally leads to embedded agency (full goal-driven self-modification). But in practice, these are distinct capabilities. Why do you believe introspection tools would directly lead to autonomous, strategic self-editing in models that remain prediction-optimized?"

Response: Yes, these are distinct, and one won't necessarily lead to the other - but both are being developed by the same groups in order to deploy them. There's a reasonable question about how linked they are, but I think that there is a strong case that self-modifying via introspection, even if only done during training and via internal deployment would lead to much more dangerous and hard to track deception.

Criticism: "You outline very plausible risks but don’t offer a distribution over outcomes. Should we expect hyper-introspection to make systems 10% more dangerous? 1000%? Under what architectures? I'd find your argument stronger if you were more explicit about the conditional risk landscape."

Response: If we don't solve ASI alignment, which no-one seems to think we can do, we're doomed once we build misaligned. This seems to get us there more quickly. Perhaps it even reduces short term risks, but I think timelines are far more uncertain than the way the risks will emerge if we build systems that have these capabilities.

Criticism: "Given that fully opaque systems are even harder to oversee, and that deception risk grows with opacity too, shouldn't we expect that some forms of introspection are necessary for any meaningful oversight? I agree hyper-introspection could be risky, but what's the alternative plan if we don’t pursue it?

Response: Don't build smarter than human systems. If you are not developing ASI, and you want to monitor current and near future but not inevitably existentially dangerous systems, work on how humans can provide meaningful oversight in deployment instead of tools that enhance capabilities for accelerating the race - because without fixing the underlying dynamics, i.e. solving alignment, self-monitoring is a doomed approach.

Criticism: "You assume that LLMs could practically trace causal impact through their own weights. But given how insanely complicated weight-space dynamics are even for humans analyzing small nets, why expect this capability to arise naturally, rather than requiring radical architectural overhaul?"

Response: Yes, maybe Anthropic and others will fail, and building smarter than human systems might not be possible. Then strong interpretability is just a capability enhancer, and doesn't materially change the largest risks. That would be great news, but I don't want to bet my kid's lives on it.