"Scaling breaks down", they say. By which they mean one of the following wildly different claims with wildly different implications:

• When you train on a normal dataset, with more compute/data/parameters, subtract the irreducible entropy [LW(p) · GW(p)] from the loss, and then plot in a log-log plot: you don't see a straight line anymore.
• Same setting as before, but you see a straight line; it's just that downstream performance doesn't improve .
• Same setting as before, and downstream performance improves, but: it improves so slowly that the economics is not in favor of further scaling this type of setup instead of doing something else.
• A combination of one of the last three items and "btw., we used synthetic data and/or other more high-quality data, still didn't help".
• Nothing in the realm of "pretrained models" and "reasoning models like o1" and "agentic models like Claude with computer use" profits from a scale-up in a reasonable sense.
• Nothing which can be scaled up in the next 2-3 years, when training clusters are mostly locked in, will demonstrate a big enough success to motivate the next scale of clusters costing around $100 billion [LW(p) · GW(p)].

Be precise. See also.

You should all be using the "Google Scholar PDF reader extension" for Chrome.

Features I like:

• References are linked and clickable
• You get a table of contents
• You can move back after clicking a link with Alt+left

Screenshot: 

↑ comment by StefanHex (Stefan42) · 2024-07-02T09:53:11.405Z · LW(p) · GW(p)

This is great, love it! Settings recommendation: If you (or your company) want, you can restrict the extension's access from all websites down to the websites you read papers on. Note that the scholar.google.com access is required for the look-up function to work.

https://www.wsj.com/tech/ai/californias-gavin-newsom-vetoes-controversial-ai-safety-bill-d526f621

“California Gov. Gavin Newsom has vetoed a controversial artificial-intelligence safety bill that pitted some of the biggest tech companies against prominent scientists who developed the technology.

The Democrat decided to reject the measure because it applies only to the biggest and most expensive AI models and leaves others unregulated, according to a person with knowledge of his thinking”

Are the straight lines from scaling laws really bending? People are saying they are, but maybe that's just an artefact of the fact that the cross-entropy is bounded below by the data entropy. If you subtract the data entropy, then you obtain the Kullback-Leibler divergence, which is bounded by zero, and so in a log-log plot, it can actually approach negative infinity. I visualized this with the help of ChatGPT:

Here, f represents the Kullback-Leibler divergence, and g the cross-entropy loss with the entropy offset. 

Why I think scaling laws will continue to drive progress

Epistemic status: This is a thought I had since a while. I never discussed it with anyone in detail; a brief conversation could convince me otherwise. 

According to recent reports there seem to be some barriers to continued scaling. We don't know what exactly is going on, but it seems like scaling up base models doesn't bring as much new capability as people hope.

However, I think probably they're still in some way scaling the wrong thing: The model learns to predict a static dataset on the internet; however, what it needs to do later is to interact with users and the world. For performing well in such a task, the model needs to understand the consequences of its actions, which means modeling interventional distributions P(X | do(A)) instead of static data P(X | Y). This is related to causal confusion as an argument against the scaling hypothesis [LW · GW]. 

This viewpoint suggests that if big labs figure out how to predict observations in an online-way by ongoing interactions of the models with users / the world, then this should drive further progress. It's possible that labs are already doing this, but I'm not aware of it, and so I guess they haven't yet fully figured out how to do that. 

What triggered me writing this is that there is a new paper on scaling law for world modeling that's about exactly what I'm talking about here. 

https://www.washingtonpost.com/opinions/2024/07/25/sam-altman-ai-democracy-authoritarianism-future/

Not sure if this was discussed at LW before. This is an opinion piece by Sam Altman, which sounds like a toned down version of "situational awareness" to me. 

https://x.com/sama/status/1813984927622549881

According to Sam Altman, GPT-4o mini is much better than text-davinci-003 was in 2022, but 100 times cheaper. In general, we see increasing competition to produce smaller-sized models with great performance (e.g., Claude Haiku and Sonnet, Gemini 1.5 Flash and Pro, maybe even the full-sized GPT-4o itself). I think this trend is worth discussing. Some comments (mostly just quick takes) and questions I'd like to have answers to:

• Should we expect this trend to continue? How much efficiency gains are still possible? Can we expect another 100x efficiency gain in the coming years? Andrej Karpathy expects that we might see a GPT-2 sized "smart" model.
• What's the technical driver behind these advancements? Andrej Karpathy thinks it is based on synthetic data: Larger models curate new, better training data for the next generation of small models. Might there also be architectural changes? Inference tricks? Which of these advancements can continue?
• Why are companies pushing into small models? I think in hindsight, this seems easy to answer, but I'm curious what others think: If you have a GPT-4 level model that is much, much cheaper, then you can sell the service to many more people and deeply integrate your model into lots of software on phones, computers, etc. I think this has many desirable effects for AI developers:
  • Increase revenue, motivating investments into the next generation of LLMs
  • Increase market-share. Some integrations are probably "sticky" such that if you're first, you secure revenue for a long time. 
  • Make many people "aware" of potential usecases of even smarter AI so that they're motivated to sign up for the next generation of more expensive AI.
  • The company's inference compute is probably limited (especially for OpenAI, as the market leader) and not many people are convinced to pay a large amount for very intelligent models, meaning that all these reasons beat reasons to publish larger models instead or even additionally. 
• What does all this mean for the next generation of large models? 
  • Should we expect that efficiency gains in small models translate into efficiency gains in large models, such that a future model with the cost of text-davinci-003 is massively more capable than today's SOTA? If Andrej Karpathy is right that the small model's capabilities come from synthetic data generated by larger, smart models, then it's unclear to me whether one can train SOTA models with these techniques, as this might require an even larger model to already exist. 
  • At what point does it become worthwhile for e.g. OpenAI to publish a next-gen model? Presumably, I'd guess you can still do a lot of "penetration of small model usecases" in the next 1-2 years, leading to massive revenue increases without necessarily releasing a next-gen model. 
  • Do the strategies differ for different companies? OpenAI is the clear market leader, so possibly they can penetrate the market further without first making a "bigger name for themselves". In contrast, I could imagine that for a company like Anthropic, it's much more important to get out a clear SOTA model that impresses people and makes them aware of Claude. I thus currently (weakly) expect Anthropic to more strongly push in the direction of SOTA than OpenAI.

New Bloomberg article on data center buildouts pitched to the US government by OpenAI. Quotes:

- “the startup shared a document with government officials outlining the economic and national security benefits of building 5-gigawatt data centers in various US states, based on an analysis the company engaged with outside experts on. To put that in context, 5 gigawatts is roughly the equivalent of five nuclear reactors, or enough to power almost 3 million homes.”
- “Joe Dominguez, CEO of Constellation Energy Corp., said he has heard that Altman is talking about building 5 to 7 data centers that are each 5 gigawatts. “
- “John Ketchum, CEO of NextEra Energy Inc., said the clean-energy giant had received requests from some tech companies to find sites that can support 5 GW of demand, without naming any specific firms.”

Compare with the prediction by Leopold Aschenbrenner in situational awareness: 

- "The trillion-dollar cluster—+4 OOMs from the GPT-4 cluster, the ~2030 training cluster on the current trend—will be a truly extraordinary effort. The 100GW of power it’ll require is equivalent to >20% of US electricity production"


 

Zeta Functions in Singular Learning Theory

In this shortform, I very briefly explain my understanding of how zeta functions play a role in the derivation of the free energy in singular learning theory [? · GW]. This is entirely based on slide 14 of the SLT low 4 talk of the recent summit on SLT and Alignment, so feel free to ignore this shortform and simply watch the video.

The story is this: we have a prior $\phi \left(w\right)$, a model $p(x\mid w)$, and there is an unknown true distribution $q\left(x\right)$. For model selection, we are interested in the evidence of our model for a data set $D_n=\{x_1,\dots ,x_n\}$, which is given by

where $K_n\left(w\right)=\frac{1}{n}{\sum }_{i=1}^n\log \frac{q\left(x_i\right)}{p(x_i\mid w)}$ is the empirical KL divergence. In fact, we are interested in selecting the model that maximizes the average of this quantity over all data sets. The average is then given by

where $K\left(w\right)={\int }_{x\sim q}q\left(x\right)\log \frac{q\left(x\right)}{p(x\mid w)}$ is the Kullback-Leibler divergence. 

But now we have a problem: how do we compute this integral? Computing this integral is what the free energy formula is about [? · GW]. 

The answer: by computing a different integral. So now, I'll explain the connection to different integrals we can draw. 

Let

which is called the state density function. Here, $\delta$ is the Dirac delta function.  For different $t$, it measures the density of states (= parameter vectors) that have $K\left(w\right)=t$. It is thus a measure for the "size" of different level sets. This state density function is connected to two different things. 

Laplace Transform to the Evidence

First of all, it is connected to the evidence above. Namely, let $L\left(v\right)$ be the Laplace transform of $v$. It is a function $L\left(v\right):N\to R$ given by

In first step, we changed the order of integration, and in the second step we used the defining property of the Dirac delta. Great, so this tells us that $L\left(v\right)=\overline{Z}$! So this means we essentially just need to understand $v$.

Mellin Transform to the Zeta Function

But how do we compute $v$? By using another transform. Let $M\left(v\right)$ be the Mellin transform of $v$. It is a function $M\left(v\right):C\to C$ (or maybe only defined on part of $C$?) given by

Again, we used a change in the order of integration and then the defining property of the Dirac delta. This is called a Zeta function. 

What's this useful for?

The Mellin transform has an inverse. Thus, if we can compute the zeta function, we can also compute the original evidence as

Thus, we essentially changed our problem to the problem of studying the zeta function $M\left(v\right).$ To compute the integral of the zeta function, it is then useful to perform blowups to resolve the singularities in the set of minima of $K\left(w\right)$, which is where algebraic geometry enters the picture. For more on all of this, I refer, again, to the excellent SLT low 4 talk of the recent summit on singular learning theory. 

I think it would be valuable if someone would write a post that does (parts of) the following:

• summarize the landscape of work on getting LLMs to reason.
• sketch out the tree of possibilities for how o1 was trained and how it works in inference.
• select a “most likely” path in that tree and describe in detail a possibility for how o1 works.

I would find it valuable since it seems important for external safety work to know how frontier models work, since otherwise it is impossible to point out theoretical or conceptual flaws for their alignment approaches.

One caveat: writing such a post could be considered an infohazard. I’m personally not too worried about this since I guess that every big lab is internally doing the same independently, so that the post would not speed up innovation at any of the labs.

40 min podcast with Anca Dragan who leads safety and alignment at google deepmind: https://youtu.be/ZXA2dmFxXmg?si=Tk0Hgh2RCCC0-C7q

After the US election, the twitter competitor bluesky suddenly gets a surge of new users:

https://x.com/robertwiblin/status/1858991765942137227

This is my first comment on my own, i.e., Leon Lang's, shortform. It doesn't have any content, I just want to test the functionality.

Edit: This is now obsolete with our NAH distillation [LW · GW].

Making the Telephone Theorem and Its Proof Precise

This short form distills the Telephone Theorem [LW · GW] and its proof. The short form will thereby not at all be "intuitive"; the only goal is to be mathematically precise at every step.

Let $M_0,M_1,\dots$ be jointly distributed finite random variables, meaning they are all functions

starting from the same finite sample space with a given probability distribution $P$ and into respective finite value spaces $M_i$. Additionally, assume that these random variables form a Markov chain $M_0\to M_1\to \dots$. 

Lemma: For a Markov chain $M_0\to M_1\to M_2$, the following two statements are equivalent:

(a) $I(M_1;M_0)=I(M_2;M_0)$

(b) For all $m_1,m_2$ with $P(m_1,m_2)>0$: $P(M_0\mid m_1)=P(M_0\mid m_2)$. 

Proof:

Assume (a): Inspecting an information diagram of $M_0,M_1,M_2$ will immediately result in us also observing the Markov chain $M_0\to M_2\to M_1$. Markov chains can be turned around, thus we get the two chains

Factorizing along these two chains, we obtain:

and thus, for all $m_1,m_2$ with $P(m_1,m_2)>0:$ $P(M_0\mid m_1)=P(M_0\mid m_2)$. That proves (b).

Assume (b): We have

where, in the second step, we used the Markov chain $M_0\to M_1\to M_2$ and in the third step, we used assumption (b). This independence gives us the vanishing of conditional mutual information:

Together with the Markov chain $M_0\to M_1\to M_2$, this results, by inspecting an information diagram, in the equality $I(M_1;M_0)=I(M_2;M_0)$.  $\square$

Theorem: Let $n\ge 1$.  The following are equivalent:

(a) $I(M_n;M_0)=I(M_{n+1};M_0)$

(b) There are functions $f_n,f_{n+1}$ defined on $M_n,M_{n+1}$, respectively such that:

• $f_n\left(M_n\right)=f_{n+1}\left(M_{n+1}\right)$ with probability $1$, i.e., the measure of all $\omega \in \Omega$ such that the equality doesn't hold is zero.
• For all $m_n\in M_n$, we have the equality $P(M_0\mid M_n=m_n)=P(M_0\mid f_n(M_n)=f_n(m_n\left)\right)$, and the same for $n+1$.

Proof: The Markov chain immediately also gives us a Markov chain $M_0\to M_n\to M_{n+1}$, meaning we can without loss of generality assume that $n=1$. So let's consider the simple Markov chain $M_0\to M_1\to M_2$.

Assume (a): By the lemma, this gives us for all $m_1,m_2$ with $P(m_1,m_2)>0$: $P(M_0\mid m_1)=P(M_0\mid m_2)$. 

Define the two functions $f_i:M_i\to \Delta \left(M_0\right)$, $i=1,2$ by:

Then we have $f_1\left(m_1\right)=f_2\left(m_2\right)$ with probability 1^[1], giving us the first condition we wanted to prove. 

For the second condition, we use a trick from Probability as Minimal Map [LW · GW]: set $p:=f_1\left(m_1\right)$, which is a probability distribution. We get

That proves (b).

Assume (b): For the other direction, let $m_1,m_2$ be given with $P(m_1,m_2)>0.$ Let $\omega \in \Omega$ be such that $\left(M_1\right(\omega ),M_2(\omega \left)\right)=(m_1,m_2)$ and with $P\left(\omega \right)>0$. We have

and thus

The result follows from the Lemma.^[2]

1. ^{^}

   Somehow, my brain didn't find this obvious. Here is an explanation: 

   $P\left(\right\{\omega \mid f_1\left(M_1\right(\omega \left)\right)\ne f_2\left(M_2\right(\omega \left)\right)\left\}\right)\le P\left(\right\{\omega \mid P\left(M_1\right(\omega ),M_2(\omega \left)\right)=0\left\}\right)$

   $=P\left(\right(M_1,M_2{)}^{-1}\left(\right\{(m_1,m_2)\mid P(m_1,m_2)=0\left\}\right))={\sum }_{m_1,m_2:P(m_1,m_2)=0}P((M_1,M_2{)}^{-1}(m_1,m_2\left)\right)$

   $={\sum }_{m_1,m_2:P(m_1,m_2)=0}P(m_1,m_2)=0$

2. ^{^}

   There is some subtlety about whether the random variable $f_1\left(M_1\right)$ can be replaced by $f_2\left(M_2\right)$ in that equation. But given that they are "almost" the same random variables, I think this is valid inside the probability equation.

These are rough notes trying (but not really succeeding) to deconfuse me about Alex Turner's diamond proposal [AF · GW]. The main thing I wanted to clarify: what's the idea here for how the agent remains motivated by diamonds even while doing very non-diamond related things like "solving mazes" that are required for general intelligence?

• Summarizing Alex's summary:
  • Multimodal SSL initialization
  • recurrent state, action head
  • imitation learning on humans in simulation, + sim2real
    • low sample complexity
    • Humans move toward diamonds
  • policy-gradient RL: reward the AI for getting near diamonds
    • the recurrent state retains long-term information
  • After each task completion: the AI is near diamonds
• SSL will make sure the diamond abstraction exists
• Proto Diamond shard:
  • There is a diamond abstraction that will be active once a diamond is seen. Imagine this as being a neuron.
  • Then, hook up the "move forward" action with this neuron being active. Give reward for being near diamonds.  Voila, you get an agent which obtains reward! This is very easy to learn! More easily than other reward-obtaining computations
  • Also, other such computations may be reinforced, like "if shiny object seen, move towards it" --- do adversarial training to rule those out
  • This is all about prototypical diamonds. Thus, the AI may not learn to create a diamond as large as a sun, but that's also not what the post is about.
• Preserving diamond abstraction/shard:
  • In Proto planning, the AI primarily thinks about how to achieve diamonds. Such thinking is active across basically all contexts, due to early RL training. 
  • Then, we will give the AI other types of tasks, like "maze solving" or "chess playing" or anything else, from very easy to very hard.
    • At the end of each task, there will be a diamond and reward.
    • By default, at the start of this new training process, the diamond shard will be active since training so far ensures it is active in most contexts. It will bid for actions before the reward is reached, and therefore, its computations will be reinforced and shaped. Also, other shards will be reinforced (ones that plan how to solve a maze, since they also steer toward the reinforcement event), but the diamond shard is ALWAYS reinforced.
      • The idea here is that the diamond shard is contextually activated BY EVERY CONTEXT, and so it is basically one huge circuit thinking about how to reach diamonds that simply gets extended with more sub-computations for how to reach diamonds.
      • Caveat: another shard may be better than the diamond shard at planning toward the end of a maze than the diamond shard which "isn't specialized". And if that's the case, then reinforcement events may make the diamond shard continuously less active in maze-solving contexts until it doesn't activate anymore at start-of-maze contexts. It's unclear to me what the hypothesis is for how to prevent this. 
        • Possibly the hypothesis is captured in this paragraph of Alex, but I don't understand it: 
          "In particular, even though online self-supervised learning continues to develop the world model and create more advanced concepts, the reward events also keep pinging the invocation of the diamond-abstraction as responsible for reward (because insofar as the agent's diamond-shard guides its decisions, then the diamond-shard's diamond-abstraction is in fact responsible for the agent getting reward). The diamond-abstraction gradient starves the AI from exclusively acting on the basis of possible advanced "alien" abstractions which would otherwise have replaced the diamond abstraction. The diamond shard already gets reward effectively, integrating with the rest of the agent's world model and recurrent state, and therefore provides "job security" for the diamond-abstraction. (And once the agent is smart enough, it will want to preserve its diamond abstraction, insofar as that is necessary for the agent to keep achieving its current goals which involve prototypical-diamonds.)"
        • I don't understand what it means to "ping the invocation of the diamond-abstraction as responsible for reward". I can imagine what it means to have subcircuits whose activation is strengthened on certain inputs, or whose computations (if they were active in the context) are changed in response to reinforcement. And so, I imagine the shard itself to be shaped by reward. But I'm not sure what exactly is meant by pinging the invocation of the diamond abstraction as responsible for reward. 

This is my first short form. It doesn't have any content, I just want to test the functionality.