Situational Awareness Summarized - Part 1

post by Joe Rogero · 2024-06-06T18:59:59.409Z · LW · GW · 0 comments

Contents

  Part I: OOMs go Zoom
    Factors in Effective Compute
    Potential Bottlenecks
      The Data Wall
      Future Unhobblings
    The Drop-In Remote Worker
  Some questions I have after reading
None
No comments

This is the first post in the Situational Awareness Summarized [? · GW] sequence. Collectively, these posts represent my attempt to condense Leopold Aschenbrenner's recent report, Situational Awareness, into something more digestible. I'd like to make it more accessible to people who don't want to read 160 pages. 

I will not attempt to summarize the introduction, which is already brief and worth a full read. 

Disclaimer: As of a few weeks ago, I work for the Machine Intelligence Research Institute. Some of MIRI's basic views regarding AI policy can be found here, and Rob Bensinger wrote up a short response to Leopold's writeup here [EA · GW]. I consider Rob's response representative of the typical MIRI take on Leopold's writeup, whereas I'm thinking of this sequence as "my own personal take, which may or may not overlap with MIRI's." In particular, my questions and opinions (which I relegate to the end of each post in the sequence) don't necessarily reflect MIRI's views. 

Part I: OOMs go Zoom

This section covers the past and future speed of AI scaling. In it, Leopold traces the rapid evolution of AI capabilities in the past four years and attempts to extrapolate this progress into the near future. His primary unifying metric is orders of magnitude (OOMs) of effective compute. 

In principle, each OOM represents a tenfold increase in computational power. However, in order to address harder-to-measure factors like algorithmic progress and "unhobbling", Leopold attempts to estimate the effects of these factors on overall scaling and reports the result as effective compute. 

Focused through the lens of OOMs, Leopold weaves a vivid tapestry that follows the evolution of the cute and somewhat pathetic GPT-2 into the benchmark-shattering monolith that is GPT-4. He outlines the factors that contributed to this rapid growth, and makes the case that they will probably continue to operate for at least the next few years. The conclusion? AGI is coming, and soon. 

Factors in Effective Compute

How do we get there? To start, Leopold highlights three factors that add to the total "OOMs of effective compute" metric. 

Compute: the approximate number of floating-point operations (FLOPs) used to train each new generation of language model. The article estimates that this has increased by 3,000-10,000x, or about 3.5-4 OOMs, from GPT-2 to GPT-4, and will probably increase another 2-3 OOMs by 2027. 

Algorithmic efficiency: marginal improvements in machine learning science that allow models to accomplish similar tasks with less compute. These include advances in data use, training stack, and architecture changes like Mixture of Experts. Leopold estimates that these gains have added 1-2 OOMs of effective compute, and that we're on track to see another 1-3 OOMs by 2027. 

Unhobbling: new approaches that unlock the latent capabilities of models. This is Leopold's catchall term for such paradigm-shifting developments as Chain-of-Thought prompting (CoT), reinforcement learning with human feedback (RLHF), and access to tools like web search. It also includes further enhancements in the form of scaffolding, larger context windows, and posttraining improvements. Leopold estimates that these developments magnified effective compute by about 2 OOMs, but acknowledges that the error bars on this number are very high. 

Putting it all together, Leopold expects the future to look something like this: 

Potential Bottlenecks

Leopold devotes considerable energy to addressing two major questions that might affect the speed of progress: 

The Data Wall

In training, Llama 3 grazed on approximately the entire useful corpus of Internet text (~15 trillion tokens). Chinchilla scaling laws suggest we need twice as much data to efficiently train a model twice as large. But after digesting a meal of that size, what's left to feed our hungry herd of next-gen LLMs? Are we, against all prior expectations, actually going to run out of internet? 

Leopold thinks not. He offers two main arguments for why the data wall might be a surmountable obstacle: 

To illustrate the second point, Leopold points to in-context learning and self-play. Current LLMs, he argues, merely skim everything we feed them, like a college student speed-reading all their textbooks in the first month of school. What if we pointed LLMs at a smaller quantity of high-quality data, and gave them ways to reflect and study the content, like a student slowly pondering the problems in a math textbook? AlphaGo learned most of its tricks by playing against itself; could future models get similar gains through self-study? 

Leopold also points out that, even if the data wall is climbable, methods of overcoming it might prove to be highly protected and proprietary secrets. This might lead to increased variance in the capabilities of different AI labs, as each makes and hoards a different set of breakthroughs for working with limited data. 

Future Unhobblings

Despite recent gains, Leopold argues, AI is still very hobbled today. He expects major overhangs to be unlocked in the near future. Some key examples include: 

Leopold does think that these and other bottlenecks will slow progress eventually. Increases in spending, hardware specialization, and low-hanging algorithms can only take us so far before we (presumably) consume our overhang and slow down. But in the next few years at least, we're on track to see rapid and potentially transformative growth. Which brings us to...

The Drop-In Remote Worker

AIs of the future, Leopold suggests, may look less like "ChatGPT-6" and more like drop-in remote workers - millions of fully autonomous, computer-savvy agents, each primed with the context of a given organization, specialty, and task set. And it could be a sudden jump, too; before models reach this point, integrating them into any sort of company may require a lot of hassle and pretraining. But after they reach the level of remote human workers - a level Leopold suspects is coming fast - they will be much easier to employ at scale. 

Astute readers will notice that this particular scenario is not so much an endpoint as a recipe for truly staggering progress. The implications are not lost on Leopold, either; we'll cover what he thinks will come next in Part II. 

Some questions I have after reading

Below are some questions and uncertainties I still have after reading this section. 

0 comments

Comments sorted by top scores.