Before smart AI, there will be many mediocre or specialized AIs

post by Lukas Finnveden (Lanrian) · 2023-05-26T01:38:41.562Z · LW · GW · 2 comments


  More expensive AI → you can run more AIs with your training budget
  A rough lower bound for number of AIs the world could run
  Serial vs parallel
  A few caveats


More expensive AI → you can run more AIs with your training budget

(...assuming that we’re making them more expensive by increasing parameter-count and training data.)

We’re currently in a paradigm where:

This means that, once you have trained a highly capable model, you are guaranteed to have the resources to run a huge number of them in parallel. And the bigger and more expensive the model was — the more of them can run in parallel on your training cluster.

Here’s a rough calculation of how many language models you can run in parallel using just your training cluster:

If you take a horizon-length framework seriously, you might expect that we’ll need more training data to handle longer-horizon tasks. Let’s introduce a parameter H that describes how many token-equivalents correspond to one data-point.

Some example numbers (bolded ones are changed from the top one):

In addition, there are various tricks for lowering inference costs. For example, reducing precision (which is less important during training than inference) and knowledge distillation; see here for more discussion. These would further increase the number of models you can run in parallel.

A rough lower bound for number of AIs the world could run

The bigger the training run, the more AIs you can run with your training cluster. Conversely, if human-level AI comes earlier, with smaller training runs, you’ll be able to run fewer of them with your training cluster.

On the other hand, if a training run is very small, then it’s only using a small fraction of the world’s compute. This means that there’s a lot of room to run many models in parallel just by acquiring more compute. (It would certainly be economically efficient for a large fraction of the world’s compute to run AI systems, if we did have human-level AI — whether that happens via the developers+investors buying more compute, the developers selling their software, a government seizing the software, or some other way.)

Today, there’s about 4e21 FLOP/s out there in the form of GPUs and TPUs (source). Let’s assume that the world would want to run ~human-level AI systems on at least 25% of that (1e21 FLOP/s), given the option. If so, we can get a rough lower bound on how many ~human-level AIs could be run shortly after training by looking at the number of AIs you could run after training an AI on 1e21 FLOP/s, run for a year:

Some caveats in the footnote.[3]

Serial vs parallel

It’s not clear that you can parallelize tasks well enough to make efficient use of 23 million parallel models. To what degree is it possible to run these AIs fast, so that we get them in series after each other?

I don’t understand this super-well. Some relevant information:

In short: We’re currently at 30-40 tokens per second, which will be reduced by bigger model sizes, increased by future hardware, and increased by better techniques.

This is all for generating tokens. Reading content into the context window doesn’t add latency, since the entire context window can be processed in parallel. (Combining this with parallelism is interesting. An AI could split into 10 copies, investigate 10 different lines of thoughts, and then instantly merge and read all thoughts so-far — and then repeat.)

I feel pretty unsure about how that adds up. But if well-optimized future models (running on future hardware) could operate at, say, ~50 tokens per second, then 23 million tokens per second would correspond to ~500,000 separate streams of 50 tokens/second.


The above numbers suggest that (as long as sample efficiency doesn’t significantly improve) the world will always have enough compute to produce at least 23 million token-equivalents per second from any model that the world can afford to train (end-to-end, chinchilla-style). Notably, these are many more token-equivalents per second than we currently have human-AI-researcher-seconds per second. (And the AIs would have the further advantage of having much faster serial speeds.)

So once an AI system trained end-to-end can produce similarly much value per token as a human researcher can produce per second, AI research will be more than fully automated. This means that, when AI first contributes more to AI research than humans do, the average research progress produced by 1 token of output will be significantly less than an average human AI researcher produces in a second of thinking.[6] Instead, the collective’s intelligence will largely come from a combination of things like:

Assuming that much of this happens “behind the scenes”, a human interacting with this system might just perceive it as a single super-smart AI. Nevertheless, I think this means that AI will be more alignable at a fixed level of productivity. (Eventually, we’ll face the full alignment problem — but “more alignable at a fixed level of productivity” helps if we can use that productivity for something useful, such as giving us more time or helping us with alignment research [LW · GW].)

Most obviously, the token-by-token output of a single AI system should be quite easy for humans to supervise and monitor for danger. It will rarely contain any implicit cognitive leaps that a human couldn’t have generated themselves. (C.f. visible thoughts project [LW · GW] and translucent thoughts hypothesis [LW · GW].)

But what about collectives of AIs, or AIs thinking for a long period of time? If people get capability-boosts by fine-tuning such systems end-to-end, then the situation looks quite different. Perhaps it will prove beneficial to finetune such systems to communicate with each other using uninterpretable vector embeddings. Or even if they keep using English, they might start using steganography [LW · GW].

There are still a few reasons for why this situation seems safer (at a fixed level of AI capability) than it could have been:

A few caveats

A big caveat to this is that AI and humans will have different distributions of capabilities.[10] If there are some topics on which AI is much, much better than humans, then humans might not understand AI’s reasoning about that when looking at token-by-token output (even before end-to-end training). And outcomes-based feedback might be necessary to elicit AI’s full capabilities on that topic.

Indeed, it seems plausible that the story of AI automation won’t be one where many low-capability AIs combine to be human-ish. Instead, it might be that AI automates one task at a time, and that use cases where AI isn’t at least as good as humans aren’t ever that important (c.f. Tom Davidson’s takeoff speeds model [LW · GW] and Richard Ngo’s framework [LW · GW]). This would also have implications for the shape of early alignment, and whether early AI systems would help with later alignment — but the analysis might be quite different, and involve thinking in detail about what sort of tasks are likely to be automated in what order. I’d be interested in such analysis.

Acknowledgements: Thanks to Tom Davidson and Daniel Kokotajlo for comments. I work at Open Philanthropy but the views here are my own.


  1. Non-parallelizable training wouldn’t exactly contradict the conclusions here, but it would change what arguments I’d use for them, and it would make the world into a weirder place. (E.g. extra compute wouldn’t help to make smarter models, beyond a point, and AI progress would instead be mostly driven by software, serial time (!) necessary to train models, and maybe inference-time compute, if that was more parallelizable.) ↩︎

  2. According to The longest training run [LW · GW]: “Training runs of large Machine Learning systems are likely to last less than 14-15 months. This is because longer runs will be outcompeted by runs that start later and therefore use better hardware and better algorithms. “. ↩︎

  3. In practice, many of the world’s GPUs wouldn’t be able to efficiently run large models like this, e.g. because of a lack of memory. 25% of the world’s compute is probably an overestimate. On the other hand, specialized hardware is much more important for training than for inference. So if FLOP-supply keeps being dominated by non-specialized hardware, this pushes for more token-equivalents per second, because there would probably be many GPUs you could run your model on that you couldn’t train them on. ↩︎

  4. See page 6 for formula T<sub>comm</sub> = (√ BLF / √nchips) × 4E / network bandwidth. B is batch size; L is sequence length; F is the width-dimension of the feed-forward networks. E is the embedding/activation size. That’s per layer, so latency straightforwardly increases with more layers. But if you simultaneously scale the embedding dimension and the width of the feed-forward networks by 2x, I think you increase overall computation by 2^2=4x. That justifies increasing chips by 4x. But that leads to an overall change in T by (√2/√4) * 2 = √2? So maybe scaling width by 2x increases latency by √2? ↩︎

  5. Chinchilla has 80 (Hoffmann et al., 2022). PaLM has 118 (Chowdhery et al., 2022). ↩︎

  6. This relies on an assumption that you can make up for lack-of-intelligence by numbers or speed. Without that assumption, you could expect that AI research will be dominated by humans until AIs finally “get it”, after which they’ll take over with a huge margin. ↩︎

  7. Typical reading is ~300 wpm = 5 words per second. Typical speaking might be ~half that. ↩︎

  8. One framing of this is: The reason why the bitter lesson applied so strongly in the last few decades is plausibly that compute increased very quickly compared to researcher labor. If AI systems start contributing to AI research, that will correspond to a massive increase in researcher labor, which might reverse the trend. ↩︎

  9. C.f. this comment [LW(p) · GW(p)]. ↩︎

  10. Though as long as the best pre-training task is to predict human text, they’ll be more similar than you might otherwise have expected. ↩︎


Comments sorted by top scores.

comment by Super AGI (super-agi) · 2023-05-29T01:31:07.372Z · LW(p) · GW(p)

We’re currently in paradigm where:


Typo fix -> 

We’re currently in a paradigm where:

Replies from: Lanrian
comment by Lukas Finnveden (Lanrian) · 2023-05-29T16:35:09.383Z · LW(p) · GW(p)

Thanks, fixed.