Deepmind's Gopher--more powerful than GPT-3

Just to give the full quote:

And I'd reject LSTM -> transformer or MoE as an example because the quantitative effect size isn't that big.

But if something like that made the difference between "this algorithm wasn't scaling before, and now it's scaling," then I'd be surprised.

And the size of jump that surprises me is shrinking over time. So in a few years even getting the equivalent of a factor of 4 jump from some clever innovation would be very surprising to me.

The text you quoted was clarifying what "factor of 4" means in that sentence.

I'm not surprised by "googling relevant terms and then putting the results in context improves language modeling loss and performance on knowledge-loaded tasks." This looks like basically a great implementation of that idea, along with really solid LM infrastructure in general.

I don't really even have a firm quantitative prediction of how much this kind of thing will improve the LM loss in the world "in a few years" that the quote describes. Note that the effect of this result on downstream performance is almost certainly (much) less than its effect on LM loss, because for most applications you will already be doing something to get relevant information into the LM context (especially for a task that was anywhere near as knowledge-loaded as the LM task, which is usually pretty light on reasoning).

(ETA: as Veedrac points out [LW(p) · GW(p)], it also looks on a first skim like quite a lot of the difference is due to more effectively memorizing nearly-identical text that appeared in the training set, which is even less helpful for downstream performance. So sticking with "even if this is a 10x gain on LM task according to the formal specification, it's not nearly such a big deal for downstream tasks.")

My logic for making predictions about this kind of thing is roughly:

1. In the next few years LM inference will be using large amounts of compute, with costs likely measured in hundreds of millions per year.
2. Engineering effort to improve performance on the applications people care about is likely to be in the hundreds of millions or billions.
3. If low-hanging fruit like this hasn't been plucked at that point, people are probably correct that it's not going to be a huge effect relative to the things they are spending money on.
4. People will probably be in the regime where doubling model quality costs more like $100M than $10M. Sometimes you will get lucky and find a big winner, but almost never so lucky as "clever trick that's worth 4x. (Right now I think we are maybe at $10M/doubling though I don't really know and wouldn't want to speak specifically if I did.)
5. (And it won't be long after that before doubling model quality for important tasks costs $1B+, e.g. semiconductors are probably in the world where it costs $100B or something to cut costs by half. You'll still sometimes have crazy great startups that spend $100M and manage to get 4x, which grow to huge valuations quickly, but it predictably gets rarer and rarer.)

Also worth noting is that the model was trained in December 2020, a year ago. I don't know when GPT-3 was trained, but if the time-gap between the two is small, that sure looks like a substantial discontinuity in training efficiency. (Though I'd prefer to see long-run data).

Skimming the Rᴇᴛʀᴏ paper is weird because it looks like there's leakage everywhere, they admit leakage is everywhere, but then they sort of report results like it doesn't matter, even putting a result on their leakiest dataset in their conclusion?

On Wikitext103 and the Pile, Retro outperforms previous models trained on large scale datasets.

It looks to me like Figure 6 is saying the improvement is fairly modest in unleaky datasets?

Maybe someone who has gone over the paper in detail can chime in with thoughts.

However, he was specifically referring to a clever innovation which occurs several years in the future. If the DeepMind’s 10x more efficient claim holds up, is that a bigger jump than Paul Christiano predicted would be plausible today?

I'd also be interested in hearing Paul/Eliezer's takes on RETRO, though I don't think they reported compute scaling curves in their paper? 

Headline quote:

With a 2 trillion token database, our Retrieval-Enhanced Transformer (Retro) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25× fewer parameters.

Relevant figures for performance are figure 1 and figure 3

280 billion parameters is still far less than the human brain. It’s closer to a rat’s brain. Maybe even smaller than that.

RE: "like I'm surprised if a clever innovation does more good than spending 4x more compute"

Earlier this year, DeBERTaV2 did better on SuperGLUE than models 10x the size and got state of the art.

Models such as DeBERTaV3 can do better than on commonsense question answering tasks than models that are tens or several hundreds of times larger.

DeBERTaV3-large

Accuracy: 84.6   1  Parameters: 0.4B

T5-11B

Accuracy: 83.5  1  Parameters: 11B

Fine-tuned GPT-3

73.0  1  175B

https://arxiv.org/pdf/2112.03254.pdf#page=5

Bidirectional models + training ideas + better positional encoding helped more than 4x.