Sama Says the Age of Giant AI Models is Already Over

As far as I can tell, Sam is saying no to size. That does not mean saying no to compute, data, or scaling.

"Hundreds of complicated things" comment definitely can't be interpreted to be against transformers, since "simply" scaling transformers fits the description perfectly. "Simply" scaling transformers involves things like writing a new compiler. It is simple in strategy, not in execution.

The poll appears to be asking two, opposite questions. I'm not clear on whether a 99% means it will be  a transformer or whether it means something else is needed to get there?

Maybe Sam knows a lot I don't know but here are some reasons why I'm skeptical about the end of scaling large language models:

• From scaling laws we know that more compute and data reliably lead to better performance and therefore scaling seems like a low-risk investment.
• I'm not sure how much GPT-4 cost but GPT-3 only cost $5-10 million which isn't much for large tech companies (e.g. Meta spends billions on the metaverse every year).
• There are limits to how big and expensive supercomputers can be but I doubt we're near them. I've heard that GPT-4 was trained on ~10,000 GPUs which is a lot but not an insane amount (~$300m worth of GPUs). If there were 100 GPUs/m^2, all 10,000 GPUs could fit in a 10m x 10m room. A model trained with millions of GPUs is not inconceivable and is probably technically and economically possible today.

Because scaling laws are power laws (x-axis is logarithmic and y-axis is linear), there are diminishing returns to resources like more compute but I doubt we've reached the point where the marginal cost of training larger models exceeds the marginal benefit. Think of a company like Google: building the biggest and best model is immensely valuable in a global, winner-takes-all market like search.

I was chatting with a friend of mine who works in the AI space.  He said that the big thing that got them to GPT-4 was the data set; which was basically the entire internet.  But now that they've given it the entire internet, there's no easy way for them to go further along that axis;; that the next big increase in capabilities would require a significantly different direction than "more text / more parameters / more compute".

Maybe temporarily efficiency improvements will rule, but surely once the low and medium hanging fruit is exhausted parameter count will once again be ramped up, would bet just about anything on that