The Data Wall is Important

with the best models trained on close to all the high quality data we’ve got.

Is this including images, video and audio? Or just text?

"Data wall" is not about running out of data, at least if we are targeting a level of capabilities rather than cheaper inference with overtrained models. In the sense of literally running out of data, there is no data wall [LW(p) · GW(p)] because data can be repeated in training, and with Chinchilla scaling an increase in the amount of data (including repetition) results in quadratic increase in compute. This makes even publicly available data sufficient for $100 billion training runs. We are going to run out of compute/funding before we run out of data (absent autonomous long-horizon agents, but then resulting faster algorithmic progress makes straightforward scaling of LLMs less relevant).

The reason repeated data is rarely heard about is that there is still enough unique data at current scale. And until recently the prevailing wisdom was apparently that for LLMs repeating data is really bad [LW(p) · GW(p)], possibly until the Galactica paper, see Figure 6. But where data is more scarce, like code data in StarCoder 2, this technique is already in use (they train for up to 5 epochs). Repeating about 4 times is no worse than having 4 times more unique data, and repeating about 16 times is still highly useful (see Figure 5 in the above paper on scaling laws for repeated data).

So concerns about "data wall" are more about doing better on training quality per unit of data, whether natural or synthetic. It's about compute efficiency rather than sample efficiency with respect to external data. In which case it would be useful to apply such techniques even if there was unlimited unique data, as they should allow training smarter systems with less compute.

Modern AI is trained on a huge fraction of the internet

I want to push against this. The internet (or world wide web) is incredibly big. In fact we don't know exactly how big it is, and measuring the size is a research problem!

When they say this what they mean is it is trained on a huge fraction of Common Crawl. Common Crawl is a crawl of world wide web that is free to use. But there are other crawls, and you could crawl world wide web yourself. Everyone uses Common Crawl because it is decent and crawling world wide web is itself a large engineering project.

But Common Crawl is not at all a complete crawl of world wide web. It is very far from being complete. For example, Google has its own proprietary crawl of world wide web (which you can access as Google cache). Probabilistic estimate of size of Google's search index suggests it is 10x size of Common Crawl. And Google's crawl is also not complete. Bing also has its own crawl.

It is known that Anthropic runs its own crawler called ClaudeBot. Web crawl is highly non-trivial engineering project, but it is also kind of well understood. (Although I heard that you continue to encounter new issues as you approach more and more extreme scales.) There are also failed search engines with their own web crawls and you could buy them.

There is also another independent web crawl that is public! Internet Archive has its own crawl, it is just less well known than Common Crawl. Recently someone made a use of Internet Archive crawl and analyzed overlap and difference with Common Crawl, see https://arxiv.org/abs/2403.14009.

If the data wall is a big problem, making a use of Internet Archive crawl is like the first obvious thing to try. But as far as I know, that 2024 paper is first public literature to do so. At least, any analysis of data wall should take into account both Common Crawl and Internet Archive crawl with overlap excluded, but I have never seen anyone doing this.

My overall point is that Common Crawl is not world wide web. It is not complete, and there are other crawls, both public and private. You can also crawl yourself, and we know AI labs do. How much does it help is unclear, but I think 10x is very likely, although probably not 100x.

Here is a report from epoch on the quantity of data and when we will likely run out.

(I think you probably meant to link this in a footnote.)

I suspect that this will result in more spying on people, because that's where more human-generated data are.

I assume that everyone already trains on the public internet, plus Library Genesis, plus Sci-Hub. Where to get more data? Google can read most people's e-mails, and will probably use YouTube videos for video. Microsoft can read Teams messages, plus there is the new feature in Windows 11 that records everything you do. Microphones and cameras on your computers and smartphones can get more data, so I wonder if we will soon get devices where you cannot turn these off. Every other device (e.g. a smart TV) will probably also spy on you as much as they can, because even if they are not in the AI business, they can still sell the data to those who are.

We're definitely on/past the inflection point of the GPT S-curve, but at the same time there's been work on other AI methods and we know they're in principle possible. What is needed to break the data wall is some AI that can autonomously collect new data, especially in subjects where it's weak.

One key question is how much investment AI companies make into this. E.g. the obvious next step to me would be to use a proof assistant to become superhuman at math by self-play. I know there's been a handful of experiments with this, but AFAIK it hasn't been done at huge scale or incorporated into current LLMs. Obviously they're not gonna break into the next S-curve until they try to do so.

I wonder if it even economically pays to break the data-wall. Like, let's say an AI company could pivot their focus to breaking the data-wall rather than focusing on productivizing their AIs. That means they're gonna lag behind in users, which they'd have to hope they could make up for by having a better AI. But once they break the data-wall, competitors are presumably gonna copy their method.

For good analysis of this, search for the heading “The data wall” here.

Did you mean to insert a link here?