This is definitely the core challenge of the language model approach, and may be the reason it fails. I actually believe language models aren't the most likely approach to achieve superintelligence. But I also place a non-trivial probability on this occurring, which makes it worth thinking about for me.

Let me try to explain why I don't rule this possibility out. Obviously GPT-3 doesn't know more than a human, as evident in its sub-human performance on common tasks and benchmarks. But suppose we instead have a much more advanced system, a near-optimal sequence predictor for human-written text. Your argument is still correct - it can't output anything more than a human would know, because that wouldn't achieve minimum loss on the training data. But does that imply it can't know more than humans? That is, is it impossible for it to make use of facts that humans don't realize as an intermediate step in outputting text that only includes facts humans do realize?

I think not necessarily. As an extreme example, one particular optimal sequence predictor would be a perfect simulation, atom-for-atom, of the entire universe at the time a person was writing the text they wrote. Trivially, this sequence predictor "knows" more than humans do, since it "knows" everything, but it will also never output that information in the predicted text.

More practically, sequence prediction is just compression [LW · GW]. More effective sequence prediction means more effective compression. The more facts about the world you know, the less data is required to describe each individual piece of text. For instance, knowing the addition algorithm is a more space-efficient way to predict all strings like "45324 + 58272 =" than memorization. As the size of the training data you're given approaches infinity, assuming a realistic space-bounded sequence predictor, the only way its performance can improve is with better world/text modeling. The fact that humans don't know a certain fact wouldn't prohibit it from being discovered if it allows more efficient sequence prediction.

Will we reach this superhuman point in practice? I don't know. It may take absurd amounts of computation and training data to reach this point, or just more than alternative approaches. But it doesn't seem impossible to me in theory.

Even if we reach this point, this still leaves the original problem - the model will not output anything more than a human would know, even if it has that knowledge internally. But even without fancy future interpretability tools, we may be heading in that direction with things like InstructGPT, where the model was fine-tuned to spit out things it was capable of saying, but wouldn't have said under pure sequence prediction.

This whole argument, together with rapid recent progress, is enough for me to not immediately write off language models, and consider strategies to take advantage of them if this scenario were to occur.