Language models can generate superior text compared to their input

Arbitrarily good prediction of human-generated text can demand arbitrarily high superhuman intelligence.

Simple demonstration #1:  Somewhere on the net, probably even in the GPT training sets, is a list of <hash, plaintext> pairs, in that order.

Simple demonstration #2:  Train on only science papers up until 2010, each preceded by date and title, and then ask the model to generate starting from titles and dates in 2020.

This is a really good point, but it only shows that superhuman reasoning might be possible, not that it is. Like, it's possible to the extent that the transition functions humans can produce are restricted by noise rather than bias. But it's unclear (at least to me) why bias can't be most of the story.

My experience over the past few years has been one of being surprised by latent capacities in existing models. A lot of stuff like prompt engineering, fine tuning, chain of thought, Open-AI-style "alignment" can be seen as not so much creating new capacities as revealing/refining latent ones. Back when GPT-3 was new, Connor Leahy said something like "GPT-3 is already general intelligence" which sounded like hyperbole to me at the time, and seems less so now. 

Though RSI still seems very plausible to me [LW · GW], one scenario I've started thinking about is a massive effective capabilities gain caused not by RSI or any non-trivial algorithmic improvement, but just the dissolution of a much larger than anticipated  "latent capacities overhang".

Possibly an absurd and confused scenario, but is it that implausible that some day we will get a model that still seems kinda dumb but is in fact one prompt away from super-criticality?

One of top bloggers in my bubble said that he tries to write a most surprising next sentence when he is working on his posts. 

You've failed to convince me that "Lideal - Ltext is noise", or even offer any arguments for that. Could you elaborate more please? This seems potentially very interesting and relevant.

True.

Something which can predict everything all the time is more formidable than any demonstrator it predicts: the upper bound of what can be learned from a dataset is not the most capable trajectory, but the conditional structure of the universe implicated by their sum (though it may not be trivial to extract that knowledge).

-- Simulators [LW · GW]

The fair-goers, having knowledge of oxen, had no bias in their guesses

[EDIT: I read this as "having no knowledge of oxen" instead of "having knowledge of oxen" - is this what you meant? The comment seems relevant nevertheless.]

This does not follow: It is entirely possible that the fair-goers had no specific domain knowledge of oxen, while still having biases arising from domain-general reasoning. And indeed, they probably knew something about oxen -- from Jaynes' Probablity Theory:

The absurdity of the conclusion [that polling billion people tells the height of China's emperor with accuracy 0.03 mm] tells us rather forcefully that the √N rule is not always valid, even when the separate data values are causally independent; it is essential that they be logically independent. In this case, we know that the vast majority of the inhabitants of China have never seen the Emperor; yet they have been discussing the Emperor among themselves, and some kind of mental image of him has evolved as folklore. Then, knowledge of the answer given by one does tell us something about the answer likely to be given by another, so they are not logically independent. Indeed, folklore has almost surely generated a systematic error, which survives the averaging; thus the above estimate would tell us
something about the folklore, but almost nothing about the Emperor.

The main point is correct, but maybe you should mention your demonstration is especially easy, but not necessarily the main reason (unless that’s what you think?). Also:

The fair-goers, having knowledge of oxen, had no bias in their guesses, thus the error was entirely due to random noise.

If you meant the crowd had no bias on average, that’s indeed the idea. But one can read your sentence as meaning that each individual had no bias, which would break the whole wisdom of crowd idea (because then Galton wouldn’t need a crowd: he could simply repeat the measurement process in one individual).

I'm so happy that I didn't go to sleep, because I got to read this masterpiece of an article as soon as it was published.