Renormalization: Why Bigger is Simpler

The phase transitions I had in mind were more "Foom" scenarios than sigmoid skill curves, though a similar logic may at times apply to both:

While any one output of the model is highly complex and contextual, the impact of a model will mainly come from it giving multiple outputs in a row. This is where renormalization comes in - these many outputs will add up, and the way they add up may cause them to undergo phase transitions.

For instance with "Foom", one phase transition would be how the AI's output impacts its power in the world. Currently, nice outputs in training may cause it to get deployed, but beyond that its power is fairly fixed. However, if an individual output could increase the AI's power in the world slightly on average, then repeated outputs could renormalize into a "Foom" scenario where it rapidly takes control. (At least in principle; there are all sorts of complications to this question which have been discussed at length in Foom debates.)

Arguably, though it is not what I had in mind when I wrote the comment, renormalization also applies to sigmoid skill curves. Usually, solving a task requires outputting multiple correct words in a row. So to get 50% accuracy on a task that requires outputting 20 correct words in sequence, it needs 0.5^(1/20)=96.5% accuracy on each individual word. There's a phase transition above 90% accuracy on the individual words where it rapidly acquires the skill, due to the skill depending on repeated outputs in a row.