Bootstrapping Language Models

The quality of the filter determines the upper-bound of performance. If the filter cannot discriminate high-quality stories from extremely-high-quality stories, it seems that performance will increase until all the stories produced by the model are indistinguishable to the filter. If humans are doing the filtering, then the bootstrapping process will be limited by people's ability to differentiate between really good short stories.

The quality of the filter certainly matters but this is a narrow way to look at it, and the filter may not be all that important.

When I think about the tangled topics of expert iteration, self-play, self-distillation, inner-monologue, finetuning, meta-learning, sim2real, domain randomization, iterated amplification, etc, I tend to ask, "where is new information coming from?" The quality of the filter is far downstream of the big picture.

For example, in AlphaGo/MuZero, there is not really any 'filter', and also we know that the bootstrapping process doesn't asymptote at any "upper bound determined by the quality of the filter". (It doesn't really 'asymptote' at all, inasmuch as far as we know, you can just keep pouring on the compute to train a bigger model to distill even more accurate tree search, and available evidence like Jones shows similar scaling law behavior.) AG doesn't need any new information because all the complexity of Go is available by interrogating the simulator; MuZero learns that, but it only needs to learn a little bit, the rules of Go, and it can do that either by some offline learned data like human games if diverse enough to represent all relevant state transitions/rewards, or carefully targeted environment interactions to play games triggering parts of the rules it's unsure of, and then it's off to the races. (Indeed, an agent hypothetically could solve Go to superhuman level even without adequate data to nail down the rules, if it has enough compute, because it can treat it as a meta-learning problem: you have a distribution over possible rules of Go, which you then attempt to master many versions of; then after you are done, you sit down with a human and perhaps use your text module to enquire, "by the way, what's komi these days?" "7.5." "Ah, thanks.", and, having figured out which MDP your POMDP plopped you down in this time, then proceed to crush them. If it's not as simple as a binary uncertainty between 6.5 and 7.5 komi where you just train half the game conditional on a komi feature set to 6.5 and the other half with it set to 7.5, you can do posterior sampling and sample a possible environment-model/rule-sets to diversify training over.) We can imagine this for a lot of other situations: for example, a Codex/AlphaCode model could potentially self-train by learning to emulate a REPL and then using its GPT-3-based natural language to pose itself arbitrarily many problems. (AlphaCode in particular could probably use way less than "tens of thousands" of generated samples per problem if it did some finetuning on its self-ranked solutions.)

On the other hand, you could look at OA's GPT-3 French self-distillation work where it prompts for French translations from a large model and finetunes a small model on it, getting the small model to translate much better than before with no need for few-shots, and note that the sample translations of straightforward sentences can contain only very little information about the French language; the small GPT-3 can't have learned all that much about French from it. French is not an algorithm which can generate new information by being executed, nor is either GPT-3 talking to a French person, nor are they looking up French text like WebGPT can, nor can French be nailed down to a few possible tasks which can be learned autonomously with no further information; it's a completely closed circle. So why did it work? Because the small GPT-3 is dumb and isn't sure which French task it is predicting: it ensembles over countless sub-models each encoding a possible 'French task', and the right one is towards the top of the rankings, but not the very top. The finetuning on GPT-3 samples, from a GPT-3 that has figured out which French task it is predicting, is enough to boost the good sub-models to victory. This boost might have taken a very long time if you had instead kept training the small GPT-3 on random Internet data (French being a small % of it, and texts providing French-English pairs a small % of that), but the very targeted boost lets the small model match the model orders of magnitude bigger. The capability was there, lurking under the surface, even if the waters seemed calm, waiting for a motive to strike.

This is also true of approaches like on a corpus of questions doing best-of=20 sampling, taking the majority vote (because so many samples will go awry but the majority will be more right), and retraining with the majority vote as the 'groundtruth', to try make it generate the majority completion every time. Simply babbling 20 times and convincing yourself that the thing you babbled 11⁄20 times is THE TRUTH™ cannot bring in any new information, but it can penalize bad sub-models and boost good ones to lock in correctness.

Like a surprise shark attack, however, this is a trick that only works once. Once the right sub-model has been boosted, then you're done. If its best is not good enough, then it's not good enough. There might be some benefits from looping a few more times to tidy things up or lock in the gains, but you cannot amplify yourself arbitrarily. (You either know your French vocab or you don't, and all the thinking in the world will not tell you the arbitrarily-decreed grammatical gender of all French nouns if you didn't at least get a hint of it during your original training.)

From this perspective, prompts like "English to French:" or "let's think this out step by step" are something of a 'prompt overhang': we (still) don't take prompt engineering seriously enough, so we think you have to have things like InstructGPT or elaborate few-shot 'Chain-of-Thought' prompts to induce inner monologue reasoning; unsurprisingly, we always underestimate the true performance of models when we do things in this careless cackhanded sort of way. (Remember: "sampling can prove the presence of knowledge but not the absence.") Hopefully over the next few years we'll get better at this, now that we know to expect these things and can begin automating them or surfacing them, and there will be fewer sharks just below the surface.

(Note to what an extent any 'filter' involved in all of this is a complete afterthought. In some cases like self-distillation of an ensemble's predictions, there is nothing even remotely like a 'filter': all inputs go in, the average of all outputs comes out.)

But relative to other techniques like scaling, it seems less likely to lead to dangerous increases in capability.

So how dangerous is this whole family of approaches? Well, that depends...

• Where does new data come from? Is it accessible, or old data in such quantities as to be new?

• Is this a task where additional information can be generated by computing, like a simulated game or robotics task? Does 'scaling' here encompass one's self-play and expert-iteration?

• Is this a task where a relatively small amount of data could enable in silico thinking, such as meta-learning?

  • Is this a task where a few very carefully-chosen, Bayes-optimal, design-of-experiment-style samples can eliminate uncertainty and a model go from working in silico to working IRL near-instantaneously?

• Is there an 'overhang' where a large model is smart enough to do it by default but a small model could do it and just happens to not do it (yet)?

  • Is this a task where the model has been trained specifically on heavily, so that there are no sub-models lurking below the water's surface, and WYSIWYG?

So a concrete example would be a Codex model. It's trained to generate code. But it could also be used to discover vulnerabilities & hack stuff. A Codex model will have learned a lot about hacking simply because many patches are security fixes, many pieces of code are security tools, there's a lot of discussion in comments etc, but it's not in any way specialized for hacking; that is merely one of the myriads of capabilities it's picked up along the way as part of its pretraining. You could prompt it to try to write a hack of some code. It'll sometimes do it but especially if the prompt is unnatural, this is not very likely to work. If you scale up Codex much further, it'll get better, and it will probably do so at a rate that looks power-law/logarithmic like everything else, so you'll need to scale it 10x or 100x to get large hacking gains. It will get better because it's gotten better at - along with everything else coding related - picking up what an odd prompt means, and because it's done better at locating coding & hacking sub-models, and this may achieve your goals. But those sub-models might well be the same ones that you could boost to the surface with some self-distillation on the original small model. On the other hand, maybe it hasn't learned those sub-models well enough to make them realistically locatable and there is a hard phase transition underneath the surface which hasn't yet happened, in which case its hacking performance won't improve much at all. Which one is it?

¯\_(ツ)_/¯ I have no way of knowing. It's a purely empirical question. The 'distance' between small models and large models will depend on how large the scales are, where the phase transitions in knowledge/capability are, where you can trade compute / data, how incidental the target task is to the pretraining task which creates a gap between latent capabilities and obviously-promptable capabilities, how good your search and self-distillation techniques are...

I don't find this particularly comforting from a safety perspective because it implies that differences in size which look impressive could in fact be highly illusory. You could easily have a model 100x smaller - "wow, that seems very safe and harmless, we'd better worry about the large one" - which nevertheless can, when prompted just the right way, beat the large one.

For investigation of the kind of thing you suggest, take a look at Anthropic's "A General Language Assistant as a Laboratory for Alignment" and more importantly "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback".

They focus on training a helpful / harmless assistant rather than good short stories, but using human-filtered model-output to improve behavior is the basic paradigm.