Posts
Comments
some of the Chinese food samples looked nauseating to me
Filtering for difficulty like that is tricky. In particular the most difficult samples are random noise or Chinese or something that the model can't begin to comprehend.
Some approaches I would consider:
Curriculum learning -- Have a bunch of checkpoints from a smaller GPT. Say the big GPT currently has a LM loss of 3. Then show it the examples where the smaller GPT's loss improved most rapidly when its average loss was 3.
Quality -- Put more effort into filtering out garbage and upsampling high quality corpuses like Wikipedia.
Retrieval -- Let the model look things up when its confused, like MARGE from Pretraining via Paraphrasing does.
No, the number of iterations is irrelevant. You can derive Kelly by trying to maximize your expected log wealth for a single bet. If you care about wealth instead of log wealth, then just bet the house every opportunity you get.
A bigger issue with Kelly is that it doesn't account for future income and debt streams. There should be an easy fix for that, but I need to think a bit.
I think people generally use zero sum to refer to zero sum (or constant sum) rewards e.g. one seat in congress or one minute of a viewer's attention. Even rock, paper, scissors would be negative sum if someone tried to disturb his opponent's sleep or spent a million dollars bribing the ref or fanatically practiced for a million games.