Estimating training compute of Deep Learning models

This is fantastic. Really appreciate both the detailed deep-dive in the document, and the summary here. This is also timely, given that teams working on superscale models with concerning capabilities haven't generally been too forthcoming with compute estimates. (There are exceptions.)

As you and Alex point out in the sibling thread, the biggest remaining fudge factors seem to be:

1. Mixture models (or any kind of parameter-sharing, really) for the first method, which will cause you to systematically overestimate the "Operations per forward pass" factor; and
2. Variable effective utilization rates of custom hardware for the second method, which will cause an unknown distribution of errors in the "utilization rate" factor.

Nonetheless, my flying guess would be that your method is pretty much guaranteed to be right within an OOM, and probably within a factor of 2 or less. That seems pretty good! It's certainly an improvement over anything I've seen previously along these lines. Congrats!

This seems pretty well done!  Some thoughts on future research in this direction:

• It seems like you probably could have gotten certainty about compute for at least a handful of the models studied in question (either because the model was open sourced, or you have direct access to the org training it like Eleuther) -- it would be interesting to see how the estimation methods compared to the exact answer in this case.  (Probably doable with GPT-J for example)
• While I agree with dropout not significantly reducing computation I think two more contemporary techniques are worth considering here: structured sparsity in weights ('blocksparse'), and mixture-of-experts gating ('switch transformer').  I think the latter is more important because it changes both the training compute and inference compute.
• Comparing custom ML hardware (e.g. Google's TPUs or Baidu's Kunlun, etc) is tricky to put on these sorts of comparisons.  For those I think the MLPerf Benchmarks are super useful.  I'd be curious to hear the authors' expectations of how this research changes in the face of more custom ML hardware.
• In general I think it'd be good to integrate a bunch of the performance benchmarks that are publicly available (since hardware providers are usually pretty eager to show off stats that make their hardware look good) into calibrations for this method.  It's also usually pretty straightforward to compute the operations and exact utilization in these runs, since they're heavily standardized on the exact model and dataset.