A compressed take on recent disagreements

executing the same basic algorithm (genetic differences are tiny compared to size of genome).

This seems moderately misleading. People start out nearly the same, but apply their algorithm to somewhat different domains. Running a year of human-level compute on different data should be expected to produce much more divergent results than is captured by the genetic differences.

Specialization on different topics likely explains much more than algorithmic tweaks explain.

There are other reasons why top mathematicians could have better output compared to average mathematicians. They could be working on more salient problems, there's selection bias in who we call a "top mathematician", they could be situated in an intellectual microcosm more suitable for mathematical progress, etc.

Also, there's a question of how directly you can apply optimization pressure towards a given goal. When you select the best of n mathematicians by highest long run net output, it's true that you're only applying log(n) bits of optimization pressure towards high long run net output, and so the fact that this results in a lot of improvement does show that long run net output is, in some sense, "easy" to optimize for. However, those log(n) bits of optimization pressure are being directly applied towards that goal, and it's not easy to have a learning process that applies optimization pressure in a similarly direct manner (as opposed to optimizing for something like "ability to do well on this math problem dataset"). 

E.g., optimizing a GPT model for predictive loss does not directly optimize it for coherent text, as demonstrated by the fact that sampling from a GPT model can get you much more coherent text with only a handful of samples. It still cost millions of dollars to turn GPT-2 into GPT-3.

An analogy: it's clear that, if you take the "most aligned"^[1] out of n humans, you get a human that's much more "aligned" than the average person. Does this mean it's also "easy" to optimize for alignment?^[2]

Finally, AI Impacts has done a number of investigations into how long it took for AI systems to go from ~human level to better than human level in different domains. E.g., it took 10 years for diagnosis of diabetic retinopathy. I think this line of research is more directly informative on this question.

1. ^{^}

   I'm leaving the operationalization of "most aligned" deliberately vague. Most likely, humans will vary significantly along whatever axis you choose. 

2. ^{^}

   I actually think it does mean optimizing for alignment is "easy", in the sense that a few bits of optimization pressure directed purely towards alignment will go a long way in improving alignment. Again, that doesn't mean it's easy to actually direct optimization pressure in such a manner. (Though directing optimization pressure towards capabilities is probably easier than directing it towards alignment.)

I think the "factor of a thousand" here is mostly mathematics itself being a very unusual field. If you can reach even a tiny bit further in some "concept-space" than others, for whatever reasons internal or external, then you can publish everything you find in that margin and it will all be new. If you can't, then you pretty much have to work on the diminishing unpublished corners of already reached concept-space and most will look derivative.

I would certainly expect AI to blow rapidly past human mathematicians at some point due to surpassing human "reach". Whether that would also enable breakthroughs in other various sciences that rely on mathematics remains to be seen. Advances in theoretical physics may well need new abstract mathematical insights. Technology and engineering probably does not.