Scott Aaronson is joining OpenAI to work on AI safety

I think it might be super recent that mainstream academics are expressing this sentiment, probably because the harbingers are probably obvious, and yes this is probably not that hard to fund by certain EA causes. Or how much money does he actually want?

Hopefully. I have a feeling it won't be so easy, but we'll see.

In fact, starting a scientific field, as opposed to continuing, is poorly funded, it's not just AI risk. Another way to say this is that AI risk, as a scientific field, is pre-paradigmic.

Right, I was going to bring up the snowball effect as well but I forgot. I think that's a huge point.

I've also been perplexed by the focus on Tao in particular. In fact, I've long thought that if it's a good idea to recruit a top mathematician to alignment, then Peter Scholze would be a better choice since

1. he's probably the greatest active mathematician
2. he's built his career out of paradigmatizing pre-paradigmatic areas of math
3. he has an interest in computer proof-checking.

That said, I'm quite confident that Scholze is too busy revolutionizing everything he touches in mathematics to be interested in switching to alignment, so this is all moot.

(Also, I recognize that playing the "which one mathematician would be the single best to recruit to alignment?" game is not actually particularly useful, but it's been a pet peeve of mine for a while that Tao is the poster child of the push to recruit a mathematician, hence this comment.)

I've always used "Tao" to mean "brillant mathematicians" but I also think he has surprisingly eclectic research interests and in particular has done significant work in image processing, which shows a willingness to work on applied mathematics and may be relevant for AI work.

I must say however that I've changed my mind on this issue and that AI alignment research would be better served by hiring a shit ton of PhD students with a promise of giving 80% of them 3-5 years short term research positions after their PhD and giving 80% of those tenure afterward. I think we made a mistake by assuming that pre paradigmatic research means only genius are useful, and that a pure number strategy would help a lot (also genius mathematicians willing to work on interesting problems are not that much interested in money overwise they would work in finance, but they are very much interested by getting a fixed stable position before 35. The fact that France is still relevant in mathematical research while under paying its researcher by a lot is proof of that).

I always assumed that "Why don't we give Terence Tao a million dollars to work on AGI alignment?" was using Tao to refer to a class of people. Your comment implies that it would be especially valuable for Tao specifically to work on it. 

When I've talked about this, I've always meant both literally hire Tao and try to find young people of the same ability.  

While I too was using Tao as a reference class, it's not the only reason for mentioning him. I simply expect that people with IQs that ridiculously high are simply better suited to tackling novel topics, and I do mean novel, building a field from scratch, ideally with mathematical precision.

All the more if they have a proven track record, especially in mathematics, and I suspect that if Tao could be convinced to work on the problem, he would have genuinely significant insight. That and a cheerleader effect, which wouldn't be necessary in an ideal world, but that's hardly the one we live in is it?

Why should we believe that Tao would be especially likely to be able to make progress on AGI alignment (e.g. compared to other recent fields medal winners like Peter Scholze)?

Well, his name is alliterative, so there's that.

(I'm being glib here, but I agree that there's a much broader class of people who have a similar level of brilliance to Tao, but less name recognition, who could contribute quite a lot if they were to work on the problem.)