AI x-risk reduction: why I chose academia over industry

I've discussed this question with a good number of people, and I think I've generally found my pro-academia arguments to be stronger than their pro-industry arguments (I think probably many of them would agree?)

I... think we've discussed this? But I don't agree, at least insofar as the arguments are supposed to apply to me as well (so e.g. not the personal fit part).

Some potential disagreements:

1. I expect more field growth via doing good research that exposes more surface area for people to tackle, rather than mentoring people directly. Partly this is because people can get mentorship comes from generic PhD programs, and because a lot of my research aims to be conceptual clarification of the field as a whole. That second reason may not apply to you.
2. I wouldn't say "radical governance solutions" or "political activism" are "likely necessary" (though I wouldn't say they are "likely unnecessary" either, it just seems pretty uncertain).
3. I didn't notice you talking about early research being more impactful than later research, which seems like an important factor to the extent you think you'll do better + faster research in industry relative to academia (as I do believe).
4. You mention "all the usual reasons for preferring industry" -- I want to note that those seem like pretty strong reasons; idk how strong you think those are. (I'd also note salary in addition to the ones you mention -- even altruistically, you can donate a much larger amount on a typical industry salary.)

Personally, I find the "bully pulpit" argument for academia most persuasive.

Btw, planned summary for the Alignment Newsletter:

This post and its comments discuss considerations that impact whether new PhD graduates interested in reducing AI x-risk should work in academia or industry.

I think one thing to consider is that the two paths don't have an equal % chance to succeed. Getting a tenure track position at a top 20 university is hard. Really hard. Getting a research scientist position is, based on my very uncertain and informal understanding, less hard.

When you say academia looks like a clear win within 5-10 years, is that assuming "academia" means "starting a tenure-track job now?" If instead one is considering whether to begin a PhD program, for example, would you say that the clear win range is more like 10-15 years?

Also, how important is being at a top-20 institution? If the tenure track offer was instead from University of Nowhere, would you change your recommendation and say go to industry?

Would you agree that if the industry project you could work on is the one that will eventually build TAI (or be one of the leading builders, if there are multiple) then you have more influence from inside than from outside in academia?

My personal thought is:

      a.  So fundamental algorithms are cool and critical contributions to CS.  But where did we get the stuff we have now?  Well, arguably the things we have now are (1) absurdly powerful silicon devices (2) a large amount of open source and proprietary software.  Most of all of this was developed by industry or non-academic open source contributors.

     b.  What will it take to make AGI a reality?  The thing is, I think it is similar to comparing Werner Von Braun's work before he had a nation backing him, and after.  I think the most probable route to AGI is as follows: 

                  1.  Today we are trying to build practical systems that work as [sensor inputs] -> [intermediate state abstractions: example, collidable objects on  a grid around the agent] -> [goal state abstractions: example, predicted $ for each path the agent takes, with negative dollars for risks ].   They are fairly simple.

                  2.  I think AGI will essentially be "more meta".  There will be many more feeder subsystems that supply more complex intermediate states.  And more layers of meta states that ultimately result in high level abstractions like 'awareness' and 'self desires' and so on.  

To me all this looks like immense scale.  You need a gigantic software infrastructure that gets reused thousands of times over.  Modern example: wordpress.  You need a massive hardware infrastructure to host it and giga-dollar budgets.  

Also I think that many of the fundamental R&D steps - finding better activation functions, better neural network architectures, finding optimal configurations for a given problem, alternative algorithms - can be found better by massive autonomous systems that explore the possibility space.  I don't think human researchers will be able to contribute much directly.  Here's one paper as an example.

If you want to be a researcher who can exploit this you need to be a damn good programmer, and you need a big budget for cloud runs.  

    c.  Exponential progress is going to come not from throwing more humans at the problem, but by building clever software that bootstraps early progress in AI to make further progress.  Example, a neural network to generate potential functions for regression - which may not even be neural networks - to solve general regression problems.

What are your thoughts for subfields of ML where research impact/quality depends a lot on having lots of compute?

In NLP, many people have the view that almost all of the high impact work has come from industry over the past 3 years, and that the trend looks like it will continue indefinitely. Even safety-relevant work in NLP seems much easier to do with access to larger models with better capabilities (Debate/IDA are pretty hard to test without good language models). Thus, safety-minded NLP faculty might end up in a situation where none of their direct work is very impactful, but all of the expected impact is by graduating students who end up going to work in industry labs in particular.  How would you think about this kind of situation?

Maybe we need a "something else" category?   An alternative other than simply business/industry and academics?   

Also, while this is maybe something of an old topic, I took some notes regarding my thoughts on this topic and and related matters posted them to:

   https://mflb.com/ai_alignment_1/academic_or_industry_out.pdf