Probably good projects for the AI safety ecosystem

1. I'm skeptical of the usefulness of such an organization. I think we have a plethora of motivated talent passing through AGISF currently that doesn't need another short workshop or a bunch of low-context 1-1s with researchers who probably have better-vetted people to spend their scarce time on (I think vetting is very hard). I think the AI safety talent development pipeline needs less centralized, short-duration, broad-funnel workshops and more programs that allow for longer-term talent incubation into specific niches (e.g., programs/low-stakes jobs that build critical skills in research vs. management vs. operations) that don't eat up valuable researcher time unnecessarily and encourage decentralization of alignment research hubs. Sorry if this sounds like bad-faith criticism; it's not intended to be.
2. Seems like a great idea. I think I'd strictly prefer "a combined research mentorship and seminar program that aims to do for AI governance research what MATS is trying to do for technical AI alignment research," because it feels like the optimal program for Cotra or Kokotajlo is a bit different from MATS and likely includes other great governance/worldview/macrostrategy researchers. However, I'll probably talk to Cotra and Kokotajlo to see if we can add value to them!
3. I think this is rather the domain of existing academic and industry research groups (e.g., Krueger's lab, Anthropic, CHAI, OpenAI safety team, DeepMind safety team, etc.), as these groups have the necessary talent and, presumably, motivation. I'd also be excited for MATS scholars and alumni and LTFF-funded independent researchers to work on this.
4. Seems good, if hard (and not something I'd expect a competition to help with, on priors, if the most capable/aligned people already are not working on this). In particular, I'd be excited to see something that discusses, from first principles, whether solving alignment is significantly harder than similar scientific problems (credit to Caleb [LW · GW] for often talking about this).
5. I'm skeptical that this would be low-risk (in regards to making researchers more skeptical of alignment and less likely to listen to AI safety spokespeople at a critical date) or a counterfactually valuable use of senior AI safety researcher time.

As far as I can tell, there are no MIRI-level AI safety research teams here in the Sinosphere. 

How about individual researchers? Is there anyone prominently associated with "risks of AI takeover" and similar topics? edit: Or even associated with "benefits of AI takeover"!

I would be all for that, personally! Can’t speak for the broader community though, so recommend asking around :)

Currently, MATS finds it hard to bring Chinese scholars over to the US for our independent research/educational seminar program because of US visa restrictions on Chinese citizens. I think there is probably significant interest in a MATS-like program among Chinese residents and certainly lots of Chinese ML talent. I'm generally concerned by approaches to leverage ML talent for AI safety that don't select hard for value alignment with the longtermist/AI alignment project as this could accidentally develop ML talent that furthers AI capabilities. That said, if participant vetting is good enough and there are clear pathways for alumni to contribute to AI safety, I'm very excited about programs based in China or that focus on recruiting Chinese talent. I think the projects that I'm most excited about in this space would be MATS-like, ARENA-like, or CAIS-like. If you have further ideas, let me know!

I don't have insider information, but I think that Aligned AI, Anthropic, ARC, CLR, Conjecture, DeepMind, Encultured AI, FAR AI, MIRI, OpenAI, and Redwood Research (not an all-inclusive list) could all probably offer visas to employees. The MATS Program currently assists scholars in obtaining US B-1 visas or ESTAs and UK Standard Visitor visas. Are you asking whether there should be an organization that aims to hire people to work long-term on AI safety niches that these organizations do not fill, and if so, which niches?

Currently, it seems that the AI Safety Unconference [EA · GW] and maybe the Stanford Existential Risks Conference are filling this niche. I think neither of these conferences is optimizing entirely for the AI safety researcher experience; the former might aim to appeal to NeurIPS attendees outside the field of AI safety, and the latter includes speakers on a variety of x-risks. I think a dedicated AI safety conference/unconference detached from existing ML conferences might be valuable. I also think boosting the presence of AI safety at NeurIPS or other ML conferences might be good, but I haven't thought deeply about this.

1. Currently, MATS is somewhat supporting rolling admissions for a minority of mentors with our Autumn and Spring cohorts (which are generally extensions of our Summer and Winter cohorts, respectively). Given that MATS is mainly focused on optimizing the cohort experience for scholars (because we think starting a research project in an academic cohort of people with similar experience with targeted seminars and workshops is ideal), we are probably a worse experience for scholars or mentors who ideally would start research projects at irregular intervals. Some scholars might not benefit as much from the academic cohort experience as others. Some mentors might ideally commit to mentorship during times of the year outside of MATS' primary Winter/Summer cohorts. Also, MATS' seminar program doesn't necessarily run year-round, and we don't offer as much logistical support to scholars outside of Winter/Summer. There is definitely free energy here for a complementary program, I think.
2. I am also scared about ML upskilling bootcamps that act as feeder grounds for AI capabilities organizations. I think vetting (including perhaps AGISF prerequisite) is key, as is a clear understanding of where the participants will go next. I only recommend this kind of project because hundreds of people seemingly complete AGISF and want to upskill to work on AI alignment but have scant opportunities. Also, MATS' theory of change includes adding value through accelerating the development of (rare) "research leads [EA · GW]" to increase the "carrying capacity" of the alignment research ecosystem (which theoretically is not principally bottlenecked by "research supporter" talent because training/buying such talent scales much easier than training/buying "research lead" talent). I will publish my reasoning for the latter point as soon as I have time.
3. Probably ask Sam Bowman [LW · GW]. At a minimum, it might consist of an office space for longtermist organizations, like Lightcone or Constellation in Berkeley, some operations staff to make the office run, and some AI safety outreach to NYU and other strong universities nearby, like Columbia. I think some people might already be working on this?

I don't think I'd change it, but my priorities have shifted [LW(p) · GW(p)]. Also, many of the projects I suggested now exist, as indicated in my comments!

IAPS AI Policy Fellowship also exists now!

SPAR exists! Though I don't think it can offer visas.