Co-found an incubator for independent AI Safety researchers (rolling applications)

post by Alexandra Bos (AlexandraB) · 2023-06-02T18:02:33.983Z · LW · GW · 13 comments


    Full-time, remote
  Why support independent AI Safety researchers?
  How will we help? 
  In what ways would this be impactful?
  Why you might want to found a non-profit
  About you
  About your co-founder
  Should you apply?
  Application process
    You can ask questions, register interest to potentially fund us, work with us, make use of our services in the future and share information 

Full-time, remote


Initial deadline: June 8th. Currently: rolling applications.

If your ideal job would be leading an impact-driven organization, being your own boss and pushing for a safer future with AI, you might be a great fit for co-founding Catalyze Impact!

Below, you will find out more about Catalyze’s mission and focus, why co-founding this org would be high-impact, how to tell if you’re a good fit, and how to apply.

In short, Catalyze will 1) help people become independent technical AI Safety researchers, and  2) deliver key support to independent AI Safety researchers so they can do their best work.



Would highly appreciate it if you could share this message with people who you think might potentially be interested in this role and upvote this post if you think more people should see it so we can find the best potential co-founder for this.

You can ask questions, register interest to potentially fund us, work with us, make use of our services in the future and share information here.

Why support independent AI Safety researchers?

Lots of people want to do AI Safety (AIS) research and are trying to get in a position where they can, yet only around 100-300 [LW · GW] people worldwide are actually doing research in this crucial area. Why? Because there are almost no AIS researcher jobs available due to AIS research organizations facing difficult constraints to scaling up. Luckily there is another way to grow the research field: having more people do independent research (where a self-employed individual gets a grant, usually from a fund). 

There is, however, a key problem: becoming and being a good independent AIS researcher is currently very difficult. It requires a lot of qualities which have nothing to do with being able to do good research: you have to be proactive, pragmatic, social, good enough at fundraising, very good at self-management and willing to take major career risks. Catalyze Impact will take away a large part of the difficulties that come with being an independent researcher, thereby making it a suitable option for more people so they are empowered to do good AIS research.

How will we help? 

This is the current design of the pilot - but you will help shape this further!

1. Fundraising support

       -> help promising individuals get funded to do research

2. Peer support networks & mentor-matching

       -> get feedback, receive mentorship, find collaborators, brainstorm and stay motivated rather than falling into isolation

3. Accountability and coaching 

       -> have structure, stay motivated and productive

4.  Fiscal sponsorship: hiring funded independent researchers as ‘employees’ 

       -> take away operational tasks which distract from research & help them build better career capital through institutional affiliation 


In what ways would this be impactful?

Alleviating a bottleneck for scaling the AIS research field by making independent research suitable for more people: it seems that we need a lot more people to be working on solving alignment. However, talented individuals who have invested in upskilling themselves to go do AIS research (e.g. SERI MATS graduates) are largely unable to secure research positions. This is oftentimes not because they are not capable enough of doing the research, but because there are simply too few positions available (see footnote). Because of this, many of these talented individuals are left with a few sub-optimal options: 

1) try to do research/a PhD in a different academic field in hopes that it will make them a better AIS researcher in the future

2) take a job working on AI capabilities (!)

3) try to become an independent AIS researcher

For many people, independent research (i.e. without this incubator) is not a good & viable option because being an independent researcher brings a lot of difficulties with it and arranging to be one requires specific skills. This drives these potential AIS researchers out of the field, delays or decreases their impact, and/or may even incentivize them to work on capabilities research instead - instead of contributing to AI Safety research.

Other ways in which this incubator could be impactful include:

• Increasing independent researchers’ productivity by offering them helpful services and centralizing certain operational tasks.

• Helping potential independent researchers get to work sooner by reducing the friction around fundraising.

• Increasing the number of research bets: additional independent research might increase the number of research directions being pursued. After all, as independent researchers individuals have more agency over deciding which research agendas to pursue. Pursuing more research bets could be very beneficial in this pre-paradigmatic field. 

Improving alignment research orgs’ applicant pool: independent researchers supported by us will arguably gain better research experience than they would through the alternative options they have. This could make alignment research organizations’ applicant pool more skilled, leading to better hires for them in the future.

Note: it seems unlikely that people will not apply for roles at/start research organizations because independent research becomes too appealing due to Catalyze’s help. However, we will keep an out for this to make sure we will not have this effect.


Why you might want to found a non-profit

About you

It’s a plus but not a prerequisite for you to have experience in (charity) entrepreneurship, working at a start-up, EA/AIS organization, or any other somewhat relevant working experience. Above all, you largely recognize yourself in the following description:


About your co-founder

Your co-founder would be Alexandra Bos, currently based in Amsterdam.

Should you apply?

A general piece of advice: when in doubt, always apply! Don’t let imposter syndrome get the better of you ;)

Candidates with all sorts of backgrounds are welcome to apply and the application should not take too much time.

Salary: dependent on your needs & fundraising outcomes.

Application process

To apply, please fill out this form. If you already have your CV ready, it should take around 10-25 mins to fill out. The second round will consist of an interview, followed by a third round (and possibly a fourth round) where we assess our fit for working together. The whole process should be finished around mid-June.

Application deadline: Thursday, June 8th (in your timezone) - but feel free to apply earlier, it will speed up the process.


You can ask questions, register interest to potentially fund us, work with us, make use of our services in the future and share information here.

Link to this post but in a Google Doc


Comments sorted by top scores.

comment by porby · 2023-05-27T17:39:43.501Z · LW(p) · GW(p)

This is a project I'd like to see succeed!

For what it's worth, I talked to Alexandra around EAG London a couple of times (I'm Ross, hi again!) and I think she has a good handle on important coordination problems. I encourage people to apply.

comment by Spencer Becker-Kahn · 2023-06-07T12:26:48.518Z · LW(p) · GW(p)

How exactly can an org like this help solve (what many people see as one of the main bottlenecks:) the issue of mentorship? How would Catalyze actually tip the scales when it comes to 'mentor matching'?

(e.g. see Richard Ngo's first high-level point in this career advice post [LW · GW])

comment by MSRayne · 2023-06-03T11:49:16.839Z · LW(p) · GW(p)

I've never had a job in my life - yes really, I've had a rather strange life so far, it's complicated - but I've been reading and thinking about topics which I now know are related to operations for years, trying to design (in my head...) a system for distributing the work of managing a complex organization across a totally decentralized group so that no one is in charge, with the aid of AI and a social media esque interface. (I've never actually made the thing, because I keep finding new things I need to know, and I'm not a software engineer, just a designer.)

So, I think I have some parts of the requisite skillset here, and a ton of intuition about how to run systems efficiently built up from all the independent studying I've done - but absolutely no prior experience with basically anything in reality, except happening to (I believe) have the right personality for operations work. Should I bother applying?

Replies from: AlexandraB
comment by Alexandra Bos (AlexandraB) · 2023-06-07T12:11:02.061Z · LW(p) · GW(p)

Hi, I'd encourage you to apply if you recognize yourself in the About you section!

When in doubt always apply is my motto personally

comment by jacquesthibs (jacques-thibodeau) · 2023-09-21T17:23:20.649Z · LW(p) · GW(p)

I’m curious to know if Catalyze Impact is moving forward, is on hold or if the project has been shut down.

Replies from: AlexandraB
comment by Alexandra Bos (AlexandraB) · 2023-09-22T12:47:56.335Z · LW(p) · GW(p)

Hi, thanks for asking! We're moving forward, got funding from Lightspeed, and plan to run our pilot in Q4 of this year.  You can subscribe at the bottom of if you want to make sure to stay in the loop about sign-ups and updates

comment by Evan R. Murphy · 2023-05-30T00:16:28.543Z · LW(p) · GW(p)

A couple of quick thoughts:

  • Very glad to see someone trying to provide more infrastructure and support for independent technical alignment researchers. Wishing you great success and looking forward to hearing how your project develops.
  • A lot of promising alignment research directions now seem to require access to cutting-edge models. A couple of ways you might deal with this could be:
    • Partner with AI labs to help get your researchers access to their models
    • Or focus on some of the few research directions such as mechanistic interpretability that still seem to be making useful progress on smaller, more accessible models
Replies from: AlexandraB, AlexandraB
comment by Alexandra Bos (AlexandraB) · 2023-06-05T11:10:18.295Z · LW(p) · GW(p)

I'd be curious to hear from the people who pressed the disagreement button on Evan's remark:  what part of this do you disagree with or not recognize?

Replies from: thomas-kwa
comment by Thomas Kwa (thomas-kwa) · 2023-06-05T11:22:42.266Z · LW(p) · GW(p)

I didn't hit disagree, but IMO there are way more than "few research directions" that can be accessed without cutting-edge models, especially with all the new open-source LLMs.

  • All conceptual work: agent foundations, mechanistic anomaly detection, etc.
  • Mechanistic interpretability, which when interpreted broadly could be 40% of empirical alignment work
  • Model control like the nascent area of activation additions [LW · GW]

I've heard that evals, debate, prosaic work into honesty, and various other schemes need cutting-edge models, but in the past few weeks transitioning from mostly conceptual work into empirical work, I have far more questions than I have time to answer using GPT-2 or AlphaStar sized models. If alignment is hard we'll want to understand the small models first.

Replies from: Evan R. Murphy, Evan R. Murphy
comment by Evan R. Murphy · 2023-06-05T20:53:04.007Z · LW(p) · GW(p)

I wasn't saying that there were only a few research directions that don't require frontier models period, just that there are only a few that don't require frontier models and still seem relevant/promising, at least assuming short timelines to AGI.

I am skeptical that agent foundations is still very promising or relevant in the present situation. I wouldn't want to shut down someone's research in this area if they were particularly passionate about it or considered themselves on the cusp of an important breakthrough. But I'm not sure it's wise to be spending scarce incubator resources to funnel new researchers into agent foundations research at this stage.

Good points about mechanistic anomaly detection and activation additions though! (And mechanistic interpretability, but I mentioned that in my previous comment.) I need to read up more on activation additions.

comment by Alexandra Bos (AlexandraB) · 2023-06-02T21:27:44.153Z · LW(p) · GW(p)

I was thinking about helping with infrastructure around access to large amounts of compute but had not considered trying to help with access to cutting-edge models but I think it might be a very good suggestion. Thanks for sharing your thoughts! 

comment by Martin Vlach (martin-vlach) · 2023-06-09T18:00:10.862Z · LW(p) · GW(p)

The website seems good, but the buttons on the 'sharing' circle on the bottom need fixing.

comment by Dennis Akar (British_Potato) · 2023-06-04T12:14:27.168Z · LW(p) · GW(p)

Yay it's back up again.