Alignment Grantmaking is Funding-Limited Right Now

This matches my impression. FAR could definitely use more funding. Although I'd still at the margin rather hire someone above our bar than e.g. have them earn-to-give and donate to us, the math is getting a lot closer than it used to be, to the point where those with excellent earning potential and limited fit for AI safety might well have more impact pursuing a philanthropic pathway.

I'd also highlight there's a serious lack of diversity in funding. As others in the thread have mentioned, the majority of people's funding comes (directly or indirectly) from OpenPhil. I think OpenPhil does a good job trying to mitigate this (e.g. being careful about power dynamics, giving organizations exit grants if they do decide to stop funding an org, etc) it's ultimately not a healthy dynamic, and OpenPhil appears to be quite capacity constrained in terms of grant evaluation. So, the entry of new funders would help diversify this in addition to increasing total capacity.

One thing I don't see people talk about as much but also seems like a key part of the solution: how can alignment orgs and researchers make more efficient use of existing funding? Spending that was appropriate a year or two ago when funding was plentiful may not be justified any longer, so there's a need to explicitly put in place appropriate budgets and spending controls. There's a fair amount of cost-saving measures I could see the ecosystem implementing that would have limited if any hit on productivity: for example, improved cash management (investing in government money market funds earning ~5% rather than 0% interest checking accounts); negotiating harder with vendors (often possible to get substantial discounts on things like cloud compute or commercial real-estate); and cutting back on some fringe benefits (e.g. more/higher-density open plan rather than private offices). I'm not trying to point fingers here: I've made missteps here as well, for example FAR's cash management currently has significant room for improvement -- we're in the process of fixing this and plan to share a write-up of what we found with other orgs in the next month.

(Speaking just for the Long-Term Future Fund)

It’s true that the Long-Term Future Fund could use funding right now. We’re working on a bunch of posts, including an explanation of our funding needs, track record over the last year, and some reflections that I hope will be out pretty soon.

I’d probably wait for us to post those if you’re a prospective LTFF donor, as they also have a bunch of relevant updates about the fund.

I'm told that a few professors in AI safety are getting approached by high net worth individuals now but don't have a good way to spend their money. Seems like there are connections to be made.

From what I can tell, the field have been funding constrained since the FTX collapse.

What I think happened: 
FTX had lots of money and a low bar for funding, which meant they spread a lot of money around. This meant that more project got started, and probably even more people got generally encouraged to join. Probably some project got funded that should not have been, but probably also some really good projects got started that did not get money before because not clearing the bar before due to not having the right connections, or just bad att writing grant proposals. In short FTX money and the promise of FTX money made the field grow quickly. Also there where where also some normal field growth. AIS has been growing steadily for a while. 

Then FTX imploded. There where lots of chaos. Grants where promised but never paid out. Some orgs don't what to spend the money they did get from FTX because of risk of clawback risks. Other grant makers cover some of this but not all of this. It's still unclear what the new funding situation is.

Some months later, SFF, FTX and Nonlinear Network have their various grant rounds. Each of them get overwhelmed with applications. I think this is mainly from the FTX induced growth spurt, but also partly orgs still trying to recover from loss of FTX money, and just regular growth. Either way, the outcome of these grant rounds make it clear that the funding situation has changed. The bar for getting funding is higher than before. 

Plug: CAIS is funding constrained.

Thanks for posting this, this seems very correct. 

Plug: I recently published a long post on the EA Forum on AI safety funding: An Overview of the AI Safety Funding Situation [EA · GW].

I also have this impression, except it seems to me that it's been like this for several months at least. 

The Open Philanthropy people I asked at EAG said they think the bottleneck is that they currently don't have enough qualified AI Safety grantmakers to hand out money fast enough. And right now, the bulk of almost everyone's funding seems to ultimately come from Open Philanthropy, directly or indirectly.

This matches my impression. At EAG London I was really stunned (and heartened!) at how many skilled people are pivoting into interpretability from non-alignment fields.

What about public funding? A lot of people are talking to politicians, but requesting more funding doesn't seem to be a central concern - I've heard more calls for regulation than calls for public AI x-risk funding.

↑ comment by johnswentworth · 2023-07-22T19:17:01.427Z · LW(p) · GW(p)

Given a choice between public funding and just not having funding, I'd take a lack of funding. The incentives/selection pressures which come with public funding, especially in a field with already-terrible feedback loops, spell doom for the field.

Replies from: derpherpize, alexander-gietelink-oldenziel

↑ comment by Lao Mein (derpherpize) · 2023-07-25T06:06:41.466Z · LW(p) · GW(p)

What do you think happens in a world where there is $100 billion in yearly alignment funding? How would they be making less progress? I want to note that even horrifically inefficient systems still produce more output than "uncorrupted" hobbyists - cancer research would produce much fewer results if it were done by 300 perfectly coordinated people, even if the 300 had zero ethical/legal restraints. 

Replies from: johnswentworth

↑ comment by johnswentworth · 2023-07-26T17:14:54.015Z · LW(p) · GW(p)

Let's take cancer as an analogy for a moment. Suppose that, as a baseline, cancer research is basically-similar to other areas of medical research. Then, some politician comes along and declares "war on cancer", and blindly pumps money into cancer research specifically. What happens? Well...

So even just from eyeballing that chart, it's pretty plausible to me that if cancer funding dropped by a factor of 10, the net effect would be that clinical trial pass rates just return to comparable levels to other areas, and the actual benefits of all that research remain roughly-the-same.

... but that's ignoring second-order effects.

Technical research fields have a "median researcher" problem: the memetic success of work in the field is not determined by the best researchers, but by the median researchers. Even if e.g. the best psychologists understand enough statistics to recognize crap studies, the median psychologist doesn't (or at least didn't 10 years ago), so we ended up with a field full of highly-memetically-successful crap studies which did not replicate (think Carol Dweck).

Back to cancer: if the large majority of the field is doing work which is predictably useless, then the field will develop standards for "success" which are totally decoupled from actual usefulness. (Note that the chart above doesn't actually imply that most of the work done in the field is useless, let alone predictably useless; there's not a one-to-one map between cancer research projects and clinical trials.) To a large extent, the new standards would be directly opposed to actual usefulness, in order to defend the entrenched researchers doing crap work - think Carol Dweck arguing that replication is a bad standard for psychology.

That's the sort of thing I expect would happen if a government dumped $100B into alignment funding. There'd be a flood of people with nominally-alignment-related projects which are in fact basically useless for solving alignment; they would quickly balloon to 90+% of the field. With such people completely dominating the field, first memetic success and then grant money would mostly be apportioned by people whose standards for success are completely decoupled from actual usefulness for alignment. Insofar as anything useful got done, it would mostly be by people who figured out the real challenges of alignment for themselves, and had to basically hack the funding system in order to get money for their actually-useful work.

In the case of cancer, steady progress has been made over the years despite the mess; at the end of the day, clinical trials provide a good ground-truth signal for progress on cancer. Even if lots of shit is thrown at the wall, some of it sticks, and that's useful. In alignment, one of the main frames for why the problem is hard is that we do not have a good ground-truth signal for whether we're making progress. So all these problems would be much worse than usual, and it's less likely to be the actually-useful shit which sticks to the metaphorical wall.

Replies from: alexander-gietelink-oldenziel, stephen-mcaleese

↑ comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) · 2023-07-26T17:33:12.592Z · LW(p) · GW(p)

Many things here. 

the issues you mention don't seem tied to public versus private funding but more about size of funding + an intrinsically difficul scientific question.  I agree that at some point more funding doesn't help. At the moment, that doesn't seem to be the case in alignment. Indeed, alignment is not even as large in number of researchers as a relatively small field like linguistics. 

Replies from: johnswentworth

↑ comment by johnswentworth · 2023-07-26T18:15:40.315Z · LW(p) · GW(p)

How well the funders understand the field, and can differentially target more-useful projects, is a key variable here. For public funding, the top-level decision maker is a politician; they will in the vast majority of cases have approximately-zero understanding themselves. They will either apportion funding on purely political grounds (e.g. pork-barrel spending), or defer to whoever the consensus "experts" are in the field (which is where the median researcher problem kicks in).

In alignment to date, the funders have generally been people who understand the problem themselves to at least enough extent to notice that it's worth paying attention to (in a world where alignment concern wasn't already mainstream), and can therefore differentially target useful work, rather than blindly spray money around.

Replies from: alexander-gietelink-oldenziel

↑ comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) · 2023-07-26T19:57:09.584Z · LW(p) · GW(p)

Seems overstated. Universities support all kinds of very specialized long-term research that politicians don't understand. 

From my own observations and from talking with funders themselves most funding decisions in AI safety are made on mostly superficial markers - grantmakers on the whole don't dive deep on technical details. [In fact, I would argue that blindly spraying around money in a more egalitarian way  (i.e. what SeriMATS has accomplished) is probably not much worse than the status-quo.] 

Academia isn't perfect but on the whole it gives a lot of bright people the time, space and financial flexibility to pursue their own judgement. In fact, many alignment researchers have done a significant part of work in an academic setting or being supported in some ways by public funding. 

↑ comment by Stephen McAleese (stephen-mcaleese) · 2023-07-28T14:42:02.375Z · LW(p) · GW(p)

At first, I predicted you were going to say that public funding would accelerate capabilities research over alignment but it seems like the gist of your argument is that lots of public funding would muddy the water and sharply reduce the average quality of alignment research.

That might be true for theoretical AI alignment research but I'd imagine it's less of a problem for types of AI alignment research that have decent feedback loops like interpretability research and other kinds of empirical research like experiments on RL agents.

One reason that I'm skeptical is that there doesn't seem to be a similar problem in the field of ML which is huge and largely publicly funded to the best of my knowledge and still makes good progress. Possible reasons why the ML field is still effective despite its size include sufficient empirical feedback loops and the fact that top conferences reject most papers (~25% is a typical acceptance rate for papers at NeurIPS).

Replies from: johnswentworth

↑ comment by johnswentworth · 2023-07-28T16:02:27.214Z · LW(p) · GW(p)

Yeah, to be clear, acceleration of capabilities is a major reason why I expect public funding would be net negative, rather than just much closer to zero impact than naive multiplication would suggest. 

Ignoring the capabilities issue, I think there's lots of room for uncertainty about whether a big injection of "blind funding" would be net positive, for the reasons explained above. I think we should be pretty confident that the results would be an OOM or more less positive than the naive multiplication suggests, but that's still not the same as "net negative"; the net positivity/negativity I see as much more uncertain (ignoring capabilities impact).

Accounting for capabilities impact, I think the net impact would be pretty robustly negative.

That might be true for theoretical AI alignment research but I'd imagine it's less of a problem for types of AI alignment research that have decent feedback loops like interpretability research and other kinds of empirical research like experiments on RL agents.

The parts where the bad feedback loops are, are exactly the places where the things-which-might-actually-kill-us are [LW · GW]. Things we can see coming are exactly the things which don't particularly need research to stop, and the fact that we can see them is exactly what makes the feedback loops good. It is not an accident that the feedback loop problem is unusually severe for the field of alignment in particular.

(Which is not to say that e.g. interpretability research isn't useful - we can often get great feedback loops on things which provide a useful foundation for the hard parts later on. The point is that, if the field as a whole streetlights on things with good feedback loops, it will end up ignoring the most dangerous things.)

↑ comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) · 2023-07-26T16:47:29.272Z · LW(p) · GW(p)

This seems implausible. Almost all contributions to AI alignment (from any perspective) has been through by people having implicitly or explicitly outside funding - not by hobbyist doing alignment next to their dayjob.

Replies from: johnswentworth

↑ comment by johnswentworth · 2023-07-26T16:49:36.959Z · LW(p) · GW(p)

I am not claiming that a hobbyist-only research community would outperform today's community. I'm claiming that a hobbyist-only research community would outperform public (i.e. government) funding. Today's situation is better than either of those two: we have funding which provides incentives which, for all their flaws, are far far better than the incentives which would come with government funding.

Replies from: alexander-gietelink-oldenziel

↑ comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) · 2023-07-26T17:27:33.806Z · LW(p) · GW(p)

Roll-to-disbelieve. Can you name one kind of research that wouldn't have counterfactually happened if alignment was publicly funded? Your own research seems like a good fit for academia for instance. 

Replies from: johnswentworth

↑ comment by johnswentworth · 2023-09-02T16:16:51.518Z · LW(p) · GW(p)

Can you name one kind of research that wouldn't have counterfactually happened if alignment was publicly funded?

Wrong question.

The parable of the leprechaun is relevant here:

One day, a farmer managed to catch a leprechaun. As is usual for these tales, the leprechaun offered to show the farmer where the leprechaun's gold was buried, in exchange for the leprechaun's freedom. The farmer agreed. So the leprechaun led the farmer deep into the woods, eventually stopped at a tree, and said "my gold is buried under this tree".

Unfortunately, the farmer had not thought to bring a shovel. So, the farmer tied a ribbon around the tree to mark the spot, and the leprechaun agreed not to remove it. Then the farmer returned home to fetch a shovel.

When the farmer returned, he found a ribbon tied around every tree in the forest. He never did find the gold.

This is the problem which plagues many academic fields (e.g. pre-crisis psychology is a now-clear example). It's not mainly that good research goes unfunded, it's that there's so much crap that the good work is (a) hard to find, and (b) not differentially memetically successful.

A little crap research mostly doesn't matter, so long as the competent researchers can still do their thing. But if the volume of crap reaches the point where competent researchers have trouble finding each other, or new people are mostly onboarded into crap research, or external decision-makers can't defer to a random "expert" in the field without usually getting a bunch of crap, then all that crap research has important negative effects.

Replies from: alexander-gietelink-oldenziel, whitehatStoic

↑ comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) · 2023-09-02T22:46:04.433Z · LW(p) · GW(p)

It's a cute story John but do you have more than an anecdotal leprechaun?

I think the simplest model (so the one we should default to by Occam's mighty Razor) is that whether good research will be done in a field is mostly tied to 

• intrinisic features of research in this area (i.e. how much feedback from reality, noisy vs nonnoisy, political implication, and lots more I don't care to name) 
• initial fieldbuilding driving who self-selects into the research field
• Number of Secure funded research positions

the first is independent of funding source  - I don't think we have much evidence that the second would be much worse for public funding as opposed to private funding.  

in absence of strong evidence, I humbly suggest we should default to the simplest model in which :

• more money & more secure positions -> more people will be working on the problem

The fact that France has a significant larger number of effectively-tenured positions per capita than most other nations, entirely publicly funded, is almost surely one of the most important factors in its (continued) dominance in pure mathematics as evidenced by its large share of Fields medals (13/66 versus 15/66 for the US). I observe in passing that your own research program is far more akin to academic math than cancer research. 

As for the position that you'd rather have no funding as opposed to public funding is ... well let us be polite and call it ... American. 

Replies from: johnswentworth, quetzal_rainbow

↑ comment by johnswentworth · 2023-09-03T17:07:30.060Z · LW(p) · GW(p)

(Probably not going to respond further here, but I wanted to note that this comment really hit the perfect amount of sarcasm and combativity for me personally; I enjoyed it.)

↑ comment by quetzal_rainbow · 2023-09-03T16:27:25.942Z · LW(p) · GW(p)

Alignment is almost exactly opposite of abstract math? Math has a good quality of being checkable - you can get a paper, follow all its content and become sure that content is valid. Alignment research paper can have valid math, but be inadequate in questions such as "is this math even related to reality?", which are much harder to check.

Replies from: alexander-gietelink-oldenziel, sharmake-farah

↑ comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) · 2023-09-03T17:41:38.426Z · LW(p) · GW(p)

That may be so.

• Wentworths own work is closest to academic math/theoretical physics, perhaps to philosophy.

• are you claiming we have no way of telling good (alignment) research from bad? And if we do, why would private funding be better at figuring this out than public funding?

↑ comment by Noosphere89 (sharmake-farah) · 2023-09-03T17:20:36.885Z · LW(p) · GW(p)

To be somewhat more fair, there are probably thousands of problems with the property that they are much easier to check than they are to solve, and while alignment research is maybe not this, I do think that there's a general gap between verifying a solution and actually solving the problem.

Replies from: alexander-gietelink-oldenziel

↑ comment by Alexander Gietelink Oldenziel (alexander-gietelink-oldenziel) · 2023-09-03T20:20:17.189Z · LW(p) · GW(p)

The canonical examples are NP problems.

Another interesting class are problems that are easy to generate but hard to verify.

John Wentworth told me the following delightfully simple example Generating a Turing machine program that halts is easy, verifying that an arbitrary TM program halts is undecidable.

Replies from: sharmake-farah

↑ comment by Noosphere89 (sharmake-farah) · 2023-09-03T22:03:04.690Z · LW(p) · GW(p)

Yep, I was thinking about NP problems, though #P problems for the counting version would count as well.

↑ comment by MiguelDev (whitehatStoic) · 2023-09-03T07:14:56.489Z · LW(p) · GW(p)

But if the volume of crap reaches the point where competent researchers have trouble finding each other, or new people are mostly onboarded into crap research, or external decision-makers can't defer to a random "expert" in the field without usually getting a bunch of crap, then all that crap research has important negative effects

I agree with your observation. The problem is that many people are easily influenced by others, rather than critically evaluating whether the project they're participating in is legitimate or if their work is safe to publish. It seems that most have lost the ability to listen to their own judgment and assess what is rational to do.

This would help explain some things I've been scratching my head about (revealed preferences implied by visible funding, assuming ample cash on hand, didn't seem to match well with a talent-constrained story). I'm currently trying to figure out how to prioritize earning to give versus direct research; it would be really helpful to know more about the funding situation from the perspective of the grantmaking organizations.

In particular, it'd be great to have some best-guess projections of future funding distributions, and any other information that might help fill in different values of X, Y and Z for propositions like "We would prefer X dollars in extra funding received in year Y over Z additional weekly hours of researcher time starting today."

committing to an earning-to-give path would be a bet on this situation being the new normal.

Is that true, or would it just be the much more reasonable bet that this situation ever occurs again? because at least in theory, a dedicated earning-to-give person could just invest money during the funding-flush times and then donate it the next time we're funding-constrained?

This lines up with our experience running our first Nonlinear Network [EA · GW] round. We received ~500 applications, most of which did not apply to other EA funders like the LTFF. Full writeup coming soon.

If you'd like to help out, you can apply to be a funder for our next round in Oct/Nov. 

It's really good to see this said out loud. I don't necessarily have a broad overview of the funding field, just my experiences of trying to get into it - both into established orgs, or trying to get funding for individual research, or for alignment-adjacent stuff - and ending up in a capabilities research company.

I wonder if this is simply the result of the generally bad SWE/CS market right now. People who would otherwise be in big tech/other AI stuff, will be more inclined to do something with alignment. Similarly, if there's less money in overall tech (maybe outside of LLM-based scams), there may be less money for alignment.

Today I received a rejection email from Oliver Hybryka on behalf of Lightspeed Grants. It says Lightspeed Grants received 600 applicants, ~$150M in default funding requests, and ~$350M in maximum funding requests. Since the original amount to be distributed was $5M, only ~3% of applications could be funded.

I knew they received more applicants than they could fund but I'm surprised by how much was requested and how large the gap was.

Thank you for sharing. It's great to have a better picture of how complex this is, the non-alignment related issues that we have to solve as new researchers on top of trying to help solve the alignment problem. I think for my case, not seeking funding for the moment is the best option - as it seems to me that dwelling on the funding situation without a clear path is a waste of precious time that I can instead focus to extracting neural activity and doing model interpretability. I think its beneficial for my limited brain space/ cognitive capacity, just think of my research as a volunteer project for the next 6 months (or more).

I created a Guesstimate model that estimates that $2-3 million (range: $25k - $25 million) in high-quality grants could have been requested for the Lightspeed grant ($5 million was available).

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?