MichaelDickens's Shortform

Adding my two cents as someone who has a pretty different lens from Habryka but has still been fairly disappointed with OpenPhil, especially in the policy domain. 

Relative to Habryka, I am generally more OK with people "playing politics". I think it's probably good for AI safely folks to exhibit socially-common levels of "playing the game"– networking, finding common ground, avoiding offending other people, etc. I think some people in the rationalist sphere have a very strong aversion to some things in this genre, and labels like "power-seeking" and "deceptive" get thrown around too liberally. I also think I'm pretty with OpenPhil deciding it doesn't want to fund certain parts of the rationalist ecosystem (and probably less bothered than Habryka about how their comms around this wasn't direct/clear).

In that sense, I don't penalize OP much for trying to "play politics" or for breaking deontological norms. Nonetheless, I still feel pretty disappointed with them, particularly for their impact on comms/policy. Some thoughts here:

• I agree with Habryka that it is quite bad that OP is not willing to fund right-coded things. Even many of the "bipartisan" things funded by OP are quite left-coded. (As a useful heuristic, whenever you hear of someone launching a bipartisan initiative, I think one should ask "what % of the staff of this organization is Republican?" Obviously just a heuristic– there are some cases in which a 90%-Dem staff can actually truly engage in "real" bipartisan efforts. But in some cases, you will have a 90%-Dem staff claiming to be interested in bipartisan work without any real interest in Republican ideas, few if any Republican contacts, and only a cursory understanding of Republican stances.)
• I also agree with Habryka that OP seems overly focused on PR risks and not doing things that are weird/controversial. "To be a longtermist grantee these days you have to be the kind of person that OP thinks is not and will not be a PR risk, IE will not say weird or controversial stuff" sounds pretty accurate to me. OP cannot publicly admit this because this would be bad for its reputation– instead, it operates more subtly. 
• Separately, I have seen OpenPhil attempt to block or restrain multiple efforts in which people were trying to straightforwardly explain AI risks to policymakers. My understanding is that OpenPhil would say that they believed the messengers weren't the right people (e.g., too inexperienced), and they thought the downside risks were too high. In practice, there are some real tradeoffs here: there are often people who seem to have strong models of AGI risk but little/no policy experience, and sometimes people who have extensive policy experience but only recently started engaging with AI/AGI issues. With that in mind, I think OpenPhil has systematically made poor tradeoffs here and failed to invest into (or in some cases, actively blocked) people who were willing to be explicit about AGI risks, loss of control risks, capability progress, and the need for regulation. (I also think the "actively blocking" thing has gotten less severe over time, perhaps in part because OpenPhil changed its mind a bit on the value of direct advocacy or perhaps because OpenPhil just decided to focus its efforts on things like research and advocacy projects found funding elsewhere.)
• I think OpenPhil has an intellectual monoculture and puts explicit/implicit cultural pressure on people in the OP orbit to "stay in line." There is a lot of talk about valuing people who can think for themselves, but I think the groupthink problems are pretty real. There is a strong emphasis on "checking-in" with people before saying/doing things, and the OP bubble is generally much more willing to criticize action than inaction. I suspect that something like the CAIS statement or even a lot of the early Bengio comms would not have occured if Dan Hendrycks or Yoshua were deeply ingrained in the OP orbit. It is both the case that they would've been forced to write 10+ page Google Docs defending their theories of change and the case that the intellectual culture simply wouldn't have fostered this kind of thinking. 
• I think the focus on evals/RSPs can largely be explained by a bias toward trusting labs. OpenPhil steered a lot of talent toward the evals/RSPs theory of change (specifically, if I recall correctly, OpenPhil leadership on AI was especially influential in steering a lot of the ecosystem to support and invest in the evals/RSPs theory of change.) I expect that when we look back in a few years, there will be a pretty strong feeling that this was the wrong call & that this should've been more apparent even without the benefit of hindsight.
• I would be more sympathetic to OpenPhil in a world where their aversion to weirdness/PR risks resulted in them having a strong reputation, a lot of political capital, and real-world influence that matched the financial resources they possess. Sadly, I think we're in a "lose-lose" world: OpenPhil's reputation tends to be poor in many policy/journalism circles even while OpenPhil pursues a strategy that seems to be largely focused on avoiding PR risks. I think some of this is unjustified (e.g., a result of propaganda campaigns designed to paint anyone who cares about AI risk as awful). But then some of it actually is kind of reasonable (e.g., impartial observers viewing OpenPhil as kind of shady, not direct in its communications, not very willing to engage directly or openly with policymakers or journalists, having lots of conflicts of interests, trying to underplay the extent to which its funding priorities are influenced/constrained by a single Billionaire, being pretty left-coded, etc.)

To defend OpenPhil a bit, I do think it's quite hard to navigate trade-offs and I think sometimes people don't seem to recognize these tradeoffs. In AI policy, I think the biggest tradeoff is something like "lots of people who have engaged with technical AGI arguments and AGI threat models don't have policy experience, and lots of people who have policy experience don't have technical expertise or experience engaging with AGI threat models" (this is a bit of an oversimplification– there are some shining stars who have both.) 

I also think OpenPhil folks probably tend to have a different probability distribution over threat models (compared to me and probably also Habryka). For instance, it seems likely to me that OpenPhil employees operate in more of a "there are a lot of ways AGI could play out and a lot of uncertainty– we just need smart people thinking seriously about the problem. And who really know how hard alignment will be, maybe Anthropic will just figure it out" lens and less of a "ASI is coming and our priority needs to be making sure humanity understands the dangers associated with a reckless race toward ASI, and there's a substantial chance that we are seriously not on track to solve the necessary safety and security challenges unless we fundamentally reorient our whole approach" lens. 

And finally, I think despite these criticisms, OpenPhil is also responsible for some important wins (e.g., building the field, raising awareness about AGI risk on university campuses, funding some people early on before AI safety was a "big deal", jumpstarting the careers of some leaders in the policy space [especially in the UK]. It's also plausible to me that there are some cases in which OpenPhil gatekeeping was actually quite useful in preventing people from causing harm, even though I probably disagree with OpenPhil about the # and magnitude of these cases). 

This should be a top-level post.

It feels sorta understandable to me (albeit frustrating) that OpenPhil faces these assorted political constraints.  In my view this seems to create a big unfilled niche in the rationalist ecosystem: a new, more right-coded, EA-adjacent funding organization could optimize itself for being able to enter many of those blacklisted areas with enthusiasm.

If I was a billionare, I would love to put together a kind of "completion portfolio" to complement some of OP's work.  Rationality community building, macrostrategy stuff, AI-related advocacy to try and influence republican politicians, plus a big biotechnology emphasis focused on intelligence enhancement, reproductive technologies, slowing aging, cryonics, gene drives for eradicating diseases, etc.  Basically it seems like there is enough edgy-but-promising stuff out there (like studying geoengineering for climate, or advocating for charter cities, or just funding oddball substack intellectuals to do their thing) that you could hope to create a kind of "alt-EA" (obviously IRL it shouldn't have EA in the name) where you batten down the hatches, accept that the media will call you an evil villain mastermind forever, and hope to create a kind of protective umbrella for all the work that can't get done elsewhere.  As a bonus, you could engage more in actual politics (like having some hot takes on the US budget deficit, or on how to increase marriage & fertility rates, or whatever), in some areas that OP in its quest for center-left non-polarization can't do.

Peter Thiel already lives this life, kinda?  But his model seems 1. much more secretive, and 2. less directly EA-adjacent, than what I'd try if I was a billionare.

Dustin himself talks about how he is really focused on getting more "multipolarity" to the EA landscape, by bringing in other high-net-worth funders.  For all the reasons discussed, he obviously can't say "hi, somebody please start an edgier right-wing offshoot of EA!!"  But it seems like a major goal that the movement should have, nonetheless.

Seems like you could potentially also run this play with a more fully-left-coded organization.  The gains there would probably be smaller, since there's less "room" to OP's left than to their right.  But maybe you could group together wild animal welfare, invertebrate welfare, digital minds, perhaps some David Pearce / Project Far Out-style "suffering abolition" transhumanist stuff, other mental-wellbeing stuff like the Organization for the Prevention of Intense Suffering, S-risk work, etc.  Toss in some more aggressive political activism on AI (like PauseAI) and other issues (like Georgist land value taxation), and maybe some forward-looking political stuff on avoiding stable totalitarianism, regulation of future AI-enabled technologies, and how to distribute the gains from a positive / successful singularity (akin to Sam Altman's vision of UBI supported by georgist/pigouvian taxes, but more thought-through and detailed and up-to-date.)

Finding some funders to fill these niches seems like it should be a very high priority of the rationalist / EA movement.  Even if the funders were relatively small at first (like say they have $10M - $100M in crypto that they are preparing to give away), I think there could be a lot of value in being "out and proud" (publicising much of their research and philosophy and grantmaking like OP, rather than being super-secretive like Peter Thiel).  If a small funder manages to build a small successful "alt-EA" ecosystem on either the left or right, that might attract larger funders in time.

not even ARC has been able to get OP funding (in that case because of COIs between Paul and Ajeya)

As context, note that OP funded ARC in March 2022.

Isn't it just the case that OpenPhil just generally doesn't fund that many technical AI safety things these days? If you look at OP's team on their website, they have only two technical AI safety grantmakers. Also, you list all the things OP doesn't fund, but what are the things in technical AI safety that they do fund? Looking at their grants, it's mostly MATS and METR and Apollo and FAR and some scattered academics I mostly haven't heard of. It's not that many things. I have the impression that the story is less like "OP is a major funder in technical AI safety, but unfortunately they blacklisted all the rationalist-adjacent orgs and people" and more like "AI safety is still a very small field, especially if you only count people outside the labs, and there are just not that many exciting funding opportunities, and OpenPhil is not actually a very big funder in the field". 

what actually happened is that Open Phil blacklisted a number of ill-defined broad associations and affiliations

is there a list of these somewhere/details on what happened?

As a concrete example, as far as I can piece together from various things I have heard, Open Phil does not want to fund anything that is even slightly right of center in any policy work. I don't think this is because of any COIs, it's because Dustin is very active in the democratic party and doesn't want to be affiliated with anything that is even slightly right-coded. Of course, this has huge effects by incentivizing polarization of AI policy work with billions of dollars, since any AI Open Phil funded policy organization that wants to engage with people on the right might just lose all of their funding because of that, and so you can be confident they will steer away from that.

Thanks for sharing, I was curious if you could elaborate on this (e.g. if there are examples of AI policy work funded by OP that come to mind that are clearly left of center). I am not familiar with policy, but my one data point is the Horizon Fellowship, which is non-partisan and intentionally places congressional fellows in both Democratic and Republican offices. This straightforwardly seems to me like a case where they are trying to engage with people on the right, though maybe you mean not-right-of-center at the organizational level? In general though, (in my limited exposure) I don't model any AI governance orgs as having a particular political affiliation (which might just be because I'm uninformed / ignorant).

Imo sacrificing a bunch of OpenPhil AI safety funding in exchange for improving OpenPhil's ability to influence politics seems like a pretty reasonable trade to me, at least depending on the actual numbers. As an extreme case, I would sacrifice all current OpenPhil AI safety funding in exchange for OpenPhil getting to pick which major party wins every US presidential election until the singularity.

Concretely, the current presidential election seems extremely important to me from an AI safety perspective, I expect that importance to only go up in future elections, and I think OpenPhil is correct on what candidates are best from an AI safety perspective. Furthermore, I don't think independent AI safety funding is that important anymore; models are smart enough now that most of the work to do in AI safety is directly working with them, most of that is happening at labs, and probably the most important other stuff to do is governance and policy work, which this strategy seems helpful for.

I don't know the actual marginal increase in political influence that they're buying here, but my guess would be that the numbers pencil and OpenPhil is making the right call.

I cannot think of anyone who I would credit with the creation or shaping of the field of AI Safety or Rationality who could still get OP funding.

Separately, this is just obviously false. A lot of the old AI safety people just don't need OpenPhil funding anymore because they're working at labs or governments, e.g. me, Rohin Shah, Geoffrey Irving, Jan Leike, Paul (as you mention), etc.

Thanks for the reply. When I wrote "Many people would have more useful things to say about this than I do", you were one of the people I was thinking of.

AI Impacts wants to think about AI sentience and OP cannot fund orgs that do that kind of work

Related to this, I think GW/OP has always been too unwilling to fund weird causes, but it's generally gotten better over time: originally recommending US charities over global poverty b/c global poverty was too weird, taking years to remove their recommendations for US charities that were ~100x less effective than their global poverty recs, then taking years to start funding animal welfare and x-risk, then still not funding weirder stuff like wild animal welfare and AI sentience. I've criticized them for this in the past but I liked that they were moving in the right direction. Now I get the sense that recently they've gotten worse on AI safety (and weird causes in general).

I cannot think of anyone who I would credit with the creation or shaping of the field of AI Safety or Rationality who could still get OP funding.

Nitpick, but this statement seems obviously false given what I understand your views to be? Paul, Carl, Buck, for starters.

[edit: I now see that Oliver had already made a footnote to that effect.]

Out of curiosity - “it's because Dustin is very active in the democratic party and doesn't want to be affiliated with anything that is right-coded” Are these projects related to AI safety or just generally? And what are some examples?

FHI could not get OP funding

Can you elaborate on what you mean by this? 

OP appears to have been one of FHI's biggest funders according to Sandberg:^[1]

Eventually, Open Philanthropy became FHI’s most important funder, making two major grants: £1.6m in 2017, and £13.3m in 2018. Indeed, the donation behind this second grant was at the time the largest in the Faculty of Philosophy’s history (although, owing to limited faculty administrative capacity for hiring and the subsequent hiring freezes it imposed, a large part of this grant would remain unspent). With generous and unrestricted funding from a foundation that was aligned with FHI’s mission, we were free to expand our research in ways we thought would make the most difference.

The hiring (and fundraising) freeze imposed by Oxf began in 2020. 

1. ^{^}

   See page 15

fwiw, FABRIC was able to get funding in November 2024 (who knows if this date is correct though)

nvm this was an "exit grant" lmao

If Open Phil is unwilling to fund some/most of the best orgs, that makes earning to give look more compelling.

(There are some other big funders in AI safety like Jaan Tallinn, but I think all of them combined still have <10% as much money as Open Phil.)

Some of those concerns are, indeed, overly paranoid

I am actually curious if you have any overly paranoid predictions from me. I was today lamenting that despite feeling paranoid on this stuff all the time, I de-facto have still been quite overly optimistic in almost all of my predictions on this topic (like, I only gave SPARC a 50% chance of being defunded a few months ago, which I think was dumb, and I was not pessimistic enough to predict the banning of all right-associated project, and not pessimistic enough to predict a bunch of other grant decisions that I feel weird talking publicly about). 

A few people (i.e. habryka or previously Benquo or Jessicata) make it their thing to bring up concerns frequently.

My impression is that those people are paying a social cost for how willing they are to bring up perceived concerns, and I have a lot of respect for them because of that.

People typically only select into this sort of role if they're a bit more prone to conflict about it, which means a lot of the work is kinda thankless because people are pushing back on you for being too conflicty.

Things can be done to encourage this behavior anway, such as with how the site works. Instead the opposite has been done; this is the root of my many heated disagreements with the LW team.

I should add that I don't want to dissuade people from criticizing me if I'm wrong. I don't always handle criticism well, but it's worth the cost to have accurate beliefs about important subjects. I knew I was gonna be anxious about this post but I accepted the cost because I thought there was a ~25% chance that it would be valuable to post.