Alignment Megaprojects: You're Not Even Trying to Have Ideas

I've heard that it is very difficult to get funding unless you have a paradigmatic idea, and you can't get a job without good technical AI skills. But many people who skill up to get a job in technical alignment end up doing capabilities work because they can't find employment in AI Safety, or the existing jobs don't pay enough. Apparently, this was true for both Sam Altman and Demis Hassabis? I've also experienced someone discouraging me from acquiring technical AI skills for the purpose of pursuing a career in technical alignment because they don't want me to contribute to capabilities down the line. They noted that most people who skill up to work on alignment end up working in capabilities instead, which is kinda crazy.

My thinking is that I am just built different and will come up with a fundable paradigmatic idea where most fail. But yeah, the lack of jobs heavily implies that the field is funding-constrained because talent wants to work on alignment.

Much cheaper, though still hokey, ideas that you should have already thought of at some point:

• A "formalization office" that checks and formalizes results by alignment researchers. It should not take months for a John Wentworth result to get formalized by someone else [LW · GW].
• Mathopedia [LW · GW].
• Alignment-specific outreach at campuses/conventions with top cybersecurity people.

Thanks for the post. I think it's a valuable exercise to think about how AI safety could be accelerated with unlimited money.

I think the Manhattan Project idea is interesting but I see some problems with the analogy:

• The Manhattan Project was originally a military project and to this day, the military is primarily funded and managed by the government. But most progress in AI today is made by companies such as OpenAI and Google and universities like the University of Toronto. I think a more relevant project is CERN because it's more recent and focused on the non-military development of science.
• The Manhattan Project happened a long time ago and the world has changed a lot since then. The wealth and influence of tech companies and universities is probably much greater today than it was then.
• It's not obvious that a highly centralized effort is needed. The Alignment Forum, open source developers, and the academic research community (e.g. the ML research community) are examples of decentralized research communities that seem to be highly effective at making progress. This probably wasn't possible in the past because the internet didn't exist.

I highly doubt that it's possible to recreate the Bay Area culture in a top-down way. I'm pretty sure China has tried this and I don't think they've succeeded.

Also, I think your description is overemphasizing the importance of geniuses like Von Neumann because 130,000 other people worked on the Manhattan Project too. I think something similar has happened at Google today where Jeff Dean is revered but in reality, I think most progress at Google is done by the tens of thousands of the smart but not genius dark matter developers there.

Anyway, let's assume that we have a giant AI alignment project that would cost billions. To fund this, we could:

1. Expand EA funding substantially using community building.
2. Ask the government to fund the project.

The government has a lot of money but it seems challenging to convince the government to fund AI alignment compared to getting funding from EA. So maybe some EAs with government expertise could work with the government to increase AI safety investment.

If the AI safety project gets EA funding, I think it needs to be cost-effective. The reality is that only ~12% of Open Phil's money is spent on AI safety. The reason why is that there is a triage situation with other cause areas like biosecurity, farm animal welfare, and global health and development so the goal is to find cost-effective ways to spend money on AI safety. The project needs to be competitive and has more value on the margin than other proposals.

In my opinion, the government projects that are most likely to succeed are those that build on or are similar to recent successful projects and are in the Overton window. For example:

• AI Centres for Doctoral Training in the UK: funding PhD students in the UK to work on AI projects such as AI safety.
• The NSF Safe Learning-Enabled Systems: US government funding for academic research groups and non-profits to work on AI safety.

My guess is that leveraging academia would be effective and scalable because you can build on the pre-existing talent, leadership, culture, and infrastructure. Alternatively, governments could create new regulations or laws to influence the behavior of companies (e.g. GDPR). Or they could found new think tanks or research institutes possibly in collaboration with universities or companies.

As for the school ideas, I've heard that Lee Sedol went to a Go school and as you mentioned, Soviet chess was fueled by Soviet chess programs. China has intensive sports schools but I doubt these kinds of schools would be considered acceptable in Western countries which is an important consideration given that most of AI safety work happens in Western countries like the US and UK.

In science fiction, there are even more extreme programs like the Spartan program in Halo where children were kidnapped and turned into super soldiers, or Star Wars where clone soldiers were grown and trained in special facilities.

I don't think these kinds of extreme programs would work. Advanced technologies like human cloning could take decades to develop and are illegal in many countries. Also, they sound highly unethical which is a major barrier to their success in modern developed countries like the US and especially EA-adjacent communities like AI safety.

I think a more realistic idea is something like the Atlas Fellowship or SERI MATS which are voluntary programs for aspiring researchers in their teens or twenties.

The geniuses [LW · GW] I know of that were trained from an early age in Western-style countries are Mozart (music), Von Neumann (math), John Stuart Mill (philosophy), and Judit Polgár (chess). In all these cases, they were gifted children who lived in normal nuclear families and had ambitious parents and extra tutoring.

My experience has been that we are clearly funding constrained, particularly in a post FTX world. This makes it hard to recruit top talent and is likely driving people out of alignment research.

(All opinions expressed are my own. Details of examples have been changed. None of these stories directly concern experiences with my own funding. I currently have active grant applications which has incentivised me to make this comment less harsh than it probably should be.)

I'm aware of researchers who have turned down extremely prestigious careers to pursue alignment, been promised certain amounts of funding and then had that funding substantially delayed.

I'm aware of researchers with funding giving cash loans to other researchers who were experiencing funding delays.

Such an environment does not bode well for convincing people to stick to alignment. No adult with responsibilities feels secure working contract to contract.

Beyond that, alignment salaries are smaller than what you would make in tech. This makes it difficult to poach senior talent from FAAMG.

• Indefinitely-long-timespan basic minimum income for everyone who

Looks like part of the sentence is missing

move to one place

No, we need to maintain diversity and enough decentralization (e.g. because of the need for resilience against a single missile nuclear attack (a single super-valuable site is very vulnerable)).

Moreover, since "AI existential safety" is a preparadigmatic field, it is particularly important to be able to explore various non-standard ideas (government control, and, especially, military control is not conductive in this sense).

There is a much cheaper idea (which still needs some money and, more crucially, needs some extremely strong organizational skills): promote creation of small few-person short-term collaborations exploring various non-standard ideas in "AI existential safety" (basically, let's have a lot of brainstorms, let's have infrastructure to support those brainstorms and to support sharing their fruits, let's figure out how to really make rapid progress here in a fashion which is a network and not a hierarchy (the problem is probably too difficult to be rapidly solved by a hierarchy)).

That would generate novel ideas. (We just need to keep in mind that anything which actually works in "AI alignment" (or, more generally, in any approach to "AI existential safety", whether alignment-based or not) is highly likely to be dual use and a major capability booster. No one knows how to handle this correctly.)

we don't seem to have Von Neumanns anymore

Oh, but we do. For example (and despite my misgivings about OpenAI current approach to alignment), Ilya Sutskever is of that caliber (AlexNet, GPT-3, GPT-4, and a lot of other remarkable results speak for themselves). And now he will focus on alignment.

That being said, genius scientists are great for doing genius things, but not necessarily the best policy decision makers (e.g. von Neumann strongly advocated preventive nuclear attack against Soviet Union, and Oppenheimer after opposing thermonuclear weapons actually suggested to use an early prototype thermonuclear device in the Korean war).

So technical research and technical solutions is one thing, but decision-making and security-keeping is very different (seem to require very different people with very different skills).

Another project idea that should at least be written down somewhere Official, even if it's never implemented.

You seem to be in the mindset that everything is as EY/LW says ... but there is precious little evidence for that outside the echo chamber.

An entire school system (or at least an entire network of universities, with university-level funding) focused on Sequences-style rationality in general and AI alignment in particular.

Is there evidence that extreme rationality works? (against [LW · GW] ) Is there evidence that the people with real achivemnents -- LeCunn, Ng etc -- are actually crippled by lack of rationality? Can you teach alignment separately from AI? (cf can you teach safety engineering to someone who doesn't know engineering?)

Indefinitely-long-timespan basic minimum income for everyone who is working on solely AI alignment.

How do you separate people who are actually working on alignment from scammers? How do you motivate them to produce results with an unconditional, indefinite term payment? Would minimum income be enough to allow them to buy equipment and hire assistants? (Of course, all these problems are solved by conventional research grants).

Again, you seem to be making the High Rationalist assumption that alignment is a matter of some unqualified person sitting in a chair thinking, not of a qualified person doing practical work.

EDIT: I think this comment was overly harsh, leaving it below for reference. The harsh tone was contributed from being slightly burnt out from feeling like many people in EA were viewing me as their potential ender wiggin, and internalizing it.^[1]

The people who suggest schemes like what I'm criticizing are all great people who are genuinely trying to help, and likely are.

Sometimes being a child in the machine can be hard though, and while I think I was ~mature and emotionally robust enough to take the world on my shoulders, many others (including adults) aren't.

An entire school system (or at least an entire network of universities, with university-level funding) focused on Sequences-style rationality in general and AI alignment in particular.

[...]

Genetic engineering, focused-training-from-a-young-age, or other extreme "talent development" setups.

Please stop being a fucking coward speculating on the internet about how child soldiers could solve your problems for you. Enders game is fiction, it would not work in reality, and that isn't even considering the negative effects on the kids. You aren't smart enough for galaxy brained plans like this to cause anything other than disaster.

In general rationalists need to get over their fetish for innate intelligence and actually do something instead of making excuses all day. I've mingled with good alignment researchers, they aren't supergeniuses, but they did actually try.

(This whole comment applies to Rationalists generally, not just the OP.)