A note about differential technological development

First, I totally agree, and in general think that if the AI risk community can't successfully adopt a norm of "don't advance capabilities research" we basically can't really collaborate on anything.

I'm saddened to hear you mention Anthropic. I was holding out hope after SBF invested and looking at their site that they were the only present AGI company that was serious about not advancing the public state of capabilities research. Was that naive? Where do they fall along the Operational Adequacy [LW · GW] spectrum?

FWIW, I had a mildly negative reaction to this title. I agree with you, but I feel like the term "PSA" should be reserved for things that are really very straightforward and non-controversial, and I feel like it's a bit of a bad rhetorical technique to frame your arguments as a PSA. I like the overall content of the post, but feel like a self-summarizing post title like "Most AI Alignment research is not parallelizable" would be better.

I kind of want you to get quantitative here? Like pretty much every action we take has some effect on AI timelines, but I think effect-on-AI-timelines is often swamped by other considerations (like effects on attitudes around those who will be developing AI).

Of course it's prima facie more plausible that the most important effect of AI research is the effect on timelines, but I'm actually still kind of sceptical. On my picture, I think a key variable is the length of time between when-we-understand-the-basic-shape-of-things-that-will-get-to-AGI and when-it-reaches-strong-superintelligence. Each doubling of that length of time feels to me like it could be worth order of 0.5-1% of the future. Keeping implemented-systems close to the technological-frontier-of-what's-possible could help with this, and may be more affectable than the

Note that I don't think this really factors into an argument in terms of "advancing alignment" vs "aligning capabilities" (I agree that if "alignment" is understood abstractly the work usually doesn't add too much to that). It's more like a DTD argument about different types of advancing capabilities.

I think it's unfortunate if that strategy looks actively bad on your worldview. But if you want to persuade people not to do it, I think you either need to persuade them of the whole case for your worldview (for which I've appreciated your discussion of the sharp left turn), or to explain not just that you think this is bad, but also how big a deal do you think it is. Is this something your model cares about enough to trade for in some kind of grand inter-worldview bargaining? I'm not sure. I kind of think it shouldn't be (that relative to the size of ask it is, you'd get a much bigger benefit from someone starting to work on things you cared about than stopping this type of capabilities research), but I think it's pretty likely I couldn't pass your ITT here.

Dan Hendrycks (note: my employer) and Mantas Mazeika have recently covered some similar ideas in their paper X-Risk Analysis for AI Research, which is aimed at the broader ML research community. Specifically, they argue for the importance of having minimal capabilities externalities while doing safety research and addresses some common counterarguments to this view including those similar to the ones you've described. The reasons given for this are somewhat different to your serial/parallel characterization, so I think serve as a good complement. The relevant part is here.

I found this very persuasive and convincing provided that success looks only like a comprehensive theory of alignment.

In worlds where a non rigorous "craft" of alignment is "good enough" for capabilities within a certain range, I don't think this model/your conclusions necessarily hold.

And the dynamics of takeoff will also determine if having a craft of alignment that's good enough for par human AI but not superhuman AI is success (if takeoff is slow enough, you can use your aligned HLAI to help you figure out alignment for superhuman AI).

You can generalise this further to a notion of alignment escape velocity. You don't yet have a comprehensive theory of alignment that's robust to arbitrary capability amplification, but you have a craft of alignment that's robust to current/near term capabilities and can use current aligned AI to help develop your alignment craft for more capable agents, before those agents are then developed (this lead of alignment craft over capabilities needs to hold for "alignment escape velocity". In practice it probably looks like coordination to not develop more powerful AI than we know how to align.)

I expect much slower takeoff than EY (at least a few years between a village idiot and John Von Neumann, and even longer to transition from John Von Neumann to strongly superhuman intelligence [I expect marginal returns to cognitive investment to diminish considerably as you advance along the capabilities frontier]) , so I'm very sympathetic to alignment escape velocity.

 
(All of the above said, I'm still young [24], so I'll probably devote my career [at least the first decade of it] to the "figure out a comprehensive theory of alignment" thing. It optimises for personal impact on a brighter future [which is very important to me], and it sounds like a really fun problem [I like abstract thinking].)

How do you think about empirical work on scalable oversight? A lot of scalable oversight methods do result in capabilities improvements if they work well. A few concrete examples where this might be the case:

1. Learning from Human Feedback
2. Debate
3. Iterated Amplification
4. Imitative Generalization

I'm curious which of the above you think it's net good/bad to get working (or working better) in practice. I'm pretty confused about how to think about work on the above methods; they're on the main line path for some alignment agendas but also advanced capabilities / reduce serial time to work on the other alignment agendas.

I tend to agree that burning up the timeline is highly costly, but more because Effective Altruism is an Idea Machine that has only recently started to really crank up. There's a lot of effort being directed towards recruiting top students from uni groups, but these projects require time to pay off.

I’m giving this example not to say “everyone should go do agent-foundations-y work exclusively now!”. I think it’s a neglected set of research directions that deserves far more effort, but I’m far too pessimistic about it [? · GW] to want humanity to put all its eggs in that basket.

If it is the case that more people should go into Agent Foundations research then perhaps MIRI should do more to enable it?

Curated.  This is a bit of an older post but seemed important. I know a lot of people asking "When is it a good idea to do work that furthers AI capabilities (even if it also helps alignment?)" – both researchers, and funders. I think this post adds a crisp extra consideration to the question that I hadn't seen spelled out before.

I think the word "capabilities" is actually not quite what the word we're looking for here, because the part of "capabilities" that negatively affects timelines is pretty much just scaling work.  You can make a reasonable argument that the doomsday clock is based on how fast we get to an AI that's smarter than we are; which is to say, how quickly we end up being able to scale to however many neurons it takes to become smarter than humans.  And I don't expect that prompt research or intepretability research or OOD research is really going to meaningfully change that, or at least I don't recall having read any argument to the contrary.

Put another way: do you think empirical alignment research that doesn't increase capabilities can ever exist, even in principle?

EDIT: On reflection, I guess in addition to scaling that "agent goal-directedness" work would fall under the heading of things that are both (1) super dangerous for timelines and (2) super necessary to get any kind of aligned AI ever.  I do not know what to do about this.

I'm quite confused about the toy model (1000 person-years of research, 900 of which "can be done in parallelizable 5-year chunks," 100 of which "factor into four chunks that take 25 serial years apiece").

1. What is a "person-year of research"? I'm parsing as a year of progress from someone working at an optimal level with all the necessary background knowledge. "All the necessary background knowledge" is overly simplistic, as I can imagine plenty of subproblems that take X years for one person to solve, but the 2nd year relies on knowledge in the 1st, etc. Also a year of research can accomplish much more depending on interaction effects with collaborators.
2. There's an example of 180 people each working for 5 years to do the 900 parallelizable person years, or alternatively 1800 people working for 10 years if 10% do their job correctly half the time. What if we had 1800 people doing their job correctly all the time -- would that still take 5 years in this model?
3. The remaining 100 person-years are also parallelizable into 25-year chunks, no? It seems that "more parallelizable vs less parallelizable" is a better framing than "parallelizable vs serial." 
4. It seems that the core idea here is that some problems require a minimum length of time to solve due to relevant subtasks needing to happen sequentially. This still applies to the "parallelizable 5-year chunks" -- they each take a minimum of 5 years.

Also, can someone please define the terms "serial time" and "serial alignment research"?

sidereal

typo?

Alignment seems like a really hard problem, and having smarter humans could be really important.

AGI doesn't seem to need as much intelligence -- just scale what we already have, and experiments with various tweaks.

We could have highly enhanced adult humans in 30 years (although promising projects/teams are currently significantly underfunded).

This too requires us to buy serial time. Because you can't make a baby in one month by having 9 people pregnant.

Here's my naive summary; please critique if I'm misunderstanding. EDIT: I think I'm framing things differently from the post. Here's a separate comment [LW(p) · GW(p)] with my confusions about the post's framing.

• Say there are 3 parts/chunks/segments of solving alignment that have to happen sequentially (each one can only start once the previous part is complete), and each part requires at least 5 years no matter how many people we throw at it. In this world, you need at least 15 years to solve alignment.
  • Why does each part take at least 5 years? Because of diminishing returns. There are constraints on how much you can parallelize the work, so eventually new people can’t do anything but work on projects that other people are already working on (in a way that doesn’t speed up progress on these projects). 
    • I'm not sure how rapidly returns will diminish. Researcher retreats, operations work, personal assistants, AI tools, etc could continue to shorten the time to complete each chunk even after technical researchers saturate the field. Regardless, there is some minimum length of time required across all realistic worlds.
• If your research project helps with parallelizing the current chunk so that we can do it in e.g. 7 years instead of 8 years, that seems great -- except what if it also decreases timelines from 16 years to 14 years? Then it’s a death sentence.
  • Shortening timelines burns/reduces our serial time -- the time we have left to complete all 3 chunks (basically a reframing of "timelines"?). Trading serial time for more labor (parallelized to solve the current chunk) is extremely dangerous, even though it helps us complete the current chunk faster.
• How might progress be serial? Maybe we need to deeply understand minds/optimization/agency before we can start coming up with alignment proposals that actually work. Within each of these steps there are more serial substeps. Also, progressing from one major step to the next might require large “paradigm shifts,” and these shifts might not happen without some fresh eyes entering the field (due to shortcomings of human psychology?), which might take several years at a minimum.

Suppose that aligning an AGI requires 1000 person-years of research.

• 900 of these person-years can be done in parallelizable 5-year chunks (e.g., by 180 people over 5 years — or, more realistically, by 1800 people over 10 years, with 10% of the people doing the job correctly half the time).
• The remaining 100 of these person-years factor into four chunks that take 25 serial years apiece (so that you can't get any of those four parts done in less than 25 years).

Do you have a similar model for just building (unaligned) AGI? Or is the model meaningfully different? On a similar model for just building AGI, then timelines would mostly be shortened by progressing through the serial research-person-years instead of the parallelisable research-person-years. If researchers who are progressing both capabilities and aligning are doing both in the parallelisable part, then this would be less worrying, as they're not actually shortening timelines meaningfully.

Unfortunately I imagine you think that building (unaligned) AGI quite probably doesn't have many more serial person-years of research required, if any. This is possibly another way of framing the prosaic AGI claim: "we expect we can get to AGI without any fundamentally new insights on intelligence, using (something like) current methods."

I like the distinction between parallelizable and serial research time, and agree that there should be a very high bar for shortening AI timelines and eating up precious serial time.

One caveat to the claim that we should prioritize serial alignment work over parallelizable work, is that this assumes an omniscient and optimal allocator of researcher-hours to problems. Insofar as this assumption doesn't hold (because our institutions fail, or because the knowledge about how to allocate researcher-hours itself depends on the outcomes of parallelizable research) the distinction between parallelizable and serial work breaks down and other considerations dominate.