Some disjunctive reasons for urgency on AI risk

Solving problems is mostly a matter of total resources devoted, not time devoted. ... If we get signs of a problem looming, and can devote a lot of resources then.

Hmm. I don't have as strong opinions about this, but this premise doesn't seem obviously true.

I'm thinking about the "is science slowing down [LW · GW]?" question – pouring 1000x resources into various scientific fields didn't result in 1000x speedups. In some cases progress seemed to slow down. The three main hypotheses I have are:

• Low hanging fruit got used up, so the problems got harder
• Average careerist scientists don't matter much, only extremely talented, naturally motivated researchers matter. The naturally motivated researchers will do the work anyway.
• Coordination is hard and scales with the number of people coordinating. If you have 1000x the researchers in a field, they can't find each other's best work that easily.

I agree that "time spent" isn't the best metric, but it seems like what actually matters is "quality researcher hours that build on each other in the right way," and it's not obvious how much you can scale that.

If it's just the low hanging fruit hypothesis then... that's fine I guess. But if the "extreme talent/motivation" or "coordination" issues are at play, then you want (respectively) to ensure that:

a) at any given time, talented people who are naturally interested in the problem have the freedom to work on it, if there are nonzero things to do with it, since there won't be that many of them in the future.

b) build better coordination tools so that people in the future can scale their efforts better.

(You may also want to make efforts not to get mediocre careerist scientists involved in the field)

I think that this problem is in the same broad category as "invent general relativity" or "prove the Poincare conjecture". That is, for one thing quantity doesn't easily replace talent (you couldn't invent GR just as easily with 50 mediocre physicists instead of one Einstein), and, for another thing, the work is often hard to parallelize (50 Einsteins wouldn't invent GR 50 times as fast). So, you can't solve it just by spending lots of resources in a short time frame.

In software development, a perhaps relevant kind of problem solving, extra resources in the form of more programmers working on the same project doesn't speed things up much. My guesstimate is output = time x log programmers. I assume the main reason being because there's a limit to the extent that you can divide a project into independent parallel programming tasks. (Cf 9 women can't make a baby in 1 month.)

Except that if the people are working in independent smaller teams, each trying to crack the same problem, and *if* the solution requires a single breakthrough (or a few?) which can be made by a smaller team (e.g. public key encryption, as opposed to landing a man on the moon), then presumably it's proportional to the number of teams, because each has an independent probability of making the breakthrough. And it seems plausible that solving AI threats might be more like this.

I'm not actually aware of a really good argument for AGI coming soon (i.e., within next few decades). As far as I can tell, most people use their own intuitions and/or surveys of AI researchers (both of which are of course likely to be biased). My sense is that it's hard to reason explicitly about AGI timelines (in a way that's good enough to be more trustworthy than intuitions/surveys) and there seem to be enough people concerned about foom and/or short timelines that funding isn't a big constraint so there's not a lot of incentives for AI risk people to spend time on making such explicit arguments. (ETA: Although I could well be wrong about this, and there's a good argument somewhere that I'm not aware of.) To give a sense of how people are thinking about this, I'll quote a Paul Christiano interview:

I normally think about this question in terms of what’s the probability of some particular development by 10 or 20 years rather than thinking about a median because those seem like the most decision relevant numbers, basically. Maybe one could also, if you had very short timelines give probabilities on less than 10 years. I think that my probability for human labor being obsolete within 10 years is probably something in the ballpark of 15%, and within 20 years is something within the ballpark of 35%. AI would then have, prior to human labor being obsolete, you have some window of maybe a few years during which stuff is already getting quite extremely crazy. Probably AI [risk 01:09:04] becomes a big deal. We can have permanently have sunk the ship like somewhat before, one to two years before, we actually have human labor being obsolete.

Those are my current best guesses. I feel super uncertain about … I have numbers off hand because I’ve been asked before, but I still feel very uncertain about those numbers. I think it’s quite likely they’ll change over the coming year. Not just because new evidence comes in, but also because I continue to reflect on my views. I feel like a lot of people, whose views I think are quite reasonable, who push for numbers both higher and lower, or there are a lot of people making reasonable arguments for numbers both much, like shorter timelines than that and longer timelines than that.

Overall, I come away pretty confused with why people currently are as confident as they are in their views. I think compared to the world at large, the view I’ve described is incredibly aggressive, incredibly soon. I think compared to the community of people who think about this a lot, I’m more somewhere in, I’m still on the middle of the distribution. But amongst people whose thinking I most respect, maybe I’m somewhere in the middle of the distribution. I don’t quite understand why people come away with much higher or much lower numbers than that. I don’t have a good … It seems to me like the arguments people are making on both sides are really quite shaky. I can totally imagine that after doing … After being more thoughtful, I would come away with higher or lower numbers, but I don’t feel convinced that people who are much more confident one way or the other have actually done the kind of analysis that I should defer to them on. That’s said, I also I don’t think I’ve done the kind of analysis that other people should really be deferring to me on.

My own thinking here is that even if AGI comes a century or more from now, the safest alignment approaches [LW · GW] seem to require solving a number of hard philosophical problems which may well take that long to solve even if we start now. Certainly it would be pretty hopeless if we only started when we saw a clear 10-year warning. This possibility also justifies looking more deeply into other approaches now to see if they could potentially be just as safe without solving the hard philosophical problems.

Another thought that is prompted by your question is that given funding does not seem to be the main constraint on current alignment work (people more often cite "talent"), it's not likely to be a limiting constraint in the future either, when the warning signs are even clearer. But "resources today can be traded for a lot more resources later" doesn't seem to apply if we interpret "resources" as "talent".