Why I'm Optimistic About Near-Term AI Risk

I see at least two problems with your argument:

1. There's an assumption that you need a single agent to lead to existential risk. This is not the case, and many scenarios explored require only competent and autonomous service like AIs, or foundations models. Like, CAIS is a model of intelligence explosion and has existential risks type failure modes too.
2. There's an assumption that just because the non AGI models are useful, labs will stop pursuing AGI. Yet this is visibly false, as the meme of AGI is running around and there are multiple labs who are explicitly pushing for AGI and getting the financial leeway to do it.

More generally, this post has the typical problem of "here is a scenario that looks plausible and would be nice, so there's no need to worry". Sure, maybe this is the actual scenario that will come to pass, and maybe it's possible to argue for it convincingly. But you should require one damn strong argument before pushing people to not even work to deal with the many more possible numerous worlds where things go horribly wrong.

I basically agree with this vision and I also agree that the many recent pessimistic takes are not representative of the field as a whole.

You may be right that the recent pessimistic takes aren't representative of the field as a whole... but I think you also may be wrong. I say, instead of speculating about it, let's do some surveys! I am curious.

I for one think that existential risk from AI is quite high in the next decade or so -- maybe like 50% or more. I don't know what you mean by "can't see this path leading to high existential risk" but if by "high" you mean "10% or more" then I'd bet that most people in AI safety disagree with you and if you mean "around 50% or more" then there's a substantial minority who disagree with you.

Potential counterarguments:

1. Unpredictable gain of function with model size that exceeds scaling laws. This seems to just happen every time a significantly larger model is trained in the same way on similar data-sets as smaller models.

2. Unexpected gain of function from new methods of prompting, e.g. chain-of-thought which dramatically increased PaLM's performance, but which did not work quite as well on GPT-3. These seem to therefore be multipliers on top of scaling laws, and could arise in "tool AI" use unintentionally in novel problem domains.

3. Agent-like behavior arises from pure transformer-based predictive models (Gato) by taking actions on the output tokens and feeding the world state back in; this means that perhaps many transformers are capable of agent-like behavior with sufficient prompting and connection to an environment.

4. It is not hard to imagine a feedback loop where one model can train another to solve a sub-problem better than the original model, e.g. by connecting a Codex-like model to a Jupyter notebook that can train models and run them, perhaps as part of automated research on adversarial learning producing novel training datasets. Either the submodel itself or the interaction between them could give rise to any of the first three behaviors without human involvement or oversight.

The OP doesn’t explicitly make this jump, but it’s dangerous to conflate the claims “specialized models seem most likely” and “short-term motivated safety research should be evaluated in terms of these specialized models”.

I agree with the former statement, but at the same time, the highest x-risk/ highest EV short-term safety opportunity is probably different. For instance, a less likely but higher impact scenario: a future code generation LM either directly or indirectly* creates an unaligned, far improved architecture. Researchers at the relevant org do not recognize this discontinuity and run the model, followed by disaster.

*E.g. a model proposing an improved Quoc Le style architecture search seems quite plausible to me.

I can give further evidence that this scenario is at least somewhat probable.

Due to the anthropic principle, general intelligence could have a one-in-an-octillion chance of ever randomly evolving, anywhere, ever, and we would still be here observing all the successful steps having happened, because if all the steps didn't happen then we wouldn't be here observing anything. There would still be tons of animals like ants and chimpanzees because evolution always creates a ton of alternative "failed" offshoots. So it's always possible that there's some logical process that's necessary for general intelligence, and we're astronomically unlikely to discover it randomly, through brute forcing or even innovation, until we pinpoint all the exact lines of code in the human brain that distinguishes our intelligence from chimpanzees.

Basically, the anthropic principle indicates the possibility of at least one more AI winter ahead, since even if we suddenly pumped out an AI at the chimpanzee level of intelligence, we could still be astronomically far away from the last steps for human-level general intelligence. We just have no idea how unlikely human-level intelligence is to emerge randomly, just like how we have no idea how unlikely life is to emerge randomly.

However, it's only a possibility that this is the case. General intelligence could still be easy to brute force, and we'd also still be here. The recent pace of AI development definitely indicates bad news for AGI timelines, and it doesn't make sense to unplug a warning light instead of looking for the hazard it corresponds to. 

But in terms of "log odds of human survival beyond 20 years", that's a pretty unreasonable estimate. There isn't nearly enough evidence to conclude that the human race is "almost certainly doomed soon", only "significantly more likely than before to worry about nearer-term AGI".

I find the story about {lots of harmless, specialized transformer-based models being developed} to be plausible. I would not be surprised if many tech companies were to follow something like that path.

However, I also think that the conclusion --- viz., it being unlikely that any AI will pose an x-risk in the next 10-20 years --- is probably wrong.

The main reason I think that is something like the following:

In order for AI to pose an x-risk, it is enough that even one research lab is a bit too incautious/stupid/mistakenly-optimistic and "successfully" proceeds with developing AGI capabilities. Thus, the proposition that {AI will not pose an x-risk within N years} seems to require that $$\forall \text{ResearchLab}\in \left\{\text{all research labs that will exist over next N years}\right\}\forall \text{Techniques}\subseteq \{\text{ML techniques available in next N years}\text{ResearchLab}\text{can/will NOT develop dangerous AGI with}\text{Techniques}$$

And the above is basically a large conjunction over many research labs and as-yet-unknown future ML technologies. I think it is unlikely to be true. Reasons why I think it is unlikely to be true:

• It seems plausible to me that highly capable, autonomous (dangerous) AGI could be built using some appropriate combination of already existing techniques + More Compute.

• Even if it weren't/isn't possible to build dangerous AGI with existing techniques, a lot of new techniques can be developed in 10 years.

• There are many AI research labs in existence. Even if most of them were to pursue only narrow/satisfactory AI, what are the odds that not one of them pursues (dangerous) autonomous AGI?

• I'm under the impression that investment in A(G)I capabilities research is increasing pretty fast; lots of smart people are moving into the field. So, the (near) future will contain even more research labs and sources of potentially dangerous new techniques.

• 10 years is a really long time. Like, 10 years ago it was 2012, deep learning was barely starting to be a thing, the first Q-learning-based Atari-playing model (DQN) hadn't even been released yet, etc. A lot of progress has happened from 2012 to 2022. And the amount of progress will presumably be (much) greater in the next 10 years. I feel like I have almost no clue what the future will look like in 10-20 years.

• We (or at least I) still don't even know any convincing story of how to align autonomous AGI to "human values" (whatever those even are). (Let alone having practical, working alignment techniques.)

Given the above, I was surprised by the apparent level of confidence given to the proposition that "AI is unlikely to pose an existential risk in the next 10-20 years". I wonder where OP disagrees with the above reasoning?

Regarding {acting quickly} vs {movement-building, recruiting people into AI safety, investing in infrastructure, etc.}: I think it's probably obvious, but maybe bears pointing out, that when choosing strategies, one should consider not only {the probability of various timelines} but also {the expected utility of executing various strategies under various timelines}.

(For example, if timelines are very short, then I doubt even my best available {short-timelines strategy} has any real hope of working, but might still involve burning a lot of resources. Thus: it probably makes sense for me to execute {medium-to-long timelines strategy}, even if I assign high probability to short timelines? This may or may not generalize to other people working on alignment.)

At the moment we are seeing a host of simple generic ML algorithms. The type where GPT3 and DALLE2 are stereotypical examples. I wouldn't go so far as saying this will last 10 years, let alone 20. 

But before that, deployment of these systems will favor smaller, faster, and more auditable models leading companies to focus on distilled models specializing in specific tasks.

Suppose a task for which there is little training data on exactly that task. You can't train a good chatbot on a page of text. But you can fine tune GPT3 on a page of text. Training a general model and then fine tuning is a useful strategy. (And someone with loads of compute will go as general as they can.)

So when a large ML model is trained on  a wide variety of tasks, (lets suppose its trained using RL.) Can it be dangerous? I think this is a difficult question, and relates to how much large neural nets can learn deep patterns as opposed to shallow memorizing.

I can't see this path leading to high existential risk in the next decade or so.

Here [LW · GW]is my write-up for a reference class of paths that could lead to high existential risk this decade. I think such paths are not hard to come up with and I am happy to pay a bounty of $100 for someone else to sit for one hour and come up with another story for another reference class (you can send me a DM).

Even if the Tool AIs are not dangerous by itself, they will foster productivity. (You say it yourself: "These specialized models will be oriented towards augmenting human productivity"). There is already a many more people working in AI than in the 2010s, and those people are much more productive. This trend will accelerate, because AI benefits compound (eg. using Copilot to write the next Copilot) and the more ML applications automate the economy, the more investments in AI we will observe.

For me, the biggest most tangible risks from AI are from those AI (agent or tool AIs) that are connected to the real world and can influence human incentives, especially those that can work at a global level.

If you have a Tool AI connected to the real world and able to influence humans through strong incentives, wouldn't this be a very high risk even if they never become full AGI?

There are actual real-life examples doing this right now, most notoriously AIs used for trading financial assets. These AIs are optimizing for profit, by playing with one of the strongest human incentives: money. Changes in financial incentives worldwide can trigger bankruptcies, bank-runs, currency and economic collapses.

I am not claiming a single AI might do this (although it might if given enough resources), but we are far from understanding what happens when multiple competing AIs are trying to overperform each other on the financial markets while trying to capture the maximum profit. 

A 2nd less dangerous kind are those AIs that optimize for engagement (ie: Facebook/YouTubue/Twitter/Instagram/TikTok Feeds). The risk here is maximizing engagement equals to capturing maximum human attention, which is a zero-sum game, capturing attention away from other real-life activities (like studying, researching, bonding, helping others). Add to these AIs the capability to create content (GPT-3/DALL-E) and you might create a human-slaving tool.

These things are happening right now, in front of our noses and we are completely unaware of the damage that they might be causing and can cause in the future.