What is the best critique of AI existential risk arguments?
post by joshc (joshua-clymer) · 2022-08-30T02:18:42.463Z · LW · GW · 3 commentsThis is a question post.
Contents
Answers 12 JBlack 7 Thomas Larsen 1 Thomas Larsen 1 Phil Tanny None 3 comments
If you could link to an article or other piece of media, that would be ideal. Writing one up here is fine as well. An equivalent question would be "what is the best argument for the claim that there is a <1% probability of AI existential risk?"
Answers
Less than 1%? You need to go to fairly ridiculous extremes to get that level of certainty for even things that we know a lot about. I think that level of certainty for any such question is beyond any rational argument with current evidence.
There are plenty of arguments that would work, if only you held some particular prior belief to that level. A rock-solid belief in a higher power that would prevent it would suffice - for those who believe in such a thing at a 99%+ level. A similarly strong belief that AGI is actually impossible and not just difficult would also suffice.
80,000 hours recent problem profile on AI lists some reasons that this might be wrong.
↑ comment by Vladimir_Nesov · 2022-08-30T04:22:52.323Z · LW(p) · GW(p)
There are many good arguments, but not that particular "<1% probability" proof that the question requests. All the good arguments rely on uncertain assumptions, don't reach the requisite standard of proof, especially when considered together with the assumptions.
So by answering this way you are steelmanning the question (which it desperately needs).
An anonymous academic wrote a review of Joe Carlsmith's 'Is power seeking AI an existential risk?', in which the reviewer assigns for <1/100,000 probability of AI existential risk. The arguments given aren't very good imo, but maybe worth reading.
If we were to respond specifically to the title of the post....
What is the best critique of AI existential risk arguments?
I would cast my vote for the premise that AI risk arguments don't really matter so long as a knowledge explosion feeding back upon itself is generating ever more, ever larger powers, at an ever accelerating rate.
For example, let's assume for the moment that 1) AI is an existential risk, and 2) we solve that problem somehow so that AI becomes perfectly safe. Why would that matter if civilization is then crushed when we lose control of some other power emerging from the knowledge explosion? Remember, triumphing over existential risk will require us to win every single time, and never lose once.
If it's true that 1) the knowledge explosion is accelerating, and if it's true that 2) human ability is limited, then it follows that at some point we will be overwhelmed by one or more challenges that we can't adapt to in time.
Seventy five years after Hiroshima we still have no idea what to do about nuclear weapons, nor do we know what to do about AI, or genetic engineering. And the threats keep coming, more and more, larger and larger, faster and faster.
If it is our choice to accept an ever accelerating knowledge explosion as a given, the best critique of AI existential risk arguments seems to be that they don't really matter. Or, if you prefer, that they are a distraction from what does matter.
↑ comment by Mitchell_Porter · 2022-09-02T14:26:21.219Z · LW(p) · GW(p)
Your paradigm, if I understand it correctly, is that the self-sustaining knowledge explosion of modern times is constantly hatching new technological dangers, and that there needs to be some new kind of response - from the whole of civilization? just from the intelligentsia? It's unclear to me if you think you already have a solution.
You're also saying that focus on AI safety is a mistake, compared with focus on this larger recurring process, of dangerous new technologies emerging thanks to the process of discovery.
There are in fact good arguments that AI is now pivotal to the whole process and also to its resolution. However, I would first like to hear what your own recommendations are, before presenting an AI-centric perspective.
Replies from: Phil Tanny↑ comment by Phil Tanny · 2022-09-02T14:59:57.289Z · LW(p) · GW(p)
Thanks much for your engagement Mitchell, appreciated.
Your paradigm, if I understand it correctly, is that the self-sustaining knowledge explosion of modern times is constantly hatching new technological dangers, and that there needs to be some new kind of response
Yes, to quibble just a bit, not just self sustaining, but also accelerating. The way I often put it is that we need to adapt to the new environment created by the success of the knowledge explosion. I just put up an article on the forum which explains further:
https://www.lesswrong.com/posts/nE4fu7XHc93P9Bj75/our-relationship-with-knowledge
from the whole of civilization? just from the intelligentsia
As I imagine it, the needed adaptation would start with intellectual elites, but eventually some critical mass of the broader society would have to agree, to some degree or another. I've been writing about his for years now, and can't actually provide any evidence that intellectual elites can lead on this, but who else?
It's unclear to me if you think you already have a solution.
I don't have a ten point plan or anything, just trying to encourage this conversation where ever I go. Success for me would be hundreds of intelligent well educated people exploring the topic in earnest together. That is happening to some degree already, but not with the laser focus on the knowledge explosion that I would prefer.
You're also saying that focus on AI safety is a mistake...
I see AI discussions as a distraction, as an addressing of symptoms, rather than addressing the source of X risks. If 75% of the time we were discussing the source of X risk, I wouldn't object to 25% addressing particular symptoms.
I'm attempting to apply common sense. If one has puddles all around the house every time it rains, the focus should be on fixing the hole in the roof. Otherwise one spends the rest of one's life mopping up the puddles.
There are in fact good arguments that AI is now pivotal to the whole process and also to its resolution.
I don't doubt AI can make a contribution in some areas, no argument there. But I don't see any technology as being pivotal. I see the human condition as being pivotal.
I'm attempting to think holistically, and consider man and machine as a single operation, with the success of that operation being dependent upon the weakest link, which I propose to be us. Knowledge development races ahead at an ever accelerating rate, while human maturity inches along at an incremental rate, if that. Thus, the gap between the two is ever widening.
Please proceed to engage from whatever perspective you find useful. What I hope to be part of is a long deliberate process of challenge and counter challenge which helps us inch a little closer to some useful truth.
Thanks again!
Replies from: Mitchell_Porter↑ comment by Mitchell_Porter · 2022-09-02T19:17:39.941Z · LW(p) · GW(p)
We believe AI is pivotal because we think it's going to surpass human intelligence soon. So it's not just another technology, it's our successor.
The original plan of MIRI, the AI research institute somewhat associated with this website, was to identify a value system and a software architecture for AI, that would still be human-friendly, even after it bootstrapped itself to a level completely beyond human control or understanding, becoming the metaphorical "operating system" in charge of all life on Earth.
More recently, given the rapidity of advances in the raw power of AI, they have decided that there just isn't time to solve these design problems, before some AI lab somewhere unwittingly hatches a superintelligent AI system that steamrolls the human race, not out of malice, but simply because it has goals that aren't sufficiently finetuned to respect human life, liberty, or happiness.
Instead, their current aim is to buy time for humanity, by using early superintelligent AI, to neutralize all other dangerous AI projects, and establish a temporary regime in which civilization can deliberate on what to do with the incredible promise and peril of AI and related technologies.
There is therefore some similarity with your own idea to slow things down, but in this scenario, it is to be done by force, and by using the dangerous technology of superintelligent AI, when it first appears. Continuing the operating system metaphor, this amounts to putting AI-augmented civilization into a "safe mode" before it can do anything too destructive.
This suggests a model of the future, in which there is a kind of temporary world government, equipped with a superintelligent AI that monitors everything everywhere, and which steps in to sabotage any unapproved technology that threatens to create unfriendly superintelligence. Ideally, this period lasts as long as it takes, for humanity's wise ones to figure out how to make fully autonomous superintelligence, something that we can safely coexist with. At that point the temporary world government can be permanently replaced with that self-governing planetary operating system.
You may be wondering, why rely on AI to restrain AI? Why not just have e.g. the UN Security Council declare that AI research worldwide will be frozen indefinitely, and use the existing tools of human governance to enforce that? The problem is that technological culture is decentralized and self-enhancing. In the short term, we might throttle the development of deep learning AI by restricting access to TPU chips worldwide. But you can also run the algorithms on sufficiently large networks of ordinary computers. And ultimately, you even have to worry about things like superintelligence achieved via neuron-hacking, polymeric nanocomputers, and so forth.
The premise is that the world is too out of control to stop everyone in the entire world from ever crossing the dangerous threshold. So instead, one must work towards an outcome whereby, the first ones across the threshold will use that power to slow things down for everyone else, while responsibly trying to figure out how to safely integrate that power into our world.
OK, that's a glimpse of how some people are thinking. AI is seen as the crux of everything, because it is at the hub of everything: it can control other technologies, it can discover new technologies, it can even replace us as the chief decision-making entity in the world. And it's really "AGI" (artificial general intelligence), and especially AGI that is more intelligent than human, which is the focus of all this concern, "Narrow AI" that just drives cars or recognizes faces has its own safety issues, but isn't as all-encompassing in its implications.
3 comments
Comments sorted by top scores.
comment by Vladimir_Nesov · 2022-08-30T03:56:23.846Z · LW(p) · GW(p)
the claim that there is a <1% probability of AI existential risk
That's a serious constraint [LW · GW]. What possible argument that's not literally a demonstration of a working AGI is going to do that to the epistemic state about a question this confusing? Imagining a future where AI is not an existential risk is easy (and there are many good arguments for it being more likely than one would expect, just as there are many good arguments for it being less likely than one would expect). But imagining a present where it's known to not be an existential risk with 99% probability (or 1% probability), despite not having already been built, doesn't work for me.
Maybe there is 0.1% probability (I sorta tried to actually assess the order of magnitude for this number) that in 15 years the world's state of knowledge builds up to a point where that epistemic state becomes thinkable (conditional on actual AGI not having been built). This would most likely require shockingly better alignment theory and expectation that less aligned AGIs can't (as in alignment-by-default) or won't be built first.
comment by Jeff Rose · 2022-08-30T04:55:48.281Z · LW(p) · GW(p)
The two questions you pose are not equivalent. There are critiques of AI existential risk arguments. Some of them are fairly strong. I am unaware of any which do a good job of quantifying the odds of AI existential risk. In addition, your second question appears to be asking for a cumulative probability. It's hard to see how you can provide that absent a mechanism for eventually cutting AI existential risk to zero...which seems difficult.