Accuracy of arguments that are seen as ridiculous and intuitively false but don't have good counter-arguments
post by Christopher King (christopher-king) · 2023-04-29T23:58:24.012Z · LW · GW · 14 commentsThis is a question post.
Contents
Answers 18 jacob_cannell 13 Archimedes 12 dr_s 6 Lalartu 2 BrooksT 0 AnthonyC None 14 comments
I think one of the biggest reasons I am worried about AI doom is that there doesn't seem to be very good counter-arguments. Most of them are just bad, and the ones that aren't a bit vague (usually something along the lines of "humanity will figure it out" or "maybe LLMs will scale in nice ways").
However, I'm curious as to how accurate this heuristic is. My question: What examples in the past are there of "argument is widespreadly seen as ridiculous and intuitively false, but the argument is pretty solid and the counter-arguments aren't". (Sorry if that's a bit vague, use your best judgement. I'm looking specifically for examples that are similar to the AI x-risk debate.) And did they turn out true or false? Try to include reasons why the argument was so strong, a typical counter-argument, and the strongest counter-argument.
Please use spoiler text for the final answer so that I can try to predict it before seeing it!
Answers
The number of people who actually had the deep technical skills and knowledge to evaluate the risk of ignition of the atmosphere from nuclear weapons was very small, and near completely overlapped with the people developing the very same weapons of mass destruction that were the source of that risk.
The number of people who have the deep technical knowledge, skills, and talent necessary to correctly evaluate the actual risk of AGI doom is probably small, and probably overlaps necessarily with the people most capable of creating it.
↑ comment by Amalthea (nikolas-kuhn) · 2023-04-30T11:58:23.417Z · LW(p) · GW(p)
I'm curious how this fits into the context. Regardless of whether or not one believes it's true, doesn't it seem reasonable and intuitively right - so the opposite of what is asked for?
Replies from: AnthonyC↑ comment by AnthonyC · 2023-04-30T15:51:29.769Z · LW(p) · GW(p)
I think the argument that would have been seen as ridiculous by most in the nuclear weapons example is, "The right arrangement of (essentially, what look like) rocks and metals will not only make a big explosion, but could destroy all life on earth in seconds." The argument in favor (and eventual, correct argument against) were both highly technical and inaccessible. Also the people most involved in the deep technical weeds were both the ones capable of seeing the danger, and the ones needed to figure out if the danger was real or not.
Replies from: nikolas-kuhn↑ comment by Amalthea (nikolas-kuhn) · 2023-04-30T18:22:40.934Z · LW(p) · GW(p)
So it would be: Claim: A nuclear bomb could set the atmosphere on fire and destroy everything on earth Argument: Someone did a calculation. Counterargument: Clearly, that's absurd. Good Counterargument: Someone else did another calculation.
And I guess the analogy to AI applies foom/room a the bottom, where one can actually do calculations to at least in principle estimate some OOMs.
Historical examples of things that once sounded ridiculous but turned out to be true:
- Spherical Earth (vs Flat Earth)
- Heliocentrism (vs Geocentrism)
- Germ theory (vs e.g. Miasmatic theory)
- Evolution via natural selection
- Quantum mechanics
- Relativity
- Plate tectonics
- Black holes
It's harder to know what qualifies as false examples since they do (now) have good counterarguments, but maybe something like these:
- Phlogiston theory
- Vitalism
- Luminiferous aether
- Lamarckian evolution
- Cold fusion
- Steady State cosmology (vs Big Bang)
- Caloric theory
- Spontaneous generation
Examples of ideas with less certain status:
- String theory / quantum gravity / unified physics
- Multiverse hypothesis / simulation hypothesis
- Existence and nature of extraterrestrial life
- Nature of dark matter & dark energy
- Epigenetic roles in disease and inheritance
- Origins of life / abiogenesis / panspermia
- Nature of consciousness and reality
↑ comment by Christopher King (christopher-king) · 2023-04-30T12:55:30.914Z · LW(p) · GW(p)
Does spherical earth count? I couldn't find any sources saying the idea was seen as ridiculous, especially around the time that they actually discovered it was round via physical measurements.
Replies from: Archimedes↑ comment by Archimedes · 2023-04-30T17:25:06.332Z · LW(p) · GW(p)
There is indeed the Myth of the flat Earth that is a misconception about the beliefs of scholars in the Middle Ages and some scholars certainly understood the concept of a spherical Earth since at least Eratosthenes. I'm referring to earlier history like ancient Egyptian, Chinese, and pre-Socratic Greek cosmologies. Admittedly, it's not a great example since most of the debates about it are lost to history, and such debates wouldn't involve the same kind of reasoning and evidential standards we use today.
↑ comment by Linch · 2023-04-30T21:33:08.375Z · LW(p) · GW(p)
What's wrong with caloric theory?
Replies from: ChristianKl↑ comment by ChristianKl · 2023-04-30T23:16:44.861Z · LW(p) · GW(p)
It helps to look up what the term means: "The caloric theory is an obsolete scientific theory that heat consists of a self-repellent fluid called caloric that flows from hotter bodies to colder bodies. Caloric was also thought of as a weightless gas that could pass in and out of pores in solids and liquids."
Replies from: LinchMaybe the risk of nuclear war during the Cold War? This 1961 argument by Bertrand Russell is cogent enough to sound correct, it would have probably sounded like a pretty wild prediction to most people, and all considered, it was indeed kinda false, though modest steps along the lines of option 3 did happen (we didn't get a literal political union but we did get the fall of the Iron Curtain and globalization entangling the world into mutual economic interest, as well as the UN). Anyone betting on the status quo fallacy on a gut instinct against Russell there would have won the bet.
↑ comment by Christopher King (christopher-king) · 2023-04-30T12:17:15.636Z · LW(p) · GW(p)
Yeah that definitely seems very analogous to the current AI x-risk discourse! I especially like the part where he says the UN won't work:
Any pretended universal authority to which both sides can agree, as things stand, is bound to be a sham, like UN.
Do you know what the counter-arguments were like? I couldn't even find any.
Replies from: Jiro, dr_s↑ comment by dr_s · 2023-04-30T14:39:59.685Z · LW(p) · GW(p)
Do you know what the counter-arguments were like? I couldn't even find any.
Not really, though I expect there must have been some objections. I think digging in the 1950s-60s era and the way they talked about nuclear risk would probably be very instructive. The talk "The AI Dilemma" (look it up on YouTube if you haven't seen it already) even brings up specifically the airing of the TV movie "The Day After", and how it concluded with a panel discussion between experts on nuclear risk, including among others Henry Kissinger and Carl Sagan. The huge amount of rightful holy terror of nuclear war back in that day most likely worked, and led to enough people being scared enough of it that in the end we avoided it. Worryingly, it's now, far from the peak of that scare, that we start seeing "but maybe nuclear war wouldn't be that destructive" takes (many appeared around the beginning of the war in Ukraine).
↑ comment by Christopher King (christopher-king) · 2023-04-30T12:47:09.859Z · LW(p) · GW(p)
Oh, but one weakness is that this example has anthropic shadow [? · GW]. It would be stronger if there was an example where "has a similar argument structure to AI x-risk, but does not involve x-risk".
So like a strong negative example would be something where we survive if the argument is correct, but the argument turns out false anyways.
That being said, this example is still pretty good. In a world where strong arguments are never wrong, we don't observe Russell's argument at all.
Replies from: dr_s↑ comment by dr_s · 2023-04-30T14:35:02.967Z · LW(p) · GW(p)
Disagree about anthropic shadow, because the argument includes two possible roads to life, barbarism or a world government. If the argument was correct, conditioned upon being still alive and assuming that barbarism would lead to the argument to be lost, an observer still reading Russell's original text should see a world government with 100% probability. And we don't.
Replies from: christopher-king↑ comment by Christopher King (christopher-king) · 2023-04-30T14:43:28.632Z · LW(p) · GW(p)
Oh right, good point. I still think anthropic shadow introduces some bias, but not quite as bad since there was the world government out.
Replies from: dr_s↑ comment by dr_s · 2023-04-30T14:56:52.180Z · LW(p) · GW(p)
The idea of a bias only holds if e.g. what Russell considered 100% of all possibilities only actually constituted 80% of the possibilities: then you might say that if we could sample all branches in which an observer looks back at argument, they'd see the argument right with less-than-80% probability, because in a part of the branches in which either of those three options come to pass there are no observers.
But while that is correct, the argument is that those are the only three options. Defined as such, a single counterexample is enough to declare the argument false. No one here is denying that extinction or civilisation collapse from nuclear war have been very real possibilities. But the road we care about here - the possible paths to survival - turned out to be wider than Russell imagined.
Replies from: christopher-king↑ comment by Christopher King (christopher-king) · 2023-04-30T16:49:16.390Z · LW(p) · GW(p)
Yeah, Russell's argument is ruled out by the evidence, yes.
The idea of a bias only holds if e.g. what Russell considered 100% of all possibilities only actually constituted 80% of the possibilities
I'm considering the view of a reader of Russell's argument. If a reader thought "there is a 80% chance that Russell's argument is correct", how good of a belief would that be?
Because IRL, Yudkowsky assigns a nearly 100% chance to his doom theory, and I need to come up with the x such that I should believe "Yudkowsky's doom argument has a x% chance of being correct".
That Malthusianism is wrong (for predicting the future). Prior to the demographic transition, arguments in favor of this view could be basically summarized as "somehow it will be fine".
Doom skeptics have the daunting task of trying to prove a negative. It is very hard to conclusively prove anything is safe in a generalized, unconditional way.
As to arguments that were seen as intuitively false yet were solid, there is a huge class of human rights topics: racial and gender equality, gay rights, even going back to giving peasants the vote and democratic government. All of these were intuitively ridiculous, yet it's hard to make credible counter-arguments.
↑ comment by Christopher King (christopher-king) · 2023-04-30T16:55:20.706Z · LW(p) · GW(p)
There were some counter-arguments against democracy that seemed pretty good. Even the founding fathers were deeply worried about them. They aren't seen as credible today, but back then they were viewed as quite strong.
Replies from: BrooksTBiological evolution is the origin of species.
Simple mathematical laws govern the behavior of everything, everywhere, all the time.
Negative, imaginary, irrational, and transcendental numbers, as well as infinite sets and infinitesimals, are logically coherent and useful.
(Many of the above are things many people still don't believe).
The Roman Empire is large, powerful, and well organized, and will have no problem dealing with the new Christian religion.
↑ comment by Ilio · 2023-05-03T12:39:51.799Z · LW(p) · GW(p)
Did you notice:? You can take these statements and make the case it’s stupid, for one reasonable set of priors, or profound, for a different but somewhat possible set of priors. And that’s true for each negation. As well as for several forms of negation. Bohr called that deep truth, or something like that.
14 comments
Comments sorted by top scores.
comment by Quintin Pope (quintin-pope) · 2023-04-30T00:33:51.559Z · LW(p) · GW(p)
Not an answer to your question, but I think there are plenty of good counterarguments against doom. A few examples:
- My Objections to "We’re All Gonna Die with Eliezer Yudkowsky" [LW · GW]
- Counterarguments to the basic AI x-risk case [LW · GW]
- Where I agree and disagree with Eliezer [LW · GW]
- Evolution provides no evidence for the sharp left turn [LW · GW]
- Deceptive Alignment is <1% Likely by Default [LW · GW]
- Some of my disagreements with List of Lethalities [LW · GW]
- Evolution is a bad analogy for AGI: inner alignment [LW · GW]
- Contra Yudkowsky on AI Doom [LW · GW]
- Contra Yudkowsky on Doom from Foom #2 [LW · GW]
- A Contra AI FOOM Reading List
- Likelihood of discontinuous progress around the development of AGI
↑ comment by Christopher King (christopher-king) · 2023-04-30T01:05:23.779Z · LW(p) · GW(p)
To clarify, I'm thinking mostly about the strength of the strongest counter-argument, not the quantity of counter-arguments.
But yes, what counts as a strong argument is a bit subjective and a continuum. I wrote this post because of the counter-arguments I know I know of are strong enough to be "strong" by my standards.
Personally my strongest counter-argument is "humanity actually will recognize the x-risk in time to take alignment seriously, delaying the development of ASI if necessary", but even that isn't backed up by too much evidence (the only previous example I know of is when we avoided nuclear holocaust).
Replies from: liam-donovan-1↑ comment by Liam Donovan (liam-donovan-1) · 2023-04-30T03:29:23.624Z · LW(p) · GW(p)
What do you think are the strongest arguments in that list, and why are they weaker than a vague "oh maybe we'll figure it out"?
Replies from: christopher-king↑ comment by Christopher King (christopher-king) · 2023-04-30T12:36:15.992Z · LW(p) · GW(p)
Hmm, Where I agree and disagree with Eliezer [LW · GW] actually has some pretty decent counter-arguments, at least in the sense of making things less certain.
However, I still think that there's a problem of "the NN writes a more traditional AGI that is capable of foom and runs it".
↑ comment by avturchin · 2023-04-30T09:13:41.116Z · LW(p) · GW(p)
The strongest argument against AI doom I can imagine runs as follows:
AI can kill all humans for two main reasons: to (a) prevent a threat to itself and (b) to get human's atoms.
But:
(a)
AI will not kill humans as a threat before it creates powerful human-independent infrastructure (nanotech) as in that case, it will run out of electricity etc.
AI will also not kill humans after it creates nanotech, as we can't destroy nanotech (even with nukes).
Thus, AI will not kill humans to prevent the threat neither before, nor after nanotech, – so it will never happens for this reason.
(b)
Human atoms constitute 10E-24 of all atoms in the Solar system.
Humans may have small instrumental value for trade with aliens, for some kinds of work or as training data sources.
Even a small instrumental value of humans will be larger than the value of their atoms, as the value of atoms is very-very small.
Humans will not be killed for atoms.
Thus humans will not be killed either as a threat or for atoms.
But there are other ways how AI catastrophe can kill everybody: wrongly aligned AI performs wireheading, Singleton halts, or there will be war between several AIs. Each of this risk is not necessary outcome.But together they have high probability mass.
Replies from: Viliam↑ comment by Viliam · 2023-04-30T09:45:44.847Z · LW(p) · GW(p)
Other reasons to kill humans:
- they are not a big threat, but they are annoying (it costs resources to fix the damage they do)
- a side effect of e.g. changing the atmosphere
Also, the AI may destroy human civilization without exterminating all humans, e.g. by taking away most of our resources. If the civilization collapses because the cities and factories are taken over by robots, most humans will starve to death, but maybe 100000 will survive in various forests as hunters and gatherers, with no chance to develop civilization again in the future... that's also quite bad.
Replies from: avturchin↑ comment by avturchin · 2023-04-30T10:43:08.146Z · LW(p) · GW(p)
It all collapses to the (2) "atoms utility" vs "human instrumental utility." Preventing starvation or pollution effect for a large group of humans is relatively cheap. Just put all them on a large space station, may be 1 km long.
But disempowerment of humanity and maybe even Earth-destruction are far more likely. Even if we will get small galactic empire of 1000 stars, but will live there as pets devoted any power about Universe future, it is not very good outcome.
↑ comment by Ben Pace (Benito) · 2023-04-30T21:30:37.678Z · LW(p) · GW(p)
These don’t seem very relevant counterarguments, I think literally all are from people who believe that AGI is an extinction-level threat soon facing our civilization.
Perhaps you mean “>50% of extinction-level bad outcomes” but I think that the relevant alternative viewpoint that would calm someone is not that the probability is only 20% or something, but is “this is not an extinction-level threat and we don’t need to be worried about it”, for which I have seen no good argument for (that engages seriously with any misalignment concerns).
Replies from: christopher-king↑ comment by Christopher King (christopher-king) · 2023-04-30T22:39:17.633Z · LW(p) · GW(p)
Well, I was asking because I found Yudkowsky's model of AI doom far more complete than any other model of the long term consequences of AI. So the point of my original question is "how frequently is a model that is far more complete than it's competitors wrong?".
But yeah, even something as low as 1% chance of doom demands very large amount of attentions from the human race (similar to the amount of attention we assigned to the possibility of nuclear war).
(That said, I do think the specific value of p(doom) is very important when deciding which actions to take, because it effects the strategic considerations in the play to your outs [LW · GW] post.)
comment by Shmi (shminux) · 2023-04-30T01:02:07.937Z · LW(p) · GW(p)
there doesn't seem to be very good counter-arguments
you might be mindkilled by the local extinctionists. There are plenty of good arguments (not necessarily correct or or universally persuasive), at least good enough to be very much uncertain of the validity of either side, unless you spend thousands of hours diving into it professionally.
Replies from: benjamincosman, dr_s↑ comment by benjamincosman · 2023-04-30T10:59:47.778Z · LW(p) · GW(p)
you might be mindkilled by the local extinctionists
+1(agreement) and -1(tone): you are correct that good arguments against doom exist, but this way of writing about it feels to me unnecessarily mean to both OP and to the 'extinctionists'
Replies from: shminux↑ comment by Shmi (shminux) · 2023-04-30T21:06:56.965Z · LW(p) · GW(p)
I think (anti-)extinctionism is a better term than notkilleveryoneism. I agree that "mindkill" is a bit stronger connotation than necessary, though. My annoyance with a one-sided view matching Eliezer's in many of the newbie posters comes through.