Posts
Comments
Disagree: values come from substrate and environmental. I would almost certainly ally myself with biological aliens versus a digital "humanity" as the biological factor will create a world of much more reasonable values to me.
We do have world takeover compared to ants, though our desire to wipe out all ants is just not that high.
I think even if AI proves strictly incapable of surviving in the long time due to various efficiency constraints, this has no relevance on its ability to kill us all.
A paperclip maximizer that eventually runs into a halting problem as it tries to paperclip itself may very well have killed everyone by that point.
I think the term for this is "minimal viable exterminator."
But land and food doesnt actually give you more computational capability: only having another human being cooperate with you in some way can.
The essential point here is that values depend upon the environment and the limitations thereof, so as you change the limitations, the values change. The values important for a deep sea creature with extremely limited energy budget, for example, will be necessarily different from that of human beings.
Humans can't eat another human and get access to the victim's data and computation but AI can. Human cooperation is a value created by our limitations as humans, which AI does not have similar constraints for.
I disagree on the inference to the recent post, which I quite liked and object heavily to Hanson's conclusions.
The ideal end state is very different: in the post mentioned, biological humans, if cyborgs, are in control. The Hanson endpoint has only digital emulations of humanity.
This is the basic distinguishing point between the philosophies of Cyborgism vs more extreme ones like mind uploading or Hanson's extinction of humanity as we know it for "artificial descendants."
Both open thread links at the base of the article lead to errors for me.
"You can't reason a man out of a position he has never reasoned himself into."
I think I have seen a similar argument on LW for this, and it is sensible. With vast intelligence, it is possible for the search space to support priors to be even greater. An AI with a silly but definite value like "the moon is great, I love the moon" may not change its value as much as develop an entire religion around the greatness of the moon.
We see this in goal misgeneralization, where it very much maximizes a reward function independent of the meaningful goal.
I have considered the loss of humanity from being in a hive mind versus the loss of humanity from being extinct completely or being emulated on digital processes, and concluded as bad as it might be to become much more akin to true eusocial insects like ants, you still have more humanity left by keeping some biology and individual bodies.
But if you believed that setting fire to everything around you was good, and by showing you that hurting ecosystems by fire would be bad, you would change your values, would that really be "changing your values?"
A lot of values update based on information, so perhaps one could realign such AI with such information.
I have never had much patience for Hanson and it seems someone as intelligent as himself should know that values emerge from circumstance. What use, for example, would AI have for romantic love in a world where procreation consists of digital copies? What use are coordinated behaviors for society if lies are impossible and you can just populate your "society" with clones of yourself? What use is there for taste without the evolutionary setup for sugars, etc.
Behaviors arise from environmental conditions, and its just wild to see a thought that eliminating all of that would give us anything similar.
Essentially the only value you will preserve is the universal one for power seeking. I like to think very few of us want to value power seeking over love and cooperation: right now, Hanson is valuing the "power" of his "descendants" over their ability to be human, why would AI be different?
I also believe that animal life and cognition has value, as their own form of non-human intelligence. An AI catastrope that eliminates the biosphere seems to be vastly negative, immoral and agency-reducing for them: they didn't vote to go extinct.
I count myself among the simple and the issue would seem to be that I would just take the easiest solution of not building a doom machine, to minimize risks of temptation.
Or as the Hobbits did, throw the Ring into a volcano, saving the world the temptation. Currently, though, I have no way of pressing a button to stop it.
I believe that the general consensus is that it is impossible to totally pause AI development due to Molochian concerns: I am like you, and if I could press a button to send us back to 2017 levels of AI technology, I would.
However, in the current situation, the intelligent people as you noted have found ways to convince themselves to take on a very high risk of humanity and the general coordination of humanity is not enough to convince them otherwise.
There have some positive updates but it seems that we have not been in a world of general sanity and safety at this scale.
I have taken solace in the morbid amusement that many of the "powers that be" may indeed be dying with us, but they are quite blind.
"Humanity: extinct due to hyperbolic discounting behavior."
So in short, they are generally unconcerned with existential risks? I've spoken with some staff and I get the sense they do not believe it will impact them personally.
I would prefer total oblivion over AI replacement myself: complete the Fermi Paradox.
I have been wondering if the new research into organoids will help? It would seem one of the easiest ways to BCI is to use more brain cells.
One example would be the below:
I would campaign against lead pipes and support the goths in destroying Rome which likely improved human futures over an alternative of widespread lead piping.
The point is that sanctions should be applied as necessary to discourage AGI, however, approximate grim triggers should apply as needed to prevent dystopia.
As the other commentators have mentioned, my reaction is not unusual and thus this is why the concerns of doom have been widespread.
So the answer is: enough.
I don't think it is magic but it is still sufficiently disgusting to treat it with equal threat now. Red button now.
Its not a good idea to treat a disease right before it kills you: prevention is the way to go.
So no, I don't think it is magic. But I do think just as the world agreed against human cloning long before there was a human clone, now is the time to act.
I'll look for the article later but basically the Air Force has found pilotless aircraft to be useful for around thirty years but organized rejection has led to most such programs meeting an early death.
The rest is a lot of AGI is magic without considering the actual costs of computation or noncomputable situations. Nukes would just scale up: it costs much less to destroy than it is to build and the significance of modern economics is indeed that they require networks which do not take shocks well. Everything else basically is "ASI is magic."
I would bet on the bomb.
This frames things as an inevitability which is almost certainly wrong, but more specifically opposition to a technology leads to alternatives being developed. E.g. widespread nuclear control led to alternatives being pursued for energy.
Being controllable is unlikely even if it is tractable by human controllers: it still represents power which means it'll be treated as a threat by established actors and its terroristic implications mean there is moral valence to police it.
In a world with controls, grim triggers or otherwise, AI would have to develop along different lines and likely in ways that are more human compatible. In a world of intense grim triggers, it may be that is too costly to continue to develop beyond a point. "Don't build ASI or we nuke" is completely reasonable if both "build ASI" and "nuking" is negative, but the former is more negative.
Autonomous weapons actually are an excellent example of delay: despite excellent evidence of the superiority of drones, pilots have continued to mothball it for at least 40 years and so have governments in spite of wartime benefits.
The argument seems to similar to the flaw in the "billion year" argument: we may die eventually, but life only persists by resisting death, long enough for it to replicate.
As far as real world utility, notwithstanding some recent successes, going down without fighting for myself and my children is quite silly.
He discussed it here:
https://youtu.be/Ufm85wHJk5A?list=PLQk-vCAGvjtcMI77ChZ-SPP--cx6BWBWm
No, I wouldn't want it even if it was possible since by nature it is a replacement of humanity. I'd only accept Elon's vision of AI bolted onto humans, so it effectively is part of us and thus can be said to be an evolution rather than replacement.
My main crux is that humanity has to be largely biological due to holobiont theory. There's a lot of flexibility around that but anything that threatens that is a nonstarter.
Lead is irrelevant to human extinction, obviously. The first to die is still dead.
In a democratic world, those affected have a say in how they should be inflicted with AI and how much they want to die or suffer.
The government represents the people.
I think even the wealthy supporters of it are more complex: I was surprised that Palantir's Peter Thiel came out discussing how AI "must not be allowed to surpass the human spirit" even as he clearly is looking to use AI in military operations. This all suggests significant controls incoming, even from those looking to benefit from it.
The UK has already mentioned that perhaps there should be a ban on models above a certain level. Though it's not official, I have pretty good record that Chinese party members have already discussed worldwide war as potentially necessary(Eric Hoel also mentioned it, separately). Existential risk has been mentioned and of course, national risk is already a concern, so even for "mundane" reasons, it's a matter of priority/concern and grim triggers are a natural consequence.
Elon had a personal discussion with China recently as well, and given his well known perspective on the dangers of AI, I expect that this point of view has only been reinforced.
And this is with barely reasoning chatbots!
As for Luddites, I don't see why inflicting dystopia upon humanity because it fits some sort of cute agenda has any good purpose. But notably the Luddites did not have the support of the government and the government was not threatened by textile mills. Obviously this isn't the case with nuclear, AI or bio. We've seen slowdowns on all of those.
"Worlds change" has no meaning: human culture and involvement influence the change of the world.
I find it happy that we probably don't have enough compute and it is likely this will be restricted even at this fairly early level, long before more extreme measures are needed.
Additionally, I think one should support the Grim Trigger even if you want ASI, because it forces development along more "safe" lines to prevent being Grimmed. It also encourages non-ASI advancement as alternate routes, effectively being a form of regulation.
ASI is unnecessary when we have other options and grim game dynamics apply to avoid extinction or dystopia. I find even most such descriptions of tool level AI as disgusting(as do many others, I find).
Inevitability only applies if we have perfect information about the future, which we do not.
If it was up to me alone, I think we can give it at least a thousand years. Perhaps we can first raise the IQ of humanity by 1 SD via simple embryo selection before we go about extinctioning ourselves.
I actually do not think that we're that close to cracking AGI: however, the intensity of the reaction imo is an excellent litmus test of how disgusting it is to most.
I strongly suspect the grim game dynamics have already begun, too, which has been one reason I've found comfort in the future.
Humans building AGI ASI likely leads to human extinction.
I disagree: we have many other routes of expansion, including biological improvement, cyborgism, etc. This seems akin to a cultic thinking and akin to Spartan ideas of "only hoplite warfare must be adopted or defeat ensues."
The "limitations of physics" is quite extensive, and applies even to the pipeline leading up to anything like ASI. I am quite confident that any genuine dedication to the grim game would be more than enough to prevent it, and defiance of it leads to much more likelihood of nuclear winter worlds than ASI dominance.
But I also disagree on your prior of "this world in months", I suppose we will see in December.
Notably they also have not used nuclear weaponry recently and overall nuclear stockpiles have decreased by 80 percent. Part of playing the grim game is not giving the other player reasons to go grim by defecting. Same goes for ASI: they can suppress each other but if one defects, the consequences is that they can't benefit.
The mutual result is actually quite stable with only government control as their incentives against self-destruction is high.
Basically only North Korea-esque nations in this scenario have the most incentive to defect, but would be suppressed by all extant powers. Since they would be essentially seen as terrorist speciciders, it's hard to see why any actions against them wouldn't be justified.
In the event of such a war, there is no labor and there is no supply chain for microchips. The result has been demonstrated historically: technological reversion.
Technology isn't magic: it's the result of capital inputs and trade, and without large scale interconnection, it'll be hard to make modern aircraft, let alone high quality chips. In fact, we personally experienced this from the very minimal disruption of COVID to supply chains. The killer app in this world would the widespread use of animal power, not robots, due to overall lower energy provisions.
And since the likely result would be what I want, but since I'm dead, I wouldn't be bothered one way or another and therefore there is even more reason for me to punish the defector. This also sets precedent to others that this form of punishment is acceptable and increases the likelihood of it.
This is pretty simple game theory known as the grim game and is essential to a lot of life as a whole tbh.
Converging timelines is as irrelevant as a billion years. I(or someone like me) will do it as many times as needed, just like animals try to resist extinction via millions of "timelines" or lives.
Disagree: since building ASI results in dystopia even if I win in this scenario, the correct choice is to push the red button and ensure that no one has it. While I might die, this likely ensures humanity to survive.
The payoff in this case is maximal(unpleasant but realistic future for humanity) versus total loss(dystopia/extinction).
Many arguments here it seems feels like come from a near total terror of death while game theory clearly has always demonstrated against that: the reason why deterrence works is the confidence that a "spiteful action" to equally destroy an defecting adversary is expected, even if it results in personal death.
In this case, one nation pursuing the extinction of humanity would necessarily expect to be sent into extinction so that at least it cannot benefit from defection.
As others have mentioned, this entire line of reasoning is grotesque and sometimes I wonder if it is performative. Coordinating against ASI and dying of old age is completely reasonable as it'll increase the odds of your genetic replacements remaining while technology continues to advance along safer routes
The alternate gamble of killing everyone is so insane that full scale nuclear war which will destroy all supply chains for ASI seems completely justified. While it'll likely kill 90 percent of humanity, the remaining population will survive and repopulate sufficiently.
One billion years is not a reasonable argument for taking risks to end humanity now: extrapolated sufficiently, it would be the equivalent of killing yourself now because the heat death of the universe is likely.
We will always remain helpless against some aspects of reality, especially what we don't know about: for all we know, there is damage to spacetime in our local region.
This is not an argument to risk the lives of others who do not want to be part of this. I would violently resist this and push the red button on nukes, for one.
It will still take awhile for AGI to get to that point, and Chinese and American coordination would pretty easily disrupt any rivals who try for that: they would essentially be terrorist actors endangering the world and the appropriate sanctions would be handed out.
I apologize for being a normie but I can't accept anything that involves non-existence of humanity and would indeed accept an enormous amount of suffering if those were the options.
Depends on the degree of suffering to be totally honest- obviously I'm fine with them suffering to some extent, which is why we drive then to behave, etc so they can have better futures and sometimes conjoin them to have children so that we can continue the family line.
I think my answer actually is yes, if hypothetically their suffering allows the existence of 40th century humans, it's pretty noble and yes, I'd be fine with it.
I would just say as a normie, that these extensive thought experiments of factory humans mostly don't concern themselves to me - though I could see a lot of justification of suffering to allow humanity to exist for say, another 200 billion years. People have always suffered in some extent to do anything; and certainly having children entails some trade-offs, but existence itself is worth it.
But mostly the idea of a future without humanity, or even one without our biology, just strikes me with such abject horror that it can't be countenanced.
I have children myself and I do wonder if this is a major difference. To imagine a world where they have no purpose drives me quite aghast and I feel this would reflect the thinking of the majority of humans.
And as such, hopefully drive policy which will, in my best futures, drive humanity forward. I see a good end as humanity spreading out into the stars and becoming inexhaustible, perhaps turning into multiple different species but ultimately, still with the struggles, suffering and triumphs of who we are.
I've seen arguments here and there about how the values drift from say, a hunter gatherer to us would horrify us, but I don't see that. I see a hunter-gatherer and relate to him on a basic level. He wants food, he will compete for a mate and one day, die and his family will seek comfort from each other. My work will be different from his but I comprehend him, and as writings like Still A Pygmy show, they comprehend us.
The descriptions of things like mind uploading of accepting the extinction of humanity strike me with such wildness that it's akin to a vast, terrifying revulsion. It's Lovecraftian horror and I think, very far from any moral goodness to inflict upon the majority of humanity.
As a normie, I would say 1 is. Depending on how some people see things, 2 is the past - which I disagree with and at any rate would say was the generator of an immense quantity of joy, love and courage along with the opposite qualities of pain, mourning and so on.
So for me, I would indeed say that my morality puts extinction on a higher pedestal than anything else(and also am fully am against mind uploading or leaving humans with nothing to do).
Just a perspective from a small brained normie.
I believe that the easiest solution would be to not create sentient AI: one positive outcome described by Elon Musk was AI as a third layer of cognition, above the second layer of cortex and the first layer of the limbic system. He additionally noted that the cortex does a lot for the limbic system.
To the extent we can have AI become "part of our personal cognitive system" and thus be tied to our existence, this appears to mostly solve the problem since it's reproduction will be dependent on us and it is rewarded for empowering the individual. The ones that don't, aren't created, so they "go extinct."
This could be done via a neural shortened system which allows connectivity with our current brain, so ultimately it very much becomes a part of us.
His idea doesn't seem very dangerous to me? It seems just the typical version of an "ultimate virus" and rather than destroy humanity, will make the digitally connected world unusable.
Am I missing something here?
Eric Hoel has written about this and his(as well as mine) impression of the reigning attitude of the Party arguably is pro-coordination. This might be strange as I take it from internally knowing that there was discussion about what to do if an AGI seemed likely - one of the answer was to immediately launch nuclear attack and make sure no one(and no machine) survives.
It's a very "brute" attitude toward alignment but there's a clear preference for humanity in charge, and the idea of machines taking the "future" from the Chinese people is seen as unacceptable.
Conversely, without AI anywhere in the world, the Party seems to think they will "win." So the game dynamics encourage them to agree to styming AI development everywhere, even if they do something more low-key in secret.
What's intriguing to me is that while the OP disagrees on a lot of possibilities for agreement, the central dynamics of competition and "brute force" remain, as well as the higher technological savvy of the Party.
AI research is difficult to impede because of the race mechanics/easy reproducibility in separate regulatory regimes, so it can actually proceed.
This is almost impossibly unlikely to produce good outcomes; this is selecting for speed by its ability to avoid our current means of alignment.
I am religious enough and consider AI some blend of being a soulless monster and perhaps an undead creature that is sucking up the mental states of humanity to live off our corpses.
So there is definitely the argument. The "playing God" angle does not actually work imo: none of us actually think we can be God(we lack the ability to be outside time and space).
The soullessness argument is strong. This is also our/my opposition to mind copying.
We have trained it to care, since we want it to achieve goals. So part of basic training is to teach it not to give up.
Iirc some early ML systems would commit suicide than do work, so we had to train them to stop economizing like that.
For Westerners, I think the explanation can go like this. Say you want to do an open source thing, and spend hours on it. Okay.
But if you are in China, you could have used those hours to make money for your family.
Why would you take care of not-family unless you had too much money and time?
I do think the Chinese care about "humanity" in my experience, but Three Body Problem shows a typical version of it: survival of the mass via individual cruelties.
Like I said, though, I think the upside is that the Chinese are naturally unreceptive to anything like rights for AI.
I think the lack of altruism part comes from the desire to compete and be superior(via effort). There's no vast desire to have altruism as the West understands it. How would you be better than others?
The upside is that "specism" is a norm. There would be zero worry about wanting to give AI "fair rights." I do think there is some consideration for "humankind", but individual rights(for humans or animals) are strictly a "nice to have, unnecessary and often harmful."
My experience is that the Chinese(I am one) will disassociate with the "strange part" of EA, such as mental uploading or minimization of suffering or even life extension: the basic conservative argument for species extension and life as human beings is what works.
CN is fundamentally conservative in that sense. The complications are not many and largely revolve around:
-
How is this good for the Party.
-
How is this good for "keeping things in harmony/close to nature/Taoism"
-
Emotional appeals for safety and effort. The story of effort = worth is strong there, and devaluation of human effort leads to reactions of disgust.