Posts
Comments
So people who say “brain preservation [or cryonics] doesn’t work” are either confused or >>making a very strong claim about neuroscience (that molecular structure doesn’t encode >>identity/memories) and/or future technology (that we can know with certainty what future >>technology won’t be able to do)
That is not entirely true. Some people who say "cryonics doesn't work" mean "identity is irretrievably lost when the brain activity stops, and in the best case you will have a different person with the same memory and personality traits". Since that argument doesn't give any testable predictions, it cannot be disproved.
Amodei’s general argument is this:
"my basic prediction is that AI-enabled biology and medicine will allow us to compress the progress that human biologists would have achieved over the next 50-100 years into 5-10 years."
This may be correct, but his estimate of what is expected to be achieved in 100 years without AI is likely wildly overoptimistic. In particular, his argument for doubling of lifespan is just an extrapolation from past increase in life expectancy, which is ridiculous because progress in extending maximum human lifespan so far is exactly zero.
Cell line being immortal doesn't prove that immortal brain is possible any more than microbe strain being immortal.
This chain of logic is founded on an assumption that these technologies are possible, which I find highly dubious. If an (aligned) superintelligence is built, and we ask it for life extension, the most probable answer would be that biological immortality (and all stuff requiring nanorobots) is just plain impossible, and brain uploading wouldn't help because your copy is not you.
No, that is not realistic. Bacteria described in the article don't really eat iron, they just make corrosive chemicals as methabolic waste. They rely on other sources of energy (sulfates or organic compounds). Metal-eating bacteria (those which derive energy by reducing metals) exist but require metals dissolved in water, eating solid metals doesn't work chemically.
Generally I think Eliziers definition of weak pivotal act doesn't include civilization collapse, because there are multiple obvious ways which don't kill all humans.
Well, the more precise phrase would be "fertility decline was not caused by the invention of new birth control technologies". It is totally possible for a society to have below replacement fertility using only birth control methods available since pre-industrial era.
If birth control hasn’t been enabling fertility rates to decline, then what has?
Rising women status contributed more than everything else combined.
However, the increasing availability of the birth control pill, other contraception methods, and the legalization of abortion during the 60s and 70s (in the US) are notable >for contributing to the declining fertility rates in the US.
That was a continuation of trend which started more than a century before that, after temporary baby boom reversal ended.
No, the opposite doesn’t usually happen. For all of human history, higher fertility memes have tended to outlast lower fertility memes.
Now of course, the last 200 years are exceptional, since many lower fertility memes have overpowered higher fertility memes.
The reason that happened is that communication became much easier. So it is reasonable to expect that low fertility memes will generally win for as long as the world remains interconnected.
there’s also many high fertility memes and memeplexes that still have very high fertility rates, such as the fundamentalist Muslims, the Amish, and Ultra-Orthodox Jews.
There were many more highly religious (and fertile) communities in the past. So the default is to expect that they will follow the same path like say Quebec.
I’ve written a list of things that could be done to boost Western fertility rates.
This list looks rather US-centric. Many countries, for example in eastern Europe don't have these specific problems but have low fertility anyway. So most likely this would not help much.
There are several dubious assumptions there.
The first is that fertility decline is caused mostly by birth control. The problem is, it began long before birth control became widespread. A century ago, most developed nations had TFR somewhere between 2 and 2.5.
The second is that high fertility memes are durable. But usually exactly the opposite happens, "cultural change causes lower fertility" is the same as "high fertility memes lose". That happens with religious groups the same way - Mormons used to have much higher fertility, and now they don't.
As for adapting for deceases - that is a survivor bias. For example, several dozen amphibian species are believed to be wiped out by fungal infection in last decades. It is rather unlikely that humanity will go extinct that way, if anything there are some isolated tribes. But industrial civilization can collapse due to low population long before natural selection would cause fertility to rise again. With TFR 1.2 (like in Italy now) population would drop below 100 million in about three centuries.
Anti tank FPV drones? Almost certainly not long term as they’re more expensive than ATGMs,
That is not true at all, anti-tank fpv cost is about 1/100 of a Javelin missile. It is not obvious how much autonomous guidance would add to a drone cost, but probably less than 10000%.
The point about chaff is that a regular size sniper rifle bullet can't contain it in any significant quantity. Smalest existing chaff shells are for 23mm cannons, and a drone carrying ~20mm cannon has to be rather large.
In general, lessons from the Russo-Ukrainian war are not very relevant for a "state of the art" conflict, because both sides have weak air forces. It is like watching two armies fighting with bayonets because they are out of ammo and concluding that you should arm your soldiers with swords and shields.
Also, this makes many assumptions which are dubious (like, sniper drones aren't anywhere close to practical use, and it is not clear if they are viable), but also some which are strictly false:
- Bullets can't carry enough chaff to "surround" a tank
- Lasers can destroy artillery shells (which are made of steel) in flight, there is no practical way to harden a light drone against them.
A lot of people just don't believe it is possible, and for good reasons. Life extension as a scientific field was around for about a century, with exactly zero results so far. And these "ASI can grant immortality" stories usually assume nanotechnology, which is most likely fundamentally impossible.
If life extension was actually available, I think attitude would be different.
I disagree that "forever is really long time" in this context. To delay AI forever requires delaying it until industrial civilization collapse (from resource depletion or whatever other reason). That means 200-300 years, more likely that 50000.
I think this is not true at all. Modern video games, films or cookies aren't any more addictive compared to those 20 years ago. As for reasons of fertility drop - remember that this is not the first time! USA fertility was barely above replacement in 1930s, and some European countries were below at that time.
Again, that some estimates are given in papers doesn't mean they are even roughly correct. But if they are - then no, that scenario is not suicide. There are some nations now which have lower GDP per capita than USA had two centuries ago.
As for defense - well, that definitely wouldn't be a problem. Who and why will be willing to invade a big and very poor country, leaders of which claim they still have some nukes in reserve?
The first claim is true - but ruling a third world nation is still better than being dead and ruling nothing. If the leadership has Eliezer-level conviction that AI would kill everybody, then the choice is clear. The second isn't - destroying the ability to build AI is much easier, so the reason for abiding the treaty is not "we all die" but rather "we become much poorer and don't get the AI anyway".
I think all these claims are incorrect. First, estimates of damage from nukes are very likely to be hugely (and intentionally) overblown, the same way as pre-WWII estimates for strategic bombing were off by an order of magnitude. Second, even ignoring that, current arsenals (only warheads deployed on ICBMs, other are irrelevant in this scenario) are not sufficient for counter-population strike. Destroying large cities does not destroy the nation. Third and most importantly, leadership ordering the first strike will surely survive! They just move to some remote location before, and then claim that the other side attacked first.
The biggest problem with proposing tanks is convincing military leadership that they need them. They didn't expect trench warfare at all (and yes, some writers predicted it, and nobody believed them).
The winning strategy exists: don't build a large fleet and don't invade Belgium. The problem is, there is basically no way to convince German leadership to follow the first part, and the second part alone doesn't reliably keep Britain neutral.
Telling the leaders anything like "war will be long and bloody" will not work. They will not believe you, and you have no proof.
The only way I see is to let either side to win without world war, by breaking either Entente or Triple Alliance. Like, if British king or Russan tsar is killed by some French radicals.
This is based on assumption that defense is much easier than offense. This is not true, in fact in WWI attacker's and defender's losses were usually close (for example, ~140k vs ~160k KIA at Verdun).
These things are either unlikely to succeed or just not that important.
Including dath ilan is especially strange. It is a work of fiction, it is not really appealing, and it is not realistic (that is, not internally consistent) either.
That story of Mongol conquests is simply not true. Horse archer steppe nomads existed for many centuries before Mongols, and often tried to conquer their neighbors, with mixed success. What happened in the 1200s is that Mongols had a few exceptionally good leaders. After they died, Mongols lost their advantage.
Calling states like Khwarazmian or Jin Empire "small duchies" is especially funny.
As I understand, topicstarters claim is that civilization is not a chaotic system, and any temporary disturbances don't affect long-term trajectory. Weather is a chaotic system.
It is always possible to say in retrospect that whatever happened was inevitable. The problem is, a world where individual actions don't matter that much should be a predictable world. And ours very much isn't.
Defaults only matter due to reputation. But stopping a weird practice which no one else does doesn't really damage reputation.
So what happens when in 50 years the government just stops paying, without passing the law? Buyers of these instruments don't care about that law, so they will not object much, and there will be no reputation loss.
80% for AGI solving aging is very optimistic. Even just one single possibility, that people who decide what values should AGI have happen to be anti-immortalist is imo >20%.
560M is not from the paper, it is from the post. The paper has graph with births per year stabilizing at around 5M, which can correspond to different population sizes depending on child mortality, but all of them are unrealistically low.
I don't see any reason how the population can stabilize at the arbitrary level of 560 million. Either it will start to increase again at some point (due to cultural shift, natural selection or some other reason), or it will decline until the collapse of the industrial civilization. But in that case, with no universal education anymore, the demographic transition will reverse, and the population starts growing again to limits imposed by pre-industrial agriculture (which is somewhere around 2 to 3 billion, not half a billion).
Or, suppose there is an intelligent alien civilization that has been around for much longer than humans. Would you expect that they have definitely industrialized in some form? Or would it depend on the particular geology of their planet?
I think the answers are "no" and "yes". No fossil fuels, no cheap energy, no industrial revolution ever. There might be a different way through hydroelectric power, but that requires cheap copper.
Didn’t we pretty much always know it was going to come from one or a few giant companies or research labs?
Not at all. In that ancient era of early 2000s, the common position was that insight was far more important than resources. So the usual scenario was some bright guy building AI in his basement.
There is a counterargument to claims 6 and 7: there is only one known warm-blooded species with longer lifespan than humans (the bowhead whale), and it is large and has slow metabolism for a mammal. In contrast, there are plenty of cold-blooded long-lived species. So, it is entirely plausible that humans live about as long as possible for given metabolic level.
On Earth, yes. In a world where animals have projectile weapons, no.
There is a threshold where intelligence becomes much more useful, and this threshold is an ability to make better weapons than other animals have. In a world where this is not possible at all (for example, animals have zerg-level natural weapons, and there is no metal to make guns), having human-level intelligence is just not worth downsides from a big brain.
Because ability to believe whatever most people around believe was net beneficial. Also, the situation when going to war means risking your life and getting nothing valuable in return is rather modern.
That Malthusianism is wrong (for predicting the future). Prior to the demographic transition, arguments in favor of this view could be basically summarized as "somehow it will be fine".
RLHF is a trial-and-error approach. For superhuman AGI, that amounts to letting it kill everybody, and then telling that this is bad, don't do it again.
The Milgram experiments demonstrated Didn't it fall victim to the replication crisis? I have read somewhere that with different groups outcomes of those experiments are wildly different.
I think this scenario is not even remotely realistic. If things really go this way (which is far from granted), the government will massively expand police and other security services (and they will have money for that due to AI productivity gains). When a large percentage of population are cops, riots aren't that big problem.
I think it doesn't work this way, because jobs and housing are not constant. First, if the entire economy shrinks due to lower working population, there would be less well-paid jobs. Second, housing is expensive in places where there are jobs, and it is likely that jobs would concentrate in smaller number of cities as population shrinks.
Imagine for a second what the reversed version of those authoritarian and dystopian efforts would even look like. Realize that this too would be and is in the young adult dystopia book section. Also realize it has not happened, at least not in a long time.
Romanian dictator Nicolae Ceaușescu did exactly that. He was overthrown and executed, and this may be one of reasons why no other dictator tried similar policies so far.
My claim is different - that there is no defined threshold for significance, but on the spectrum from useless to world-changing some technologies which looked very promising decades ago still lie closer to lower end. So it is possible that in 2053 AI products would be about as important as MRI scanners and GMO crops in 2023.
I think that it is wrong. If instead of dropping nukes on mostly wooden cities they used them against enemy troops (or ships, or even cities which aren't built out of bamboo), the conclusion would be that a nuke is a not that powerful and cost-inefficient weapon.
As for "significant impact" - what impact counts as "significant"? Here are some technologies which on my opinion had no significant impact so far:
- human spaceflight
- superconductors
- genetic engineering
It is totally possible that AI goes into the same bag.
I think the defining feature of "weak pivotal act" idea was that it should be safe due to its weakness. So, any pivotal act that depends on aligned AGI (and would fail catastrophically if it is not aligned) is not weak.
I assumed that ice layer is supposed to be a few feet thick, and given figures are just for illustration that that amount of ice is trivial to make. If the plan really is to build an artificial glacier hundreds of feet thick, that creates a different set of problems, the first being that described structure wouldn't do it. Depending on temperature and wind speed, ice will either be carried away by wind, form an ice hill that would grow until it blocks nozzles, or accumulate on scaffolding until it collapses under its weight.
The problem with heavy iceboat is that its weight has to be distributed evenly on numerous skates, because otherwise skates that are more heavily loaded dig deeper and friction increases drastically. Such design was never built.
Your calculation of expenses relies on three assumptions: that this is an end-to-end route, that it takes 120 hours, and that it takes one pilot to drive an iceship (of this size and in these conditions). All of these are wrong. As for refrigeration - a much larger fraction of cargo types doesn't tolerate freezing.
I have an opposite impression. "Alignment" is usually interpreted as "do whatever a person who gave the order expected", and what author calls "strong alignment" is aligned AGI ordered to implement CEV.
All of these are simply wrong.
First, ice accumulated over terrain would not be flat. You can search how Alaskan glaciers look like, no way an ice ship can move on that. It is not necessary to pave the road - the problem is that to make an even surface, huge amount of ground has to be moved.
Second, yes, no recorded iceboat carried more than few tons of cargo, including crew.
Third, crew pay is around 5% of operating cost of a container ship. Even if it takes 50 iceships to replace it (due to higher speed and shorter way), then crew pay alone would more than double total operating cost. As for capital cost, lots of small ships are more expensive than one big ship.
Some historians claim that pre-industrial workers had much shorter working hours then those common in the 19 century. If that is true, then 8-hour workday is more a return to the historical norm rather than "progress" strictly speaking. Maybe that's why there was widespread support for it, and there isn't much for 4-hour workday now.
The main reason USA (and other nuclear powers) don't use nuclear blackmail is that it would end no-nuclear-proliferation regime. "Every state that can make nukes has them" is the natural word state, keeping non-proliferation requires effort.