Do Earths with slower economic growth have a better chance at FAI?

To stay with the lingo (also, is "arguendo" your new catchphrase?): There are worlds in which slower economic growth is good news, and worlds in which it's not. As to which of these contribute more probability mass, that's hard -- because the actual measure would be technological growth, for which economic growth can be a proxy.

However, I find it hard to weigh scenarios such as "because of stagnant and insufficient growth, more resources are devoted to exploiting the remaining inefficiencies using more advanced tech" versus "the worldwide economic upswing caused a flurry of research activities".

R&D, especially foundational work, is such a small part of worldwide GDP that any old effect can dominate it. For example, a "cold war"-ish scenario between China and the US would slow economic growth -- but strongly speedup research in high-tech dual-use technologies.

While we often think "Google" when we think tech research, we should mostly think DoD in terms of resources spent -- state actors traditionally dwarf even multinational corporations in research investments, and whether their investements are spurned or spurred by a slowdown in growth (depending on the non-specified cause of said slowdown) is anyone's guess.

Related questions:

1) Do Earths with dumber politicians have a better chance at FAI?

2) Do Earths with anti-intellectual culture have a better chance at FAI?

3) Do Earths with less missionary rationalism have a better chance at FAI?

4) How much time should we spend pondering questions like (1)-(3)?

5) How much time should we spend pondering questions like (4)?

I don't have an answer for the question, but I note that the hypothetical raises the possibility of an anthropic explanation for twenty-first century recessions. So if you believe that the Fed is run by idiots who should have , consider the possibility that in branches where the Fed did in fact , the world now consists of computronium.

I find this especially compelling in light of Japan's two "lost decades" combined with all the robotics research for which Japan is famous. Obviously the anthropic hypothesis requires the most stagnation in nations which are good at robots and AI.

I responded here.

So far as I can tell, the most likely reason we wouldn't get Friendly AI is the total serial research depth required to develop and implement a strong-enough theory of stable self-improvement with a possible side order of failing to solve the goal transfer problem.

I'm curious that you seem to think the former problem is harder or less likely to be solved than the latter. I've been thinking the opposite, and one reason is that the latter problem seems more philosophical and the former more technical, and humanity seems to have a lot of technical talent that we can eventually recruit to do FAI research, but much less untapped philosophical talent.

Also as another side note, I don't think we should be focusing purely on the "we come up with a value-stable architecture and then the FAI will make a billion self-modifications within the same general architecture" scenario. Another possibility might be that we don't solve the stable self-improvement problem at all, but instead solve the value transfer problem in a general enough way that the FAI we build immediately creates an entirely new architecture for the next generation FAI and transfer its values to its creation using our solution, and this happens just a few times. (The FAI doesn't try to make a billion self-modifications to itself because, just like us, it knows that it doesn't know how to safely do that.) (Cousin_it make a similar comment earlier.)

In all I can see three arguments for prioritizing the value transfer problem over the stable self-improvement problem: 1) the former seems harder so we need to get started on it earlier; 2) we know the former definitely needs to be solved whereas the latter may not need to be; 3) the former involves work that's less useful for building UFAI.

(On the main topic of the post, I've been assuming, without having put too much thought into it, that slower economic growth is good for eventually getting FAI. Now after reading the discussions here and on Facebook I realize that I haven't put enough thought into it and perhaps should be less certain about it than I was.)

See here and here.

This position seems unlikely to me at face value. It relies on a very long list of claims, and given the apparently massive improbability of the conjunction, there is no way this consideration is going to be the biggest impact of economic progress:

1. The most important determinant of future welfare is whether you get FAI or UFAI (this presupposes a relatively detailed model of how AI works, of what the danger looks like, etc.)
2. This will happen quite soon, and relevant AI work is already underway.
3. The main determinant of FAI vs. UFAI is whether an appropriate theoretical framework for goal-stability is in place.
4. As compared to UFAI work, the main difficulty for developing such a framework for goal-stability is the serial depth of the problem.
5. A 1% boost in economic activity this year has a non-negligible effect on the degree of parallelization of relevant AI work.

I don't see how you can defend giving any of those points more than 1/2 probability, and I would give the conjunction less than 1% probability. Moreover, even in this scenario, the negative effect from economic progress is quite small. (Perhaps a 1% increase in sustained economic productivity makes the future 0.1% worse if this story is true, and each year of a 1% increase in economic productivity makes the future 0.002% worse?)

So on balance it seems to me like this would say that a 1% increase in economic productivity would make the future 0.00002% worse? That is a ridiculously tiny effect; even if you couldn't see any particular reason that economic progress helped or hurt I think your prior should expect other effects to dominate, and you can use other considerations to get a handle on the sign. I guess you think that this is an underestimate, but I would be interested to know where you disagree.

I would guess that the positive effect from decreasing the cumulative risk of war alone are several orders of magnitude higher than that. I know you think that a world war that killed nearly everyone might be positive rather than negative, but in that case slowing down economic progress would still have positive effects that are orders of magnitude larger than the effect on FAI parallelization.

Even without doing any calculations, it is extraordinarily hard to imagine that the difference between "world at war" and "world at peace" is less than the difference between "world with slightly more parallelization in AI work" and "world with slightly less parallelization;" almost everyone would disagree with you, and as far as I can tell their reasons seem better than yours. Similarly, it is hard to imagine that the interaction between generational turnover and economic activity wouldn't swamp this by several orders of magnitude.

The "normal view" is expressed by GiveWell here. Eliezer's post above can be seen as a counterpoint to that. GiveWell does acknowledge that "One of the most compelling cases for a way in which development and technology can cause harm revolves around global catastrophic risks..."

Some Facebook discussion here including Carl's opinion:

https://www.facebook.com/yudkowsky/posts/10151665252179228

Any thoughts about what sort of society optimizes for insight into difficult problems?

Something to take into account:

Speed of economic growth affects the duration of the demographic transition from high-birth-rate-and-high-death-rate to low-birth-rate-and-low-death-rate, for individual countries; and thus affects the total world population.

A high population world, full of low-education people desperately struggling to survive (ie low on Maslow's hierarchy), might be more likely to support making bad decisions about AI development for short term nationalistic reasons.

Great Stagnation being good news

Per Thiel the computer industry is the exception to the Great Stagnation, so not sure how much it really helps. You can claim that building flying cars would take resources away from UFAI progress, though intelligence research (i.e. machine learning) is so intertwined with every industry that this is a weak argument.

A key step in your argument is the importance of the parallel/serial distinction. However we already have some reasonably effective institutions for making naturally serial work parallelizable (e.g. peer review), and more are arising. This has allowed new areas of mathematics to be explored pretty quickly. These provide a valve which should mean that extra work on FAI is only slightly less effective than you'd initially think.

You could still think that this was the dominant point if economic growth would increase the speed of both AI and AI safety work to the same degree, but I think this is very unclear. Even granting points (1)-(3) of Paul's disjunction, it seems like the most important question is the comparative relationship between the elasticity of AI work with economic growth and the elasticity of FAI work with economic growth.

I currently incline to think that in general in a more prosperous society, proportionally more people are likely to fund or be drawn into work which is a luxury in the short term, and AI safety work fits into this category (whereas AI work is more likely to have short term economic benefits, so get funding). However, this question could do with more investigation! In particular I think it's plausible that the current state of EA movement growth means we're not in a general position. I haven't yet thought carefully through all of the ramifications of this.

Nick Beckstead made comments on some related questions in this talk: http://intelligence.org/wp-content/uploads/2013/07/Beckstead-Evaluating-Options-Using-Far-Future-Standards.pdf

By the way, I can guess at some of the source of disagreement here. The language of your post (e.g. "I wish I had more time, not less, in which to work on FAI") suggests that you imagine that the current people working on FAI will account for a reasonable proportion of the total AI safety work which will be done. I think it more likely that there will be a lot more people working on it before we get AI, so the growth rate dominates.

What kinds of changes to the economy would disproportionately help FAI over UFAI? I gather that your first-order answer is "slowing down", but how much slower? (In the limit, both kinds of research grind to a halt, perhaps only to resume where they left off when the economy picks up.) Are there particular ways in which the economy could slow down (or even speed up) that would especially help FAI over UFAI?

I would also expect socialist economic policies to increase chances of successful FAI, for two reasons. First, it would decrease incentives to produce technological advancements that could lead to UFAI. Second, it would make it easier to devote resources to activities that do not result in a short-term personal profit, such as FAI research.

It's easy to see why rationalists shouldn't help develop technologies that speed AI. (On paper, even an innovation that speeds FAI twice as much as it speeds AI itself would probably be a bad idea if it weren't completely indispensable to FAI. On the other hand, the FAI field is so small right now that even a small absolute increase in money, influence, or intellectual power for FAI should have a much larger impact on our future than a relatively large absolute increase or decrease in the rate of progress of the rest of AI research. So we should be more interested in how effective altruism impacts the good guys than how it impacts the bad guys.)

I'm having a harder time accepting that discouraging rationalists from pulling people out of poverty is a good thing. We surely don't expect EA as currently practiced by most to have a significant enough impact on global GDP to reliably and dramatically affect AI research in the near term. If EA continues to flourish in our social circles (and neighboring ones), I would expect the good reputation EA helps build for rationality activists and FAI researchers (in the eyes of the public, politicians, charitable donors, and soft-hearted potential researchers) to have a much bigger impact on FAI prospects than how many metal rooves Kenyans manage to acquire.

(Possibly I just haven't thought hard enough about Eliezer's scenarios, hence I have an easier time seeing reasoning along his lines discouraging prosaic effective altruism in the short term than having a major effect on the mid-term probability of e.g. completely overhauling a country's infrastructure to install self-driving electric cars. Perhaps my imagination is compromised by absurdity bias or more generally by motivated reasoning; I don't want our goals to diverge either.)

As it stands, I wouldn't even be surprised if the moral/psychological benefits to rationalists of making small measurable progress in humanitarian endeavors outweighed the costs if those endeavors turned out to be slightly counterproductive in the context of FAI. Bracketing the open problems within FAI itself, the largest obstacles we're seeing are failures of motivation (inciting heroism rather than denial or despair), of imagination (understanding the problem's scope and our power as individuals to genuinely affect it), and of communication (getting our message out). Even (subtly) ineffective (broadly) conventional altruistic efforts seem like they could be useful ways of addressing all three of those problems.

Motivated reasoning warning: I notice that want it to be the case that economic growth improves the FAI win rate, or at least doesn't reduce it. I am not convinced of either side, but here are my thoughts.

Moore's Law, as originally formulated, was that the unit dollar cost per processor element halves in each interval. I am more convinced that this is serially limited, than I am that FAI research is serially limited. In particular, semiconductor research is saturated with money, and FAI research isn't; this makes it much more likely to have used up any gains from parallelism. This reasoning suggests that when slowing down or speeding up the overall clock, Moore's Law is one of the few things that won't be affected.

I'm also not fully convinced that RGDP growth and the amount of hack-AI research are monotonically connected. The limiting input is top programmers; economic progress does not translate straightforwardly into creating them, and stronger economies both give them more autonomy (so less likely to divert into something trivial like making people click on ads), and stronger economies let them retire more easily. These effects push in opposite directions, and it's not at all clear which is stronger.

Regardless of what happens to the economy, the first- and second-wave Internet generations will take power on a fixed schedule. That's way more kaboomy than economic growth is.

This question is quite loaded, so maybe it's good to figure out which part of the economic or technological growth is potentially too fast. For example, would the rate of the Moore's law matching the rate of the economic growth, say, 4-5% annual, instead of exceeding it by an order of magnitude, make a difference?

This question is broader than just AI. Economic growth is closely tied to technological advancement, and technological advancement in general carries great risks and great benefits.

Consider nuclear weapons, for instance: Was humanity ready for them? They are now something that could destroy us at any time. But on the other hand, they might be the solution to an oncoming asteroid, which could have destroyed us for millions of years.

Likewise, nanotechnology could create a grey goo event that kills us all; or it could lead to a world without poverty, without disease, where we all live as long as we like and have essentially unlimited resources.

It's also worth asking whether slowing technology would even help; cultural advancement seems somewhat dependent upon technological advancement. It's not clear to me that had we taken another 100 years to get nuclear weapons we would have used them any more responsibly; perhaps it simply would have taken that much longer to achieve the Long Peace.

In any case, I don't really see any simple intervention that would slow technological advancement without causing an enormous amount of collateral damage. So unless you're quite sure that the benefit in terms of slowing down dangerous technologies like unfriendly AI outweighs the cost in slowing down beneficial technologies, I don't think slowing down technology is the right approach.

Instead, find ways to establish safeguards, and incentives for developing beneficial technologies faster. To some extent we already do this: Nuclear research continues at CERN and Fermilab, but when we learn that Iran is working on similar technologies we are concerned, because we don't think Iran's government is trustworthy enough to deal with these risks. There aren't enough safeguards against unfriendly AI or incentives to develop friendly AI, but that's something the Singularity Institute or similar institutions could very well work on. Lobby for legislation on artificial intelligence, or raise funds for an endowment that supports friendliness research.

One countervailing thought: I want AGI to be developed in a high trust, low-scarcity, social-pyshoclogical context, because that seems like it matters a lot for safety.

Slow growth enough and society as a whole becomes a lot more bitter and cutthroat?

Relative to UFAI, FAI work seems like it would be mathier and more insight-based, where UFAI can more easily cobble together lots of pieces. This means that UFAI parallelizes better than FAI. UFAI also probably benefits from brute-force computing power more than FAI. Both of these imply, so far as I can tell, that slower economic growth is good news for FAI; it lengthens the deadline to UFAI and gives us more time to get the job done.

Forgive me if this is a stupid question, but wouldn't UFAI and FAI have identical or near-identical computational abilities/methods/limits and differ only by goals/values?

The Great Stagnation has come with increasing wealth and income disparity.

This is to say: A smaller and smaller number of people are increasingly free to spend an increasing fraction of humanity's productive capacity on the projects they choose. Meanwhile, a vastly larger number of people are increasingly restricted to spend more of their personal productive capacity on projects they would not choose (i.e. increasing labor hours), and in exchange receive less and less control of humanity's productive capacity (i.e. diminishing real wages) to spend on projects that they do choose.

How does this affect the situation with respect to FAI?

I think we're past the point where it matters. If we had a few lost decades in the mid-twentieth century, maybe, (and just to be cognitively polite here, this is just my intuition talking) the intelligence explosion could have been delayed significantly. We are just a decade off from home computers with >100 teraflops, not to mention the distressing trend of neuromorphic hardware (Here's Ben Chandler of the SyNAPSE project talking about his work on HackerNews)With all this inertia, it would take an extremely large downturn to slow us now. Engineering a new AI winter seems like a better idea, though I'm confused about how this could be done. Perceptrons discredited connectionist approaches for a surprisingly long time, perhaps a similar book could discredit (and indirectly defund) dangerous branches of AI which aren't useful for FAI research--but this seems unlikely, though less so than OP significantly altering economic growth either way.

Any ideas to make FAI parallelize better? Or make there be less UFAI resources without reducing economic growth?

Could you be confusing the direction of causality here? I suspect that technological growth tends to lead to economic growth rather than the reverse.

Robin Hanson came to a similar conclusion here, although his concern was emulations.

I'm not convinced that slowing economic growth would result in FAI developing faster than UFAI and I think your main point of leverage for getting an advantage lies elsewhere (explained). The key is obviously the proportion between the two, not just slowing down the one or speeding up the other, so I suggest a brainstorm to consider all of the possible ways that slow economic growth could also slow FAI. For one thought: do non-profit organizations do disproportionately poorly during recessions?

The major point of leverage, I think, is people, not the economy. How that would work:

• Selfish people have a goal that divides them. If your goal is to be an AI trillionaire, you have to start your own AI company. If your goal is to save the world from AI, your goal benefits if you co-operate with like-minded people as much as possible. The aspiring trillionaire's goals will not be furthered by co-operating with competitors, and the world-saver's goals will not be furthered by competing with potential allies. This means that even if UFAI workers outnumber FAI workers 10 to 1, it's possible for the FAI effort to unite that 10% and conquer the 90% with sheer numbers, assuming that the 90% is divided into fragments smaller than 10% each.

• An organization made up of altruistic people might be stronger and more efficient than an organization made up of selfish people. If FAI workers are more honest or less greedy, this could yield practical benefits like a reduction in efficiency-draining office politics games, a lower risk of being stolen from or betrayed by those within the organization, and being able to put more money toward research because paychecks may not need to be as large. Also, people who are passionate about their goal derive meaning from their work. They might work harder or have more moments of creative inspiration than people who are working simply to get more money.

• The FAI movement is likely to attract people who are forward-thinking and/or more sensible, realistic or rational. These people may be more likely to succeed at what they do than those attracted to UFAI projects.

• People look down on those who hurt others for profit. Right now, the average person probably does not know about the UFAI risk and how important it is that people should not work on such projects. UFAI is a problem that would affect everyone, so it's likely that the average person will eventually take interest in it and shun those who work on UFAI. If they're accurately informed about which projects should be considered UFAI, I think it would seriously deter AI workers from choosing UFAI jobs.

• It has been said by people knowledgeable about geniuses that they typically don't prioritize money as highly as others. Unfortunately, there isn't enough solid research on the gifted population (let alone geniuses) but if it's true that money is not the most important thing to a genius, you may find that a disproportionate number of geniuses would prioritize working on FAI over taking the biggest possible paycheck.

Unlike the economy, all of these are things that MIRI can take action on. If the FAI movement can take advantage of any of these to get more workers, more good minds, or more good people than the UFAI projects, I want to see it happen.

I think this depends on how much you think you have the ability to cash in on any given opportunity. eg, you gaining a ton of money is probably going to help the cause of FAI more than whatever amount of economic growth is generated helps bring about AI. So basically either put your money where your theories are or don't publicly theorize?

This might come down to eugenics. Imagine that in 15 years, with the help of genetic engineering, lots of extremely high IQ people are born, and their superior intelligence means that in another 15 or so years (absent a singularity) they will totally dominate AGI software development. The faster the economic growth rate the more likely that AGI will be developed before these super-geniuses come of age.

For FAI to beat UAI, sufficient work on FAI needs to be done before sufficient work on AI is done.

If slowing the world economy doesn't change the proportion of work done on things, then a slower world economy doesn't increase the chance of FAI over UAI, it merely delays the time at which one or the other happens. Without specifying how the worlds production is turned down, wouldn't we need to assume that EY's productivity is turned down along with the rest of the world's?

If we assume all of humanity except EY slows down, AND that EY is turning the FAI knob harder than the other knobs relative to the rest of humanity, then we increase the chance of FAI preceding UAI.

I'm not sure that humane values would survive in a world that rewards cooperation weakly. Azathoth grinds slow, but grinds fine.

To oversimplify, there seem to be 2 main factors that increase cooperation, 2 basic foundations for law. Religion and Economic growth. Of this, religion seems to be far more prone-to-volatility. It is possible to get some marginally more intelligent people to point out the absurdity of the entire doctrine and along with the religion, all the other societal values collapse.

Economic growth seems to be a far more promising foundation for law as the poor and the low in status can be genuinely assured that they will get a small share of a growing pie. If economic growth slows down too much, it's back to values ingrained by evolution.

If we were perpetually stuck at Roman Empire levels of technology, we'd never have to worry about UFAI at all. That doesn't make it a good thing.

If you are pessimistic about global catastrophic risk from future technology and you are most concerned with people alive today rather than future folk, slower growth is better unless the effects of growth are so good that they outweigh time discounting.

But growth in the poorest countries is good because it contributes negligibly to research and national economies are relatively self-contained, and more growth there means more human lives lived before maybe the end.

Also, while more focused efforts are obviously better in general than trying to affect growth, there is (at least) one situation where you might face an all-or-nothing decision: voting. I'm afraid the ~my solution here~ candidate will not be available.

Katja comments here:

As far as I can tell, the effect of economic growth on parallelization should go the other way. Economic progress should make work in a given area less parallel, relatively helping those projects that do not parallelize well.

As long as we are in a world where billions are still living in absolute poverty, low economic growth is politically radicalizing and destabilizing. This can prune world branches quite well on its own, no AI needed. Remember, the armory of apocalypse is already unlocked. It is not important which project succeeds first, if the world gets radiatively sterilized, poisoned, ect before either one succeeds. So, no. Not helpful.

Before you can answer this question, I think you have to look at a more fundamental question, which is simply: why are so few people interested in supporting FAI research/ concerned about the possibility of UFAI?

It seems like there are a lot of factors involved here. In times of economic stress, short-term survival tends to dominate over long-term thinking. For people who are doing long-term thinking, there are a number of other problems that many of them are more focused on, such as resource depletion, global warming, ect; even if you don't think those are as big a threat to our future as UFAI is, if we move towards solving them it will free up some of that "intellectual energy" for other problems. Also, pessimistic thinking about the future in general, and a widespread pessimism about our odds of ever advancing to the point of having true GAI, is also likely to be a factor.

In general, I think that the better the economy is, the more people see the world getting better, and the more other problems we are able to solve, the more likely people are to start thinking about long-term risks, such as UFAI, nanotechnological weapons, ect. I don't think the difference between a better economy or a worse economy is likely to make much different of the rate of technological change (graphs seem to demonstrate that, for example, the Great Depression didn't slow down technology much), but it is likely to have a major impact on the kind of short-term thinking that people have during a crisis and the kind of long-term thinking that people are more able to do when the short-term issues seem to be under control.

To be clear, the idea is not that trying to deliberately slow world economic growth would be a maximally effective use of EA resources and better than current top targets; this seems likely to have very small marginal effects, and many such courses are risky. The question is whether a good and virtuous person ought to avoid, or alternatively seize, any opportunities which come their way to help out on world economic growth.

It sounds like status quo bias. If growth was currently 2% higher, should the person then seize on growth-slowing opportunities?

One answer: it could be that any effort is likely to have little success in slowing world growth, but a large detrimental effect on the person's other projects. Fair enough, but presumably it applies equally to speeding growth.

Another: an organisation that aspires to political respectability shouldn't be seen to be advocating sabotage of the economy.

For economic growth, don't focus on the total number. Ask about the distribution. Increasing world wealth by 20% would have minimal impact on making people's lives better if that increase is concentrated among the top 2%. It would have huge impact if it's concentrated in the bottom 50%.

So if you have a particular intervention in mind, ask yourself, "Is this just going to make the rich richer, or is it going to make the poor richer?" An intervention that eliminates malaria, or provides communication services in refugee camps, or otherwise assists the most disadvantaged, can be of great value without triggering your fears.

If a good outcome requires that influential people cooperate and have longer time-preferences, then slower economic growth than expected might increase the likelihood of a bad outcome.

It's true that periods of increasing economic growth haven't always lead to great technology decision-making (Cold War), but I'd expect an economic slowdown, especially in a democratic country, to make people more willingly to take technological risks (to restore economic growth), and less likely to cooperate with or listen to or fund cautious dissenters, (like people who say we should be worried about AI).

Will 1% and 4% RGDP growth worlds have the same levels of future shock? A world in which production doubles every 18 years and a world in which production doubles every 70 years seem like they will need very different abilities to deal with change.

I suspect that more future shock would lead to more interest in stable self-improvement, especially on the institutional level. But it's not clear what causes some institutions to do the important but not urgent work of future-proofing, and others to not- it may be the case that in the more sedate 1% growth world, more effort will be spent on future-proofing, which is good news for FAI relative to UFAI.

Eliezer, this post reeks of an ego trip.

"I wish I had more time, not less, in which to work on FAI"... Okay, world, lets slow right down for a while. And you, good and viruous people with good for technological or economic advancement: just keep quiet until it is safe.

Am I correct in assuming you have an ethical injunction against the obvious solution of direct sabotage? 'Cause other people don't but would probably just slow you down if you're already trying it.

Edit: direct sabotage does not mean "blow things up", it means "subtly trigger crabs-in-a-bucket effects via clever voting" or something.

We know economic growth cannot be sustainable long term (do the math), if we were to think longer term (Rather than the current "race to AI" thinking), we would still need to decide when to stop pursuing economic growth. Maybe since we have to stop growth somewhere, stopping it before AI is a good idea.

I think the nearest deadline is our own lifespans. FAI is no good if you're dead before it comes along. Also cryonics requires fast economic growth.