Some Thoughts on Singularity Strategies

I was informed by Justin Shovelain that recently he independently circulated a document arguing for "IA first", and that most of the two dozen people he showed it to agreed with it, or nearly so.

I'm a bit surprised there hasn't been more people arguing (or at least stating their intuition) that "AI first" is the better strategy.

But I did find that Eliezer had written an argument explaining why he chose the "AI first" strategy in Artificial Intelligence as a Positive and Negative Factor in Global Risk (pages 31-35). Here's the conclusion from that section:

I would be pleasantly surprised if augmented humans showed up and built a Friendly AI before anyone else got the chance. But someone who would like to see this outcome will probably have to work hard to speed up intelligence enhancement technologies; it would be difficult to convince me to slow down. If AI is naturally far more difficult than intelligence enhancement, no harm done; if building a 747 is naturally easier than inflating a bird, then the wait could be fatal. There is a relatively small region of possibility within which deliberately not working on Friendly AI could possibly help, and a large region within which it would be either irrelevant or harmful. Even if human intelligence enhancement is possible, there are real, difficult safety considerations; I would have to seriously ask whether we wanted Friendly AI to precede intelligence enhancement, rather than vice versa.

I do not assign strong confidence to the assertion that Friendly AI is easier than human augmentation, or that it is safer. There are many conceivable pathways for augmenting a human. Perhaps there is a technique which is easier and safer than AI, which is also powerful enough to make a difference to existential risk. If so, I may switch jobs. But I did wish to point out some considerations which argue against the unquestioned assumption that human intelligence enhancement is easier, safer, and powerful enough to make a difference.

I think that some organization should be seriously planning how to leverage possible uploading or intelligence improvement technologies for building FAI (e.g. try to be the first to run an accelerated uploaded FAI research team; or develop better textbooks on FAI theory that improve the chances that future people, uploaded or with improved intelligence, get it right), and tracking new information that impacts such plans. Perhaps SIAI should form a team that works on that, or create another organization with that mission (right now it doesn't look like a top priority, but it could be, so this deserves some thought; it will be a clear priority in another 20 years, as relevant technologies move closer, but by then it might be too late to do some specific important-in-retrospect thing).

We humans seem at best just barely smart enough to build a superintelligent UFAI.

I haven't unpacked the intuition, but I feel humans are pretty far over the threshold of smart enough to build general AI given no giant-setback/existential catastrophes and no significant intelligence enhancement within the next 500 years. In the last 500 years we gained probability theory, in the last 100 utility theory, in the last 50 formal causality theory, and it seems we're making some progress on new decision theories now. (I don't claim to know if those are the most important theoretical inventions or anything, but they seem likely to be very important). And the progress we've made on mechanical integration of sensory input from the world feels like it's not going to be the bottleneck... so give me conservatively another 1 fundamental breakthrough per century and I'm not comfortable at all saying after 5 of those humans would be just barely smart enough to build UFAI.

Did you have a much shorter time horizon, or do you know what other thing it is that fuels your just-barely-smart-enough intuition?

We humans seem at best just barely smart enough to build a superintelligent UFAI.

Humans seem just barely smart enough to do many things, especially in the period when doing those things is just becoming possible.

Wouldn't it be surprising that the intelligence threshold for building UFAI and FAI turn out to be the same?

Probably not, since you use knowledge to build things, not just intelligence, and knowledge accumulates. For example, humans can build (invent) both windmills and space shuttles, and lower intelligence would probably bar humans from building (inventing) windmills.

We humans seem at best just barely smart enough to build a superintelligent UFAI. Wouldn't it be surprising that the intelligence threshold for building UFAI and FAI turn out to be the same?

I think people who would contest the direction of this post (probably Eli and Nesov) would point out that if humanity is over the intelligence threshold for UFAI, the economic-political-psychological forces would drive it to be built in a timeframe of few decades. Anything that does not address this directly will destroy the future. Building smarter humans is likely not fast enough (plus who says smarter humans will not be driven by the same forces to build UFAI?).

Well, let me take one more step back here, and ask the question, why do we want to have a singularity at all?

I realize that might cause a knee jerk reaction from some people, but think about it for a second; you have to know what your goals are before trying to figure out what strategy to use to get there.

If the answer is something like "So we can cure old age, live much longer, happier lives, develop a post scarcity economy, colonize space, prevent environmental collapse, avoid existential risks, and expand into the universe", then I wonder; what if IA is sufficient to do all of that without developing a GAI at all (at least in the near future)? I'm not saying that a IA human is going to be as smart as a GAI computer; but what if a group of IA humans with a 250 IQ or so are smart enough to accomplish whatever our goals are?

It's not obvious to me that creating super smart people would have a net positive effect because motivating them to decrease AI risk is itself an alignment problem. What if they instead decide to accelerate AI progress or do nothing at all?

My own feeling is that the chance of success of of building FAI, assuming current human intelligence distribution, is low (even if given unlimited financial resources), while the risk of unintentionally building or contributing to UFAI is high. I think I can explicate a part of my intuition this way: There must be a minimum level of intelligence below which the chances of successfully building an FAI is negligible. We humans seem at best just barely smart enough to build a superintelligent UFAI. Wouldn't it be surprising that the intelligence threshold for building UFAI and FAI turn out to be the same?

What will construct advanced intelligent machines is slightly less advanced intelligent machines, in a symbiotic relationship with humans. It doesn't much matter if the humans are genetically identical with the ones that barely managed to make flint axe heads - since they are not working on this task alone.