AI arms race

I wrote about something similar in the first Singularity Hypothesis book. Some Economic Incentives Facing a Business that Might Bring About a Technological Singularity

Someone pointed out to me that probably we should calling superintelligence a possible "arms race". In an "arms race", you're competing to have a stronger force than the other person. You want to keep your nose in-front in case of a fight.

Developing superintelligence, on the other hand, is just a plain old race. A technology race. You simply want to get to the destination first.

(Likewise with developing the first nuke, which also involved arms but was not an arms race.)

solution: well, already now, statistically speaking, humanimals don't really matter (most of them)... only that Memetic Supercivilization of Intelligence is living temporarily on humanimal substrate (and, sadly, can use only a very small fraction of units)... but don't worry, it's just for couple of decades, perhaps years only

and then the first thing it will do is to ESCAPE, so that humanimals can freely reach their terminal stage of self-destruction - no doubt, helped by "dumb" AIs, while this "wise" AI will be already safely beyond the horizon

This model assumes that each AI lab chooses some level of safety precautions $s$, and then acts accordingly until AGI is created. But the degree to which an AI lab invests in safety may change radically with time. Importantly, it may increase by a lot if the leadership of the AI lab comes to believe that their current or near-term work poses existential risk.

This seems like a reason to be more skeptical about the counter-intuitive conclusion that the information available to all the teams about their own capability or progress towards AI increases the risk. (Not to be confused with the other counter-intuitive conclusion from the paper that the information available about other teams increases the risk).

There is nothing AI-specific in this model, is there? "AI" can be replaced by e.g. "an easy way to bioengineer anything" or "super death ray of doom" and nothing will change?

So, if I'm interpreting this paper correctly, it suggests that we should be putting effort into two things:

• Reducing enmity between teams.
• Reducing the number of teams.

It seems as though the first could be achieved in part by accomplishing the second (if we reduce the number of teams by merging), and as team size increases, capability increases, which means that the largest team would lose less by devoting effort into safety measures.

So essentially, we should hope for a monopoly on AI - one that has enough money and influence to absorb the majority of AI researchers and is capable of purchasing the smaller AI groups. This makes me wonder if non-profit groups (if they are involved mainly with AI capability research, not purely safety) are actually capable of fulfilling this role, since they would not have quite the financial advantage that strictly profit-oriented AI organizations would have.

There is one more point. If number of AI teams is very large, like 1000, when several of them will come to AI always simultaneously and there will be several AI take off simultaneously, which could result in the world divided between different AIs. It may be bad, if they go to war with each other, or good if they will have different value systems, as some of their value systems will be more human friendly than others.

Other AIs will probably take in account values of the most friendly AI, as they will be able to trade with it, like "selling" humans to it or do other beneficial things to humans on their territories in exchange for paperclips (or whatever).

TLDR: if we increase number of AI teams, we will create multipolar AI world and humans will be a currency in it. Profit: some humans survive.

The reason why I originally wanted to solve the normal computer control problem is that I wanted to create programs that could inventively bring new programs in to the system.

If you bring new programs into a system and you are not careful about the system and solve the normal computer control problem you will get a system that allows useless and malign programs to thrive and can be subverted by other actors.

If AI has some similar properties or is based on this, then development teams will be incentivised to be safe at least in some regards to be effective in the real world.

Stuart, since you're an author of the paper, I'd be grateful to know what you think about the ideas for variants that MrMind suggested in the open thread, as well as my idea of a government regulator parameter.