AI Policy?

And it seems like, as long as the US government is aware of an AI project, there will be a lot it will be able to do to shut the project down if desired. For foreign projects, there will be the possibility of a Stuxnet-style attack, though the government might be reluctant to do that against a nuclear power like China or Russia (or would it?)

The US government doesn't even manage to shut down Chinese cyber army groups that attack US targets.

David Brin believes that high speed trading bots are a high probability route to human indifferent AI. If you agree with him, then laws governing the usage of high speed trading algorithms could be useful. There is a downside in terms of stock liquidity, but how much that will affect overall economic growth is still a research area.

One of the great things about software development is that any reasonably bright human with the right training (or self-training, given available documentation) can do it. It doesn't require special equipment; your average PC and an Internet connection is quite enough.

(And it's gotten cheaper and easier. Back when I learned BASIC on the Commodore 64, that computer cost $1400 in today's dollars, plus another $700 for the floppy disk drive and $100 for a box of blank disks. Today, kids can learn to code on a Raspberry Pi that costs $35, plus $5 for an SD card; so it's cheaper by a factor of fifty-five ... and thousands of times more powerful.)

A consequence of this is that governments cannot prevent people from doing AI research in secret.

Would it help to require ethics classes for computer-science students? Maybe — but given the political climate, I would expect them to focus more on anti-piracy, pro-espionage, and telling students that they would be horrible, evil monsters if they wrote the next PGP, Tor, BitTorrent, or Bitcoin.

Many people seem unsure of the size of the effects of political advocacy even in the short run. So this seems a little like saying, here's an NP-complete problem and for good measure take another problem in an even harder complexity class. It makes you want to throw up your hands and run.

My current guess is that AI policy is not a good idea because the ground is shifting underneath us as we walk--by the time a self-sufficient machine civilization got even close, human politics (if we were still human) would already be very different.

Edit: So when we think about the US government right before AI, it's like ancient Rome or Han China and nuclear weapons. There might well not be a US government.

And there, given the current status quo for how these things work in the US, my guess is that if the government ever became convinced that an AI project was dangerous, they would find some way to shut it down citing "national security" and basically that would work.

Who do you mean when you say "the government"?

From a policy perspective, my current hypothesis is that we would have to use the approach of solving the problems in approximately the order they are likely to come up, modified by the likelihood of them happening and the severity they would have if they occured.

As an example: It doesn't help to write policy for how to keep a super intelligent AI from using a hacked communication channel that is likely to take over everything that is connected to the internet in 2053 and then turn earth into unfriendly computronium in 2054, if a country hands over nuclear launch capabilities to an dumb AI in 2023 and because the dumb AI doesn't have a Petrov module, there is a nuclear war in 2024 that kills billions. We would probably want to write the nuclear policy first, to give ourselves time to write the communication channel policy, unless the nuclear war risk was lower then the communication channel risk by a big margin.

Also, I suppose there are two different types of AI risk.

Type A: The AI itself, through either accident, malice, poor design, causes the existential risk. Type B: Humans institute an AI as part of a countermeasure to help avoid an existential risk (Imagine automated security at a Nuclear Weapons facility). Other Humans find a flaw in the AI's abilities and cause the existential risk themselves. (Bypassing the automated security and stealing a nuclear weapon)

My understanding is that most people are generally discussing type A, with a nod to type B, since they are related. As an example, it is possible to design an AI, attempt to avoid Type B by giving it sufficient capabilities, and then accidentally cause a Type A problem because you forgot to convey what you considered to be common sense. (An AI stops a nuclear weapon from being stolen from the facility by detonating the nuclear weapon: It doesn't exist, and so can't be stolen from the facility!)

But in terms of specific policies, I am not sure what the criteria above would find as the most important policy to advance first so I am not sure what to suggest.