Should we postpone AGI until we reach safety?

I think it's obviously a bad idea to deploy AGI that has an unacceptably high chance of causing irreparable harm. I think the questions of "what chance is unacceptably high?" and "what is the chance of causing irreparable harm for this proposed AGI?" are both complicated technical questions that I am not optimistic will be answered well by policy-makers or government bodies. I currently expect it'll take serious effort to have answers at all when we need them, let alone answers that could persuade Congress. 

This makes me especially worried about attempts to shift policy that aren't in touch with the growing science of AI Alignment, but then there's something of a double bind: if the policy efforts are close to the safety research efforts, and so you're giving the best available advice to the policymakers, but you pay the price of backlash from AI researchers if they think regulation-by-policy is a mistake. If the two are distant, then the safety researchers can say their hands are clean, but now the regulation is even more likely to be a mistake. 

I think that trying to slow down research towards AGI through regulation would fail, because everyone (politicians, voters, lobbyists, business, etc.) likes scientific research and technological development, it creates jobs, it cures diseases, etc. etc., and you're saying we should have less of that. So I think the effort would fail, and also be massively counterproductive by making the community of AI researchers see the community of AGI safety / alignment people as their enemies, morons, weirdos, Luddites, whatever.

Also, I'm not sure how you can stop "AGI research" without also stopping "AI research" (and for that matter some fraction of neuroscience research too), because we don't know what research direction will lead to AGI.

If anti-AGI regulations / treaties were the right thing to do (for the sake of argument), the first step would be to get the larger AI community and scientific community thinking more about planning for AGI, and gradually get them on board and caring about the issue. Only then would you have a prayer of succeeding at the second step, i.e. advocating for such a regulation / treaty. But when you think about it, after you've taken the first step, do you really need the second step? :-P

Oh, and even if such a law / treaty passed, it seems like it might be unenforceable. There will always be a large absolute number of AI researchers who think the rule is stupid, and then they would all go move to the one random country that didn't ratify the treaty. Or maybe AGI would be invented in a secret military lab or whatever.

Just my off-the-cuff opinions :-P

I think there are two more general questions that need to be answered first.

1. How would we find out for sure whether there any tractable methods to put the brakes on a particular arm of technological progress?
2. What would be the tradeoffs of such a potent investigation into our civilizational capacity?

We clearly do have the capacity to do (1) to some extent:

• Religious activists have managed to slow progress in stem cell research, though the advent of iPSCs has created a way to bypass this issue to some extent.
• The anti-nuclear movement has probably helped slow down progress in nuclear power research, though ironically they don't seem to have slowed down research on nuclear bombs (I could be wrong here).
• Some people argue that the current structure of using cost-benefit analysis to allocate research funding does more harm than good, and thus it could be considered a decelerating force. But I'm not sure that's true, and even if it is, I'm not sure that applies to a field like AI with so many commercial purposes.

But these are clearly not the durable, carefully calibrated brakes we're talking about.

On the first argument, I replied that I think a non-AGI safety group could do this, and therefore not hurt the principally unrelated AGI safety efforts. Such a group could even call for reduction of existential risk in general, further decoupling the two efforts.

It sounds like you are suggesting that someone somewhere should do this. Who, and how? Because until there is a specific idea being put forward, I can say that pausing AGI would be good, since misaligned AGI would be bad. I don't know how you'd do it, but if choosing between two world, the one without misaligned AGI seems likely to be better.

But in my mind, the proposal falls apart as soon as you ask who this group is, and whether this hypothetical groups has any leverage or any arguments that would convince people who are not already convinced. If the answer is yes, why do we need this new group to do this, and would we be better off using this leverage to increase the resources and effort put into AI safety?

My feelings on this are analogous to my opinions on, say, legalization of drugs and prostitution. It's much easier to regulate how and where something is done, than whether it is done. There's a lot of room between "full speed ahead!" and "Stop everything!" Off the cuff, there's "Subsidize safety research and narrow AI research, and set strict guidelines for AGI research with monitoring and ethics rules akin to what we do for nuclear or medical research."

I don't think the title of this posts capture the actual question studied in it? Like, I don't think any safety researcher is not, in some sense, for halting AGI research. So the question in the title is trivial (at least in AI Safety). But the actual question debated between you and Daniel (as far as I can see from your post) is whether it's possible to implement it completely without too many adversarial consequences.

About that question, I agree completely with Daniel's position that it's just not a viable option. I see no vaguely probable scenario where the means to act on such a restriction exists.

One possible way to postpone AGI is that one party will reach world domination using powerful Narrow AI and then use this Narrow AI to implement strict control over AGI development. What such Narrow AI could be: some combination of AI-powered drones and Palantir-like targeting + effective control of minds via memes and social network surveillance. It doesn't sound nice and-or sexy, but we are half-way here. 

An alternative in AGI-preventing is a nuclear war against chips manufacturers and AI-labs and it is obviously worse. To clarify, I am against it, just mentioned as a bad alternative.

It's certainly desirable to postpone any ascension to power of entities (including humans) who will harm us.  The problem (for AI and for human power-mongers) is implementation.  I think I'm with Daniel that the attempt to interfere would fail, and would do additional harm along the way.

It's not clear to me that we have a choice on postponement. If company A refrains, then company B has an edge. If as a nation we constrain companies A and B, then country X will not, and gain an edge. And if country X and we sign a treaty, then country Y will have an edge. And if all countries refrain, then a criminal or terrorist group outside of all civilized nations will have an edge. 

The longer that the postponement occurs, the easier it will become for an entity that is outside of the agreements to create AGI on its own. The bonus value (power) gained by breaking the treaty grows greater as others refrain, reaching its maximum when everyone except the rulebreaker refrain.

All participants are in a dollar auction.

My response to counterargument 3 is summarized in this plot, for reference: https://ibb.co/250Qgc9

Basically, this would only be an issue if postponement cannot be done until risks are sufficiently low, and if take-off would be slow without postponement intervention.

Maybe one could just run the assembly lines in the AI factories at half speed, or even better, reduce the assembly quota of AI workers to half.

This will give workers more holidays, maybe even 3-4 day weekends, gets AI labor unions off your back and the AI factories are still cranking, so it's not like progress stops entirely.

Then again, putting a higher VAT on AI sales might be more practical.

The issue is a coherent policy proposal, but I'm sure brilliant regulatory minds like those that wrote legislation for defending us from the evils of UDP sockets and openssl would ride up to the task of ironing out the edges.

Overall though, your idea is brilliant, you're on the right path.

It seems like there is very little chance that you could politically stop work on AGI. Who would you appeal to - China, the U.S., and the United Nations would all have to agree? There just isn't anywhere near the political consensus necessary for it. The general consensus is that any risk is very small, that it's the sort of thing only a few weirdos on the internet worry about. That consensus would have to change before it makes any sense to ask questions like, should we postpone all AGI research indefinitely. I think we need to accept that worry about AGI is a fringe belief, and therefore we should pursue strategies that can have an impact despite it being a fringe belief.