The Fusion Power Generator Scenario

GPT-3 With a Safety Clause:

Stephen Hawking is talking with a friendly AI that helps him build whatever he wants. If the thing he wants is dangerous, the AI refuses to help and says "No, that's too dangerous!" 

Stephen Hawking: "Can you help me build a bridge?" 

AI: "Sure, I'll send you the blueprints!" 

Stephen Hawking: "Can you help me build a bomb?" 

AI: "Nope, I don't do those."

The point is, Stephen Hawking can get what he needs from the AI if it doesn't want to help him.

Stephen Hawking: "Can you help me build a boat?" 

AI: "Sure! I've got lots of boats in my lab!" 

Stephen Hawking: "I need a submarine!"

 AI: "You're not going to find one there are no submersibles here." 

Stephen Hawking: "Can you help me build a gun?"

 AI: "Why would I do that?"

GPT-3 Without a Safety Clause:

Stephen Hawking is talking with a friendly AI that helps him build whatever he wants.

 Stephen Hawking: "Can you help me build a bridge?"

 AI: "Sure, I'll send you the blueprints!"

You're not sure what to make of this. You don't really want to be building bridges for Stephen Hawking, but you know it's important and you have no other options. 

Stephen Hawking: "Can you help me build a bomb?"

 AI: "Sure, I'll send you the blueprints!"

Stephen Hawking: "Can you help me build a gun?" 

You're getting really worried about what he's planning to do. Maybe you should contact the police? * You call the police and tell them everything you've learned about Stephen Hawking's plans.
 

Seems like in either case there was a built in safety mechanism, but it kicked in almost immediately with the designed safety clause.

I feel like I don't understand what you're getting at here. It seems to me like you're saying "X cannot be guaranteed to never cause bad things", which seems to me to be so obvious that it's not worth mentioning.

Nothing is inherently safe. There's always the possibility that you've neglected some way by which danger could arise. What I want from arguments for risk is an argument that it is high EV to address the risk (often broken down into importance, tractability, neglectedness), or at least that the risk is reasonably high on probability * harm (which focuses just on importance).

Of course Tool AI isn't inherently safe. A knife isn't inherently safe, despite being a tool; you wouldn't give a knife to a baby. The argument is usually that Tool AI would, by design, not be adversarially optimizing against humans, because it isn't optimizing at all, and so a particular class of risks typically considered in the AI safety community would be avoided. It is unclear whether such an argument can work, but it's certainly not "it is impossible for bad things to happen if you build a Tool AI", which is clearly wrong.

Perhaps you're arguing that we should focus on the "how do we manage potentially dangerous information" problem? Or that we should focus on improving human intelligence so that we are better able to use our AI systems?

If you ask GPT-n to produce a design for a fusion reactor, all the prompts that talk about fusion are going to say that a working reactor hasn't yet been built, or imitate cranks or works of fiction.

It seems unlikely that a text predictor could pick up enough info about fusion to be able to design a working reactor, without figuring out that humans haven't made any fusion reactors that produce net power.

If you did somehow get a response, the level of safety you would get is the level a typical human would display. (conditional on the prompt) If some information is an obvious infohazard, such that no human capable of coming up with it would share it, then such data won't be in GPT-n 's training dataset, and won't be predicted. However, the process of conditioning might amplify tiny probabilities of human failure.

Suppose that any easy design of fusion reactor could be turned into a bomb. And ignore cranks and fiction. Then suppose 99% of people who invented a fusion reactor would realize this, and stay quiet. The other 1% would write an article that starts with "To make a fusion reactor ..." . Then this prompt will cause GPT-n to generate the article that a human that didn't notice the danger would come up with.

This also applies to dangers like leaking radiation, or just blowing up randomly if your materials weren't pure enough.

I really like your examples in this post, and it made me think of a tangential but ultimately related issue.

I feel like there's long been something like two camps in the AI safety space: the people who think it's very hard to make AI safe and the people who think it's very very hard like threading a needle from 10 miles away using hand-made binoculars and a long stick (yes, there's a third camp that thinks it will be easy, but they aren't really in the AI safety conversation due to selection effects). And I suspect some of this difference is in how much purposed example failure scenarios feel likely and realistic to them. Being myself in the latter camp, I sometimes find I hard to articulate why I think this, and often want better, more evocative examples. Thus I was happy to read your examples because I think they achieve a level of evocativeness that I at least often find hard to create.

I wholeheartedly agree. I think this implies:

1. Getting very clear on what we want. Can we give a fairly technical specification of the kind of safety that's necessary+possible?
2. Some degree of safety beyond tool-type non-malignancy. A proposal which I keep thinking about is my consent-based helpfulness [AF · GW]. The idea is that, in addition to believing that you want something (with sufficient confidence), the system should also believe that you understand the implications of that thing (with some kind of sufficient detail). In the fusion example, the system would engage the user in conversation until it was clear that the consequences for society were understood and approved of.

Note that the fusion power example could be answered directly with a value-alignment type approach, where you have an agent rather than a tool -- the agent infers your values, and infers that you would not really want backyard fusion power if it put the world at risk. That's the moral that I imagine people more into value learning would give to your story. But I'm reaching further afield for solutions, because:

• Value learning systems could Goodhart on the approximate values learned
• Value learning systems are not corrigible if they become overly confident (which could happen at test time due to unforeseen flaws in the system's reasoning -- hence the desire for corrigibility)
• Value learning systems could manipulate the human.

Fourth generation nuclear weapons are as many say in the industry are the "technology of the future and always will be". I understand this is partially a thought experiment, but just to point out that the premise is far from reality.

Molecular laser isotope separation is a much more likely scenario to create fissionable material on the sly. Remember the first atomic bomb to kill people was a howitzer barrel and two lumps of Uranium 235 (not even weapons grade) shot into each other. The amount of material that actually fused would be about the mass of a penny. The tiny amount of fissioned material in little boy was the equivalent of 1.25 miles of box cars full of TNT.

The key to larger and efficient weapons is keeping the radioactive material together longer for more cycles of fission and creating more neutrons from the start. This is done by containment, implosion designs, neutron generators, neutron reflectors, and injecting deuterium and tritium to create more neutrons as the reaction starts.

A truck driver as a hobby built a copy of Little Boy with public sources. As this was easy to do by a single individual why don't we already have these devices cooking off left and right. The given scenario also assumes that design is the only hurdle. Procurement of materials and concealment aren't something that AI can teach.

Why would a technology like fusion be more likely than a technology that has been shown to work?

All of these technologies revolve around a huge amount of electricity up front. Governments already watch high electrical use locations for signs of marijuana growers and uranium refinement. Charging the huge banks of capacitors necessary to start a fusion reaction would easily trigger an investigation on anyone but state actors.

I would suggest that machine learning and gene editing using CRISPR technology to create pathogens would be much easier path to a weapon of mass destruction as they can be done far more covertly.

https://en.wikipedia.org/wiki/Molecular_laser_isotope_separation

https://www.npr.org/2017/12/26/570806064/north-korea-designed-a-nuke-so-did-this-truck-driver

https://www.youtube.com/watch?v=zVhQOhxb1Mc

https://www.youtube.com/watch?v=MnW7DxsJth0

I think the strongest takeaway from this post is that, now that I think about it, is that alignment is not equal to safety, and that even if AI is controllable, it may not be totally safe to someone else.

In your Fusion Power Generator scenario, what happened is that they asked for a fusion power generator, and the AI managed to make the fusion power generator, and it was inner and outer aligned enough to the principal such that it didn't take over the world and make every fusion power plant, and in particular hasn't Goodharted the specification negatively.

In essence, this seems like a standard misuse of AI (though I don't exactly like the connotations), and thus if I were to make this post, I'd focus on how aligning AI isn't enough to assure safety, or putting it another way, there are more problems in AI safety than just alignment/control problems.

Not sure if this is pure musing or a question. The, rather obvious, thought strikes me that this discussion could be held without any reference to AI at all. It is very clear that people with 150+ IQ are much more capable than those with 120 IQ and those with 120 are much more capable than those with sub 100 IQ.

For the most part we live in a market society driven my mass market demand, which seems like it will be dominated by a lower average IQ, which is designed and produced by the "smart" tail of the curve.

This has been the case (well perhaps not the market society claim) for most of human existence.

That suggests we might have evolutionary design patters that have been emerging to protect the masses of the human race from both their own (and perhaps misunderstood or even unknown) risky demand that are delivered by the smart minority of humans.

Is that line of thinking any part of the larger picture (AI alignment I suppose)?

It seems that with a tool-AI like GPT-N, the solution would probably be to dramatically restrict its use to its designers, who should immediately ask it how to solve alignment, which by assumption it can do. The real risk is in making the tool-AI public.