Parable: The Bomb that doesn't Explode

post by Lone Pine (conor-sullivan) · 2022-06-20T16:41:14.611Z · LW · GW · 5 comments

Contents

5 comments

You're an engineer working for a military contractor. One day, your project manager comes to you and asks you to design a container for plastic explosive. It is to contain several kilograms of C4, enough to destroy a building. This is pretty dangerous, but you know that C4 is actually pretty safe. It can't be accidentally detonated by fire, impact or bullets. Only a detonator (another explosive) can trigger C4, so you don't include a detonator in your design, for safety.

Your PM reviews the design and gives you some feedback. "It would be a lot more useful if you put a blasting cap inside." You grimace. More useful, yes. But a lot less safe. Nevertheless, you do your job and install a blasting cap. The blasting cap has two electrical leads. If a voltage is applied across those leads, the C4 will explode, killing everyone around. To keep things safe, you snip off the the leads and put the blasting cap inside a pill-shaped plastic container, inside the larger container that contains the C4.

"Well, that's no good," your PM replies, "What if someone does want to put a voltage across the leads?" You grimace even more. It is getting very difficult to make this design safe, you think. But you have a solution: inside the pill-shaped plastic container, you install a Raspberry Pi. You install SELinux on the Pi and set it up to connect to WiFi -- but only secured networks. You program up a fancy web interface that allows the user to specify exactly what voltage they want to apply to the leads. The interface caps the voltage at a low level. It does not allow the user to apply a sufficient voltage to actually trigger the blasting cap -- or at least you hope that's how blasting caps work.

"Well, that's no good," your PM replies, "What if someone wants to apply a higher voltage?" You grimace again. Then it'll explode! It'll kill everyone! But this is not a workplace where you raise objections, so instead you just diligently do your job. If you don't make this dangerous thing, the company will hire someone else without your moral scruples, and certainly that person would make something really dangerous! So you alter the web interface, allowing the user to specify any voltage up to the limit of what the Pi can output. You add a big red warning screen, explaining how C4 is dangerous, and forcing the user to click "Yes, I'm really sure I want to apply this voltage." You also add a fancy cryptographic security key.

Your PM reviews the final design. She's a bit confused why this bomb is so damn complicated, but no matter, it serves its purpose well. The bomb goes on to kill someone whose name you cannot pronounce, thus defending the American people from great evil. (/s)

The moral of the story is, if you don't want a dangerous thing to get built, you have to actually not build it. [LW(p) · GW(p)]

5 comments

Comments sorted by top scores.