Allegory On AI Risk, Game Theory, and Mithril

I like this. I wonder what would happen if you post this on some LOTR fanfiction subreddit or something, with a link to a discussion of AI risk at the end.

"But, Bifur, the prophecies are not that clear. It's possible the Balrog will annihilate us, but it's also possible he will eradicate poverty, build us dwarf-arcs to colonize other planets, and grant us immortality. Our previous mining efforts have produced some localized catastrophes, but the overall effect has been fantastically positive, so it's reasonable to believe continued mining will produce even more positive outcomes."

This seems like a better metaphor for fossil fuel extraction and climate change than AI risk.

An interesting metaphor, given how the balrog basically went back to sleep after eating the local (and only the local) dwarves. And after some clumsy hobbitses managed to wake him up again, he was safely disposed of by a professional. In no case did the balrog threaten the entire existence of the Middle-Earth.

This Thorin guy sounds pretty clever. Too bad he followed his own logic straight to his demise, but hey he stuck to his guns! Or pickaxe, as it were.

His argument attempting to prevent Bifur from trying to convince fellow Dwarves against mining into the Balrog's lair sounds like a variation on the baggage carousel problem (this is the first vaguely relevant link I stumbled across, don't take it as a definitive explanation)

Basically, everyone wants resource X, resulting in a given self-interested behavior whose result is to collectively lower everyone's overall success rate, but where the solution that maximizes total success directly goes against each person's self-interest. This results in an equilibrium where everyone works sub-optimally.

In this variation, the action of Thorin's operation achieving resource M moves everyone slightly closer to negative consequence B. So the goal is no longer to maximize resource collection, but to minimize it. Doing so goes against everyone's self-interest, resulting in etc. That is what Thorin is so eloquently trying to prevent Bifur from doing.

There are a couple ways Bifur can approach this.

He could do it through logical discourse: Thorin is in error when he claims

Each individual miner would correctly realize that just him alone mining Mithril is extraordinarily unlikely to be the cause of the Balrog awakening

because it assumes unearthing the Balrog is a matter of incrementally filling a loading bar, where each Dwarf's contribution is miniscule. That's the naive way to imagine the situation, since you see in your mind the tunnel boring ever closer to the monster. But given that we can't know the location or depth of the Balrog, each miner's strike is actually more like a dice roll. Even if it's a large dice roll, recontexualizing the danger in this manner will surely cause some dwarves to update their risk-reward assessment of mining Mithril. A campaign of this nature will at least lower the number of dwarves willing to join Thorin's operation, although it doesn't address the "Balrog isn't real" or "Balrog isn't evil" groups.

Alternatively, he could try to normalize new moral behavior. People are willing to work against their self-interest if doing so demonstrates a socially accepted/enforced moral behavior. If he were a sly one, he could sidestep the divisive Balrog issue altogether and simply spread the notion that wearing or displaying Mithril is sinful within the context of Dwarven values. eg maybe it's too pragmatic, and not extravagant enough for a proper ostentatious Dwarven sensibility. That could shut down Thorin's whole operation without ever addressing the Balrog.

But Bifur probably sees the practical value of Mithril beyond its economic worth. As Thorin says, it's vital for the war effort - completely shutting down all Mithril mining may not be the best plan if it results in a number of Dwarf casualties similar to or greater than what he estimates a Balrog could do. So a more appetizing plan might be to combine the manipulation of logic and social norms. He could perform a professional survey of the mining systems. Based on whatever accepted arbitrary standards of divining danger the Dwarves agree to (again, assuming the location of the Balrog is literally unknowable before unearthing it due to magic), Bifur could identify mining zones of ever increasing danger within whatever tolerances he's comfortable with. He could then shop these around to various mining operations as helpful safety guidelines until he has a decent lobby behind him in order to persuade the various kings to ratify his surveys into official measuring standards. Dwarves are still free to keep mining deeper if they wish, but now with a socially accepted understanding that heading into each zone ups their risk relative to their potential reward, naturally preventing a majority of Dwarves from wanting to do so. Those who believe the Balrog doesn't exist or is far away would be confronted with Bifur's readily available surveys, putting them on the defensive. There would still be opposition from those who see the Balrog as "not evil", but the momentum behind Bifur's social movement should be enough to shout them down. This result would allow Thorin's operation to continue to supply the realm with life-saving Mithril, while at least decreasing the danger of a Balrog attack for as long as Bifur's standards are recognized.

Finally, Bifur could try to use evidenced-based research and honestly performed geological surveys, but even in the real world where locating the Balrog beforehand is technologically possible, that tends to be a weaker tactic than social manipulation. Only other experts will be able to parse them, his opponents will have emotional arguments that will give them the upper hand, and Thorin's baggage carousel logic would remain unchallenged.

Bifur should tell everyone that he is going to try to wake the Balrog, and dig directly towards it, openly advertising his intent to be the first to wake it. Spreading rumors that he intends to yoke the Balrog to a plow, and that he alone has a specific plan for accomplishing this would be helpful too.

The action taken to control the insanity of that one crazy dwarf might prevent the catastrophe outright.

Totally unrealistic.

Thorin was never in a position to hire mirthril miners. He had sufficient capital for only a very brief time before dying at the Battle of Five Armies.

In other words, if you set up the allegory so as to force a particular conclusion, that proves that that's the proper conclusion in real life, because we all know that the allegory must be correct.

You assume that balrogs can only be stopped by unmined bedrock. Since the chance of a given balrog being stopped by bedrock but not by the combined efforts of the dwarves is muniscule compared to the chance of a weak one that can be stopped by mithril-clad soldiers or a strong one that can dig through mere stone, the best defense against balrogs is to mine and guard the mines well.

I have the feeling you still don't agree with Thorin. Why not?

If you really believe in this allegory, you should try to intervene before people choose what research field to specialize in. You are not going to convince people to give up their careers in AI after they've invested years in training. But if you get to people before they commit to advanced training, it should be pretty easy to divert their career trajectory. There are tons of good options for smart idealistic young people who have just finished their undergraduate degrees.

I've been hired by a rich dwarf to fortify his castle, and I found that instead of using the world-endangering Mithril you get about 80% or the strength with an alloy that contains very small amount of Mithril and large amounts of other metals that does not require deep excavation (we patented the alloy as Mothril(TM). While the strength is a little less it's much cheaper and you can make it up in volume. If people need an even stronger metal I think they should be working on Mothril++

So I think the economic explanation for Mithril is somewhat weaker than the dwarfs' desire to keep digging deeper. They really need to be re-educated (I hope wizards are also working on that)

or even used it to hire a wizard to work on an admittedly long-shot, Balrog control spell

I have a higher probability of a group of very dedicated wizards succeeding, worth re-doing the above decision analysis with those assumptions