Moloch: optimisation, "and" vs "or", information, and sacrificial ems

Because the length of Scott's Moloch post greatly exceeds my working memory (to the extent that I had trouble remembering what the point was by the end) I made these notes. I hope this is the right place to share them.

Notes on Moloch (ancient god of child sacrifice)

http://slatestarcodex.com/2014/07/30/meditations-on-moloch/

1. Intro - no real content.

2. Moloch as coordination failure: everyone makes a sacrifice to optimize for a zero-sum competition, ends up with the same relative status, but worse absolute status.

   • 10 examples: Prisoner's Dilemma, dollar auctions, fish-farming story (tragedy of the commons), Malthusian trap, ruthless/exploitative Capitalist markets, the two-income trap, agriculture, arms races, cancer, political race to the bottom (lowering taxes to attract business)
   • 4 partial examples: inefficient education, inefficient science, government corruption (corporate welfare), Congress (representatives voting against good of nation for good of constituency)

3. Existing systems are created by incentive structures, not agents, e.g. Las Vegas caused by a known bias in human reward circuitry, not optimization for human values.

4. But sometimes we move uphill anyway. Possible explanations:

   • Excess resources / we are in the dream time and can afford non-competitive behavior.
   • Physical limitations to what can be sacrificed
   • Economic competition actually producing positive utility for consumers (but this is fragile)
   • Coordination, e.g. via governments, guilds, friendships, etc.

5. Technology/ingenuity creates new opportunities to fall into such traps. Technology overcomes physical limitations, consumes excess resources. Automation further decouples economic activity from human values. Technology can improve coordination, but can also exacerbate existing conflicts by giving all sides more power.

AGI opens up whole new worlds of traps: Yudkowsky's paperclipper, Hanson's subsistence-level ems, Bostrom's Disneyland with no children.

6 & 7. Gnon - basically the god of the conservative scarcity mindset. Nick Land advocates compliance; Nyan wants to capture Gnon and build a walled garden. Scott warns that Moloch is far more terrifying than Gnon and will kill both of them anyway.

8 & 9. So we have to kill this Moloch guy, by lifting a better God to Heaven (Elua).

This effect can be combated to some extent with extra information. If the customers (or journalists, bloggers, etc...) know about this, then the coffee plantations will suffer.

At the moment the US food industry successfully lobbied for laws to prevent animal right activists from filming how they treat their animals.

A corporation can argue that they exact mix that they feed to animals is a trade secret and go after people who spread information about it. Companies have much more resources to fight legal battles than bloggers or journalists in underfunded newsrooms.

UK libel law would be another instance of how corporation can fight journalists who spread the information that a product does harm to customers health.

As I mentioned in the comments of Scott's post, I've been thinking about turning my "Technology will destroy human nature" essay into a formal paper, but I'd probably need someone more familiar with evolutionary biology as my co-author to make sure that the analogies to evolution make sense and to otherwise help develop it. TWDHN was basically talking about various physical limits that are currently stopping us from racing to the bottom but which technology will eventually overcome. Now that I've read Meditations, I might want the paper to discuss the other things holding Moloch in check that Scott talks about (excess resources, utility maximization, coordination), too.

First of all, it's not obligatory for an optimisation process to trample everything we value into the mud.

In an intense competition environment, it is obligatory. Any resources spent not optimizing on a fitness axis necessarily make the entity more likely to lose.

Which implies to me that the only way out is to compete so thoroughly and be rich enough that we can act like we're not in competition, and can afford to waste effort on values other than survival/expansion.

Ok, now to the point. The "Moloch" idea is very interesting, and, at the FHI, we may try to do some research in this area (naming it something more respectable/boring, of course, something like "how to avoid stable value-losing civilization attractors").

Given that we don't want to stay with jargon that's not open to outsiders it would be great if we could find a good catch phrase that not as long as the one you suggest.

A recurring problem with these forms of civilizational inadequacy is bystander effect/first-mover disadvantage/prisoners' dilemma/etc, and the obvious solutions (there might be others) are coordination or enforcement. Seeing if there's other solutions and seeing how far people have already run with coordination and enforcement seems promising. Even if one is pessimistic about how easily the problems can be addressed and thinks we're probably screwed anyway but slightly less probably screwed if we try, then the value of information is still very high; this is a common feature of FHI's work, which, by the way, I consider extremely valuable!

What reasons might we have to believe or disbelieve that we can do better than (or significantly improve) governments, the UN, NATO, sanctions, treaty-making, etc.?

Why is Elua still here?

One word which is notably missing from Yvain's excellent blog post is "externalities". The concept is there, but acknowledging that it's externalities we're talking about would be helpful. There is a fair amount of literature on them.

It seems to me that people have a preference against defection when they are able to grasp that it is in fact a PD scenario. This creates a lower bound on how complex a scenario has to be before people will be confused enough to defect reliably and not either notice or be able to come up with reasonable safeguards. As we create more robust complex things I would naively expect the problem to get worse, but OTOH we have the countervailing trend of coordination and information dissemination getting easier. So...we should want to live in a world where we take it more seriously when economists point out perverse incentives? That starts getting political since many economists have policy axes to grind.

Yvain's post is confused in a number of ways, in fact I get the feeling that he hasn't added much to the articles he links at the beginning.

The most blatant problem is his Las Vegas example. He asserts:

Like all good mystical experiences, it happened in Vegas. I was standing on top of one of their many tall buildings, looking down at the city below, all lit up in the dark. If you’ve never been to Vegas, it is really impressive. Skyscrapers and lights in every variety strange and beautiful all clustered together. And I had two thoughts, crystal clear:

It is glorious that we can create something like this.

It is shameful that we did.

What reason does Yvain give for it being shameful? That it's an inefficient use of resources. This is an interesting objection, given that the rest of the essay consists of him objecting to the process of Moloch/Gnon destroying all human values in the name of efficiency. So when faced with an example of Gnon doing the opposite, i.e., building something beautiful the the middle of the desert despite concerns about inefficiency how does he react? By declaring that "there no philosophy on earth that would endorse" its existence.

Yeah, Yvain is not off to a good start here.