Moloch Hasn’t Won

Moloch's biggest enemy is change. In a rapidly changing environment, "Moloch" presents as crippling overspecialization. Since "slack" is useful in a wide variety of circumstances, rapid change selects for slack.

There's a lot to be said for the dream-time argument. Inefficiency gives room for slack, and individual misalignment with group averages is inefficient. There are fewer people than the planet could support (in the near-term; hard to know what will happen in the longer term), easing the competitive pressure.

Limiting the number of children that industrial people have lets them maintain the wealth concentrations that make their lives pleasant.

This review is more broadly of the first several posts of the sequence, and discusses the entire sequence. 

Epistemic Status: The thesis of this review feels highly unoriginal, but I can't find where anyone else discusses it. I'm also very worried about proving too much. At minimum, I think this is an interesting exploration of some abstract ideas. Considering posting as a top-level post. I DO NOT ENDORSE THE POSITION IMPLIED BY THIS REVIEW (that leaving immoral mazes is bad), AND AM FAIRLY SURE I'M INCORRECT.

The rough thesis of "Meditations on Moloch" is that unregulated perfect competition will inevitably maximize for success-survival, eventually destroying all value in service of this greater goal. Zvi (correctly) points out that this does not happen in the real world, suggesting that something is at least partially incorrect about the above mode, and/or the applicability thereof. Zvi then suggests that a two-pronged reason can explain this: 1. most competition is imperfect, and 2. most of the actual cases in which we see an excess of Moloch occur when there are strong social or signaling pressures to give up slack. 

In this essay, I posit an alternative explanation as to how an environment with high levels of perfect competition can prevent the destruction of all value, and further, why the immoral mazes discussed later on in this sequence are an example of highly imperfect competition that causes the Molochian nature thereof. 

First, a brief digression on perfect competition: perfect competition assumes perfectly rational agents. Because all strategies discussed are continuous-time, the decisions made in any individual moment are relatively unimportant assuming that strategies do not change wildly from moment to moment, meaning that the majority of these situations can be modeled as perfect-information situations. 

Second, the majority of value-destroying optimization issues in a perfect-competition environment can be presented as prisoners dilemmas: both agents get less value if all agents defect, but defection is always preferable to not-defection regardless of the strategy pursued by other agents. 

Now, let's imagine our "rational" agents operate under simplified and informal timeless decision theory: they take 100% predictable opponent's strategy into account, and update their models of games based on these strategies (i.e. Prisoner's Dilemma defect/cooperate with two of our Econs has a payout of -1+0*n,5+0*n)

(The following two paragraphs are not novel, they is a summary of the thought experiment that provides a motive for TDT) Econs, then, can pursue a new class of strategies: by behaving "rationally," and having near-perfect information on opposing strategies because other agents are also behaving "rationally," a second Nash equilibria arises: cooperate-cooperate.  The most abstract example is the two perpetually betrayed libertarian jailbirds: in this case, from the outset of the "market," both know the other's strategy. This creates a second Nash equilibria: any change in P1's strategy will be punished with a change in P2's strategy next round with extremely high certainty. P1 and P2 then have a strong incentive to not defect, because it results in lots of rounds of lost profit. (Note that because this IPD doesn't have a known end point, CDT does not mandate constant defection). In a meta-IPD game, then, competitive pressures push out defector agents who get stuck in defect-defect with our econs. 

Fixed-number of player games are somewhat more complex, but fundamentally have the same scenario of any defection being punished with system-wide defection, meaning defection in highly competitive scenarios with perfectly rational agents will result in system-wide defection, a significant net negative to the potential defector. The filters stay on in a perfect market operating under TDT.

Then, an informal form of TDT (ITDT [to be clear, I'm distinguishing between TDT and ITDT only to avoid claiming that people actually abide by a formal DT)) can explain why all value is not destroyed in the majority of systems, even assuming a perfectly rational set of agents. Individually, this is interesting but not novel or particularly broad: the vast majority of the real-world examples discussed in this sequence are markets, so it's hard to evaluate the truth of this claim without discussing markets. 

Market-based games are significantly more complex: because free entry and exit are elements of perfect competition, theoretically, an undercutter agent could exploit the vulnerabilities in this system by pursing the traditional strategy, which may appear to require value collapses as agents shore up against this external threat by moving to equilibrium pricing. Let's look at the example of the extremely rational coffee sellers, who have found a way to reduce their costs (and thus, juice their profits and all them to lower prices, increasing market share) by poisoning their coffee. At the present moment, CoffeeA and CoffeeB both control 50% of the coffee industry, and are entirely homogenous. Under the above simple model, assuming rational ITDT agents, neither agent will defect by poisoning coffee, because there's no incentive to destroy their own profit if the other agent will merely start poisoning as well. However, an exploiter agent could begin coffee-poisoning, and (again assuming perfect competition) surpass both CoffeeA and CoffeeB, driving prices to equilibrium. However, there's actually no incentive, again assuming a near-continuous time game, for CoffeeA and CoffeeB to defect before this actually happens. In truely perfect competition, this is irrelevant, because an agent arises "instantly" to do so, but in the real world, this is relaxed. However, it's actually still not necessary to defect even with infinite agents: if the defection is to a 0-producer surplus price, the presence of additional agents is irrelevant because market share holds no value, so defection before additional agents arrive is still marginally negative. If the defection is to a price that preserves producer surplus, pre-defecting from initial equilibria E1 to E2 price only incentives the stable equilibria to be at a lower price, as the new agent is forced to enter at a sub-E2 price, meaning the final equilibria is effectively capped, with no benefits. Note that this now means that exploiter agents are incentivized to enter at the original equilibria price, because they "know" any other price will trigger a market collapse to that exact price, so E1 maximizes profit. 

This suggests that far from perfect competition destroying value, perfect competition may preserve value with the correct choice of "rational agents." However, any non-rational agents, or agents optimizing for different values, immediately destroy this peaceful cooperation! As EY notes here [LW · GW], TDT conceals a regress when other agents strategy is not predictable, which means that markets that substantially diverge from perfect competition, with non-perfect agents and/or non-perfect competition are not subject to our nice toy model.

Under this model, the reason why the airline industry is so miserable is because bailouts are accepted as common practice. This means that agents can undercut other agents to take on excess risk safely, effectively removing the 0-producer-surplus price (because agents can disguise costs by shuffling them to risk), and make strategy unpredictable and not subject to our cooperative equilibria. 

Let's look at the middle-manager example brought up in the later part of the article. Any given middle manager, assuming all middle managers were playing fully optimized strategies, would not have a strong incentive to (WLOG) increase their hours at the office. However, the real world does not behave like this: as Zvi notes, some peers shift from "successful" to "competent," and despite the assertion that middle-management is an all-or-nothing game, I suspect that middle management is not totally homogenous in terms of willingness to erode ever-more valuable other time. This means that there are massive incentives to increase time at the office, in the hopes that peers are not willing to. The other dynamics noted by Zvi are all related to lack of equilibria, not the cause thereof.  

This is a (very) longwinded way of saying that I do not think Zvi's model is the only, the most complete way, or the simplest way to model the dynamics of preserved value in the face of Moloch. I find several elements of the ITDT explanation quite appealing: it explains why humans often find traditional Econs so repulsive, as many of the least intuitive elements of traditional "rationality" are resolved by TDT. Additionally, I dislike the vague modeling of the world as fine because it doesn't satisfy easy-to-find price information intuitively: I don't find the effect strong enough to substantially preserve value intuitively. In the farmers market scenario specifically, I think the discrepancy between it being a relatively perfect competitive environment and having a ton of issues with competitiveness was glossed over too quickly; this type of disagreement seems to me as though it has the potential to have significant revelatory power. I think ITDT better explains the phenomena therein: farmer's market's aren't nearly as cutthroat as financial markets in  using tools developed under decision that fails the prisoner's dilemma, meaning that prisoner's dilemmas are more likely to follow ITDT-type strategies. If desired, or others think it would offer clarity, I'd like to see either myself or someone else go through all of the scenarios discussed here under the above lens: I will do this if there is interest and this idea of this post doesn't have obvious flaws.

However, I strongly support curation of this post: I think it poses a fascinating problem, and a useful framing thereof. 

tl;dr: the world could also operate under informal TDT,  this has fairly strongly explanatory power for observed Moloch/Slack differentials, this explanation has several advantages.

This is an important clarification/modification to a very important community concept, and as such, is deserving of further canonization in the LW review process.

Asking the question "why are things so bad?" (or, as it was put in Meditations on Moloch, "what does it?") leads to a lot of clarification useful for fighting the problems in the world. Yet, seeing all those mechanisms in detail can also lead to a lot of hopelessness.

Asking the mirror question, "why are things so good?" is also very helpful, particularly once one has traveled far down the road of understanding why things are so bad.

I have objections to most of your list of Elua wins, and that's despite me being an optimist. For now I'll just say that defensive tech outrunning offensive tech allows for capital formation.

I'm quite interested in the rest of this. Though I did find the idea of Moloch useful for responding to the most naive forms of "If we all did X everything would be perfect", I also have a vague feel that rationalist's belief in Moloch being all powerful prevents them from achieving totally achievable levels of group success.

I have repeatedly thought back to and referenced this series of posts, which improved my mental models for how people engage within corporations.

> If you’re a king with 5 districts, and you have 20 competent managers who trust each other… one thing you can do is assign 4 competent managers to each fortress, to ensure the fortress has redundancy and resilience and to handle all of its business without any backstabbing or relying on inflexible bureaucracies. But another thing you can do is send 10 (or 15!) of the managers to conquer and reign over *another* 5 (or 15!) districts.

> This is bad if you’re one of the millions of people who live in the kingdom, who have to contend with werewolves.

> It’s an acceptable price to pay if you’re actually the king. Because if you didn’t pay the price, you’d be outcompeted by an empire who did. And meanwhile it doesn’t actually really affect your plans that much.

...

The key instinct is that any price that can be paid to be stronger or more competitive, must be paid, therefore despair: If you didn’t pay the price, you’d be out-competed by someone who did. People who despair this way often intuitively are modeling things as effectively perfect competition at least over time, which causes them to think that everything must by default become terrible, likely right away.

[...]

Kingdoms don’t reliably expand to their breaking points.

Anthropics vs Goals

I didn't get around to replying to this until today, but this wasn't my main point and I think it's pretty important.

The issue isn't whether you'll fail to achieve your goals if you don't expand. The issue is "from an anthropic reasoning perspective, what sort of world will most people live in?"

I have shifted some of my thinking around "you'll be outcompeted and therefore it's in your interest to expand". I think I agree with "it's generally not worth trying to be the winner-take-all winner, because a) you need to sacrifice all the things you cared about anyway, b) even if you do, you're not actually likely to win anyway."

But that was only half the question – if you're looking around the world, trying to build a model of what's going on, I think the causal explanation is that "organizations that expand end up making up most of the world, so they'll account for most of your observations."

The reason this seems important is, like, I see you and Benquo looking in horror at the world. And... it is a useful takeaway that "hmm, I guess I don't need to expand in order to compete with the biggest empires in order to be happy/successful/productive, I can just focus on making a good business that delivers value and doesn't compromise it's integrity." (And having that crystallized has been helpful to my own developing worldview)

Nonetheless... the world will continue to be a place you recoil in horror from until somehow, someone creates something either stops mazes, or outcompetes them, or something. 

Breaking Points vs Realistic Tradeoffs

I also disagree with the characterization "kingdoms don't expect to breaking point." 

The original concept here was "why do people have a hard time detecting obfuscators and sociopaths?". A realistic example (to be clear I don't know much about medieval kingdoms), is a corporation that ends up creating multiple departments (i.e. hiring a legal team), or expanding to new locations.

This doesn't mean you expand to your breaking point – any longterm organization has to contend with shocks and robustness. The organizations I expect to be most successful will expand carefully, not overextending. But if you're asking the question "why are there obfuscators everywhere?", I think the answer is because the relative profitability of extinguishing obfusctators,  vs. not worrying as much about it, points towards the latter direction.

This is, in part, because extinguishing obfuscating or other mazelike patterns is a rare, high skill job that, like, even small organizations don't usually have the capacity to deal with. I think if you can make it much cheaper, it's probably possible to shift the global pattern. But I think the status quo is that the profit-maximizing thing to do focus on expansion over being maze-proof, and there's a lot of profit-maximizing-entities out there.

It's not worth it for the king to try to expand to take over the world. It still seems, for many kings in many places, that expanding reasonably, robustly, is still the right strategy given their goals (or at least, they think it's their goal, and you'd have your work cut out for you convincing them otherwise), and that meanwhile worrying about werewolves in the lawyer department is probably more like a form of altruism than a form of self-interest. 

Or, reframing this as a question (since I'm honestly not that confident)

If your inner circle is safe, how much selfish reason does a CEO have to make sure the rest of the organization is obfuscator-proof?

Oh you firstwolder, you. Scott is so right somewhere.

(and how do you know that "Most communities do get most people to pitch in"?)

I think the main reason Moloch doesn't succeed very effectively is just because the common response to "hey, you could sacrifice everything of value and give up all slack to optimize X" is "yeah but have you considered just, yanno, hanging out and watching TV?"

And most people who optimize X aren't actually becoming more competitive in the grand scheme of things. They'll die (or hopefully not die) like everybody else and probably have roughly the same number of kids. The selection process that created humans in the first place won't even favor them!

As a result, I'm not worried about Moloch imminently taking over the world. Frankly, I'm more short-term concerned with people just, yanno, hanging out and watching TV when this world is abjectly horrifying.

I am long-term concerned about Moloch as it pertains to value-drift. I doubt the sound of Moloch will be something like "people giving up all value to optimize X" and expect it to be something more like "thousands of years go by and eventually people just stop having our values."

Some day I might go read the background here.

I do wonder if the old saying about evil triumphing only if good people stay quiet doesn't apply. Perhaps that is the source of all those unfortunate accidents Elua enjoys. But that is a pretty weak thesis. What might the model be that gets us some ratios related to number of good, bad and indifferent and perhaps a basic human trait about feeling better inside if we don't do bad. That last bit then allows a large number in the group be indifferent but display a propensities more aligned with Elua than Moloch.

There also seems to be (assuming I actually get the whole map-territory view right) an analytical concern. Perfect competitions is a fiction made up to allow the nice pictures to be drawn on the chalkboard. They are maps, and rather poor, simplistic ones at that, rather than the underlying territory. Seems like Moloch is given power based on the map rather than the underlying territory. Perhaps that is why you offered the technical note so look forward to where you take that. I suppose one path might be to suggest the accidents are not so accidental or surprising.

Nominating this whole sequence. I learned a lot from it.

Slack: Getting Past Burnout, Busywork, and the Myth of Total Efficiency is about why businesses fail if they ignore all other values in favor of maximizing profit-- they lose too much flexibility.

I'm looking forward to the rest of this series.

The real world is high dimentional, and many people will go slightly out of their way to help. If the coffee place uses poisonous pesticides, people will tell others, an action that doesn't cost them much and helps others a lot.

Your Moloch traps only trap when they are too strong for the Moloch haters to destroy. The Moloch haters don't have a huge amount of resources, but in a high dimensional system, there is often a low resource option.

It seems like the stability point of a lot of systems is Moloch-like. (Monopolies, race to the bottom, tragedy of the commons, dictatorships, etc.) It requires energy to keep the systems out of those stability points.

Lots of people need to make lots of sacrifices to keep us out of Moloch states. It is not accidents. It is revolutions, and voter registration, and volunteering and standing up to bullies. It is paying extra for fair trade coffee and protesting for democracy in Hong Kong.

Moloch has a huge advantage. If we do nothing, it will always win. We need to remember that.

I can't wait to read the rest of this sequence.

My starting opinion is still that Elua has to exist to win, and Moloch doesn't.

There is a big difference between a universe with -hugenum value, and a universe with 0 value. Moloch taking over would produce a universe with 0 value, not -hugenum (not just because we might assume that pain and pleasure are equally energy-efficient).

When one then considers how much net value there is in the universe (or how much net disvalue!), I suspect Eluas winning, while probably positive, isn't that great: Sure, sometimes someone learns something in the education system, but many other people also waste intellectual potential, or get bullied.

I strongly suggest you rewrite your summary of "physical limitations". The original was slightly problematic; your summary is, to me, a train-wreck.

Scott's original point was, I believe, "slavery itself may be an example of a bad collective equilibrium, but work-people-to-death antebellum southern slavery was even worse than that." He spent so much effort showing how the WPTD version was inefficient that he forgot to reiterate the obvious point that both versions are morally bad; and since he was contrasting the two, it would be possible to infer that he's actually saying that non-WPTD slavery is not so bad morally; but he clearly deserves the benefit of the doubt on that, and anybody who's read that far is likely to give it to him.

Your summary is shorter, so it's easier to misinterpret, and "people unlikely to give you the benefit of the doubt" are more likely to read it. Furthermore, using "you" to mean slavers makes it actually worse than Scott's version. I, for one, really don't want to be asked to put myself into slavers' shoes unless it's crucial to the point being made, and in this case it clearly isn't.

I suggest you remove the "you" phrasing, and also explicitly say that even non-WPTD slavery is bad; that this is an example of physical limitations slightly ameliorating a bad equilibrium, but not removing it altogether. You can, I believe, safely imply that that's what Scott believes too, even though he doesn't explicitly say it.