On Chesterton's Fence

post by trentbrick · 2020-09-10T15:10:42.321Z · LW · GW · 3 comments

Contents

  Footnotes
None
3 comments

TLDR; Chesterton’s Fences are important, and very hard to identify/evaluate. With finite time, bountiful stupidity and inflated egos, it is too easy to not look deeply enough at existing ways of doing things and understand why they are the way they are before attempting to “fix” them. Reading Secrets of our Success and Seeing Like a State has strengthened my prior to dig deeper on why things are done, in proportion to how long they have stood the tests of time. Writing this piece has helped me develop a framework (in the form of fitness landscapes) to think about Chesterton’s Fences and how uncovering both the motivations and mechanisms behind them is often intractable, requiring clever trial and error along with the acceptance of unfortunate, unintended consequences.

---

Chesterton's Fence states that if you encounter a fence in the middle of nowhere, you should stop and first understand why it was put there before taking it down. There is probably a good reason the fence is there in the first place, and finding out the hard way might be really bad and irreversible. Chesterton's Fence was the original motivation for the creation of Slate Star Codex and is a principle I have thought a lot about recently while reading Secrets of our Success and Seeing Like a State.

Both of these books convey endless appreciation for the complexity, nuance, and unintended consequences that local expertise accounts for and the naive outsider ignores at risk of their own demise. Cherry picking some fascinating examples:

"As one of the world’s staple crops, manioc (or cassava) is a highly productive, starch-rich tuber that has permitted relatively dense populations to inhabit drought-prone tropical environments. However, depending on the variety of manioc and the local ecological conditions, the tubers can contain high levels of cyanogenic glucosides, which release toxic hydrogen cyanide when the plant is eaten. If eaten unprocessed, manioc can cause both acute and chronic cyanide poisoning. Chronic poisoning, because it emerges only gradually after years of consuming manioc that tastes fine, is particularly insidious and has been linked to neurological problems, developmental disorders, paralysis in the legs, thyroid problems (e.g., goiters), and immune suppression. These so-called ”bitter” manioc varieties remain highly productive even in infertile soils and ecologically marginal environments, in part due to their cyanogenic defenses against insects and other pests. In the Americas, where manioc was first domesticated, societies who have relied on bitter varieties for thousands of years show no evidence of chronic cyanide poisoning. In the Colombian Amazon, for example, indigenous Tukanoans use a multistep, multiday processing technique that involves scraping, grating, and finally washing the roots in order to separate the fiber, starch, and liquid. Once separated, the liquid is boiled into a beverage, but the fiber and starch must then sit for two more days, when they can then be baked and eaten. Figure 7.1 shows the percentage of cyanogenic content in the liquid, fiber, and starch remaining through each major step in this processing.

Such processing techniques are crucial for living in many parts of Amazonia, where other crops are difficult to cultivate and often unproductive. However, despite their utility, one person would have a difficult time figuring out the detoxification technique. Consider the situation from the point of view of the children and adolescents who are learning the techniques. They would have rarely, if ever, seen anyone get cyanide poisoning, because the techniques work. And even if the processing was ineffective, such that cases of goiter (swollen necks) or neurological problems were common, it would still be hard to recognize the link between these chronic health issues and eating manioc.

Most people would have eaten manioc for years with no apparent effects. Low cyanogenic varieties are typically boiled, but boiling alone is insufficient to prevent the chronic conditions for bitter varieties. Boiling does, however, remove or reduce the bitter taste and prevent the acute symptoms (e.g., diarrhea, stomach troubles, and vomiting). So, if one did the common-sense thing and just boiled the high-cyanogenic manioc, everything would seem fine. Since the multistep task of processing manioc is long, arduous, and boring, sticking with it is certainly nonintuitive. Tukanoan women spend about a quarter of their day detoxifying manioc, so this is a costly technique in the short term.

At the beginning of the seventeenth century, the Portuguese transported manioc from South America to West Africa for the first time. They did not, however, transport the age-old indigenous processing protocols or the underlying commitment to using those techniques. Because it is easy to plant and provides high yields in infertile or drought-prone areas, manioc spread rapidly across Africa and became a staple food for many populations. The processing techniques, however, were not readily or consistently regenerated. Even after hundreds of years, chronic cyanide poisoning remains a serious health problem in Africa. Detailed studies of local preparation techniques show that high levels of cyanide often remain and that many individuals carry low levels of cyanide in their blood or urine, which haven’t yet manifested in symptoms. In some places, there’s no processing at all, or sometimes the processing actually increases the cyanogenic content. On the positive side, some African groups have in fact culturally evolved effective processing techniques, but these techniques are spreading only slowly."^[1] - Joseph, Henrich. The Secret of Our Success (p. 95-97). Princeton University Press.

An example on the forced villagization of Tanzania in the 1970s:

"A typical cultivator in Tigray, a location singled out for harsh measures, planted an average of fifteen crops a season (such cereal crops as teff, barley, wheat, sorghum, corn, millet; such root crops as sweet potatoes, potatoes, onions; some legumes, including horsebeans, lentils, and chickpeas; and a number of vegetable crops, including peppers, okra, and many others). It goes without saying that the farmer was familiar with each of several varieties of any crop, when to plant it, how deeply to sow it, how to prepare the soil, and how to tend and harvest it. This knowledge was place specific in the sense that the successful growing of any variety required local knowledge about rainfall and soils, down to and including the peculiarities of each plot the farmer cultivated. It was also place specific in the sense that much of this knowledge was stored in the collective memory of the locality: an oral archive of techniques, seed varieties, and ecological information.

Once the farmer was moved, often to a vastly different ecological setting, his local knowledge was all but useless. As Jason Clay emphasizes, “Thus, when a farmer from the highlands is transported to settlement camps in areas like Gambella, he is instantly transformed from an agricultural expert to an unskilled, ignorant laborer, completely dependent for his survival on the central government.” ...

Instead of the unrepeatable variety of settlements closely adjusted to local ecology and subsistence routines and instead of the constantly changing local response to shifts in demography, climate, and markets, the state would have created thin, generic villages that were uniform in everything from political structure and social stratification to cropping techniques. The number of variables at play would be minimized. In their perfect legibility and sameness, these villages would be ideal, substitutable bricks in an edifice of state planning. Whether they would function was another matter." - James C. Scott, Seeing Like a State (p. 250-255). Yale University Press.

This example and its context in the book presents a meta-Chesterton Fence. At the levels of each individual tribe, farmer, and even plot of land, there is a unique environment that cultural and agricultural expertise evolved to harness over time.

On a seemingly silly but potentially highly effective hunting strategy:

"When hunting caribou, Naskapi foragers in Labrador, Canada, had to decide where to go. Common sense might lead one to go where one had success before or to where friends or neighbors recently spotted caribou. ... Hunters want to match the locations of caribou while caribou want to mismatch the hunters, to avoid being shot and eaten. If a hunter shows any bias to return to previous spots, where he or others have seen caribou, then the caribou can benefit (survive better) by avoiding those locations (where they have previously seen humans). Thus, the best hunting strategy requires randomizing. Can cultural evolution compensate for our cognitive inadequacies?

Traditionally, Naskapi hunters decided where to go to hunt using divination and believed that the shoulder bones of caribou could point the way to success. To start the ritual, the shoulder blade was heated over hot coals in a way that caused patterns of cracks and burnt spots to form. This patterning was then read as a kind of map, which was held in a prespecified orientation. The cracking patterns were (probably) essentially random from the point of view of hunting locations, since the outcomes depended on myriad details about the bone, fire, ambient temperature, and heating process. Thus, these divination rituals may have provided a crude randomizing device that helped hunters avoid their own decision-making biases." - Joseph, Henrich. The Secret of Our Success (p. 104-105). Princeton University Press.

An interesting tidbit on how difficult it is for humans to be random from Scott Aaronson:

In a class I taught at Berkeley, I did an experiment where I wrote a simple little program that would let people type either “f” or “d” and would predict which key they were going to push next. It’s actually very easy to write a program that will make the right prediction about 70% of the time. Most people don’t really know how to type randomly. They’ll have too many alternations and so on. There will be all sorts of patterns, so you just have to build some sort of probabilistic model. Even a very crude one will do well. I couldn’t even beat my own program, knowing exactly how it worked. I challenged people to try this and the program was getting between 70% and 80% prediction rates. Then, we found one student that the program predicted exactly 50% of the time. We asked him what his secret was and he responded that he “just used his free will.” - Try it for yourself here, source, hat-tip.

In all of the examples provided, failure to acknowledge local expertise, painfully acquired through trial and error and retained across generations, leads to a suboptimal outcome. This suboptimality is not only for the very metric desired, for example, crop yields, but also for overall welfare where the Tanzanian’s crops became more vulnerable, less nutritionally diverse and their community ties were severed. There are many further interesting examples of Chesterton’s Fence in the two books, from rituals for pregnant mothers to not eat shark, to the perils of scientific forestry and the high-modernist planning ideals that made Brasilia one of the least livable cities on earth.

Beyond creating a suboptimal outcome for the desired metric, the very creation of this metric induces Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." often because the metric is exploited. An amusing yet tragic example from Seeing Like a State is:

"The door-and-window tax established in France under the Directory [1795] and abolished only in 1917 is a striking case in point. Its originator must have reasoned that the number of windows and doors in a dwelling was proportional to the dwelling’s size. Thus a tax assessor need not enter the house or measure it but merely count the doors and windows. As a simple, workable formula, it was a brilliant stroke, but it was not without consequences. Peasant dwellings were subsequently designed or renovated with the formula in mind so as to have as few openings as possible. While the fiscal losses could be recouped by raising the tax per opening, the long-term effects on the health of the rural population lasted for more than a century." - James C. Scott, Seeing Like a State (pp. 47-48). Yale University Press.

I find Chesterton's Fence a very useful principle that we are prone to forget. However, from reading page after page about the failures of societal interventions, I feel the sheer complexity of the natural world and severity of the unintended consequences of our actions can induce a state of "epistemic learned helplessness" that leads me into the comforting arms of the appeal to nature. So what should we keep in mind as we gaze upon this fence in the middle of nowhere and consider dismantling it?

I think it is useful to perceive Chesterton’s Fence as a local optimum on a very non-linear, high dimensional fitness landscape. Evolution, primarily of our culture but also of our genome, has converged on this local optimum through trial, error, and improvement. This local optimum is often deeper and more optimal than the naive outside observer is aware. Dismantling Chesterton's Fence shifts our current solution’s parameters away from this optimum and the search for a new optimum can be expensive, slow, and result in a worse new solution we can’t backtrack from.

I like this fitness landscape framing for a few reasons:

Fitness landscapes are often intractable to optimize because either the evaluation of a particular solution is too expensive or the space is too large to be effectively searched. In the case of modifying our lives, social norms, or society, both of these difficulties are present. As a result, we are basically blind in taking steps across the state space and need to use trial and error, making these steps very small so that they are cheap to perform and reversible.^[2] This means running experiments (ideally RCTs) and using evidence from them to constantly diagnose how new solutions are performing on old problems, and inform directions in which further improvements are sought.

But even RCTs aren't enough. Seeing Like a State highlights well-intentioned failures of scientific agriculture as the result of failing to test solutions in sufficiently diverse environments. For any scientific experiment there are only so many variables that can be feasibly tested and controlled for. Not only is this number often much smaller than necessary to provide findings that work in the real world, but also it is plausible that no solution will generalize across such diverse environments. It is within this context that Seeing Like a State argues science should have a deeper appreciation for cultural practices and tacit knowledge, at the very least using them as a form of hypothesis generation.

For example, traditional medicinal practices have been a boon to drug discovery. Western medicine uses RCTs to assess efficacy but very often lacks a mechanistic understanding of why something works. We are still uncovering the mechanisms underlying aspirin and one of the most successful cancer drugs, paclitaxel, was discovered after sprinkling a compound from the bark of a Pacific Yew tree on cancer cells and still has unknown mechanisms.

Another reason the fitness landscape is a useful framing for Chesterton’s Fence is that the “fitness” used to determine a landscape is arbitrary. The No Free Lunch Theorem states that if we were to average the quality of our solution across all possible definitions of fitness, all solutions would perform equally. Therefore, we must first define fitness. Moreover, the fitness landscape our culture and genomes have been optimizing for in the past have been defined by evolution, not the ideals of a modern liberal democracy. And as causally opaque and crucial some rituals are to survival, such as cassava processing, there also exist clearly harmful and unnecessary practices such as female genital mutilation.

On the problem of defining fitness, I wish there was more discussion about utopias to know what end goals or ideals we should be optimizing for. I'd love suggestions for Utopian reading, the most compelling political system/utopia I have encountered is: this but my reference class is very small^[3].

Aside from “fitness” being arbitrary, the landscape is constantly changing as our physical and societal environments present new ridges and valleys for the existing local optimum that culture and evolution are always optimizing. For example, accelerating technological progress is making "increasing numbers of things we like [...] into things we like too much" by becoming addictive, hacking our reward pathways like an Oreo. In this context, the status quo is maintained only with the presence, not absence of efforts to resist novel addictions.

Seeing Like a State certainly acknowledges the expensive search for often worse local optima ridden with unintended consequences. However, it also recognizes that this disruption is necessary to enable greater State intervention, which in turn can produce great outcomes. This is because local culture is too nuanced to be measured, summarized, and understood from a central decision making body. Greater State intervention is a prerequisite to the public health, sanitation and welfare miracles of the last couple centuries. Therefore, uprooting Chesterton’s Fences can be a long term investment by the State in better wellbeing, leading to superior local optima.

I believe that the concept of Chesterton's Fence is powerful. Obtaining more information before taking any action is always desirable. However, this information acquisition can often be too costly, the Fence may rest on dubious morals, and be transforming already and independently. Under these circumstances, Chesterton’s Fences should be challenged in the name of human flourishing, but only with great humility, careful experimentation, and apologies for the inevitable unintended consequences. Good intentions alone, as Seeing Like a State documents in detail, are not enough and can be downright dangerous because of the dramatic changes they inspire. A quote from Yudkowsky comes to mind that can be modified to read: "not all changes to Chesterton’s Fences lead to better outcomes but all better outcomes come from changes to Chesterton’s Fences." Thus, with a hat tip to the No Free Lunch Theorem, let us find humanity a restaurant where lunch isn't free but the greatest number finds it tastier.

Thanks very much to Joe Choo-Choy, Max Farrens, and Miles Turpin for reading drafts of this piece.

The original post has been cross posted from: https://trentbrick.github.io/On-Chestertons-Fence/ .

Footnotes

---

1. It would be interesting to know if the processing techniques used by the Amazonians and lacking in much of Africa were developed simply because of time, if the Amazonian diet is more restricted which would increase selection pressure, or if there are other analogous food processing techniques readily generalized from other foods already. ↩︎

2. Joe Choo-Choy rightly points out the complication that steps must be sufficiently large to override any noise naturally occurring to the parameters. ↩︎

3. I like a quote in Seeing Like a State from Albert Howard: “The discovery of the things that matter is three quarters of the battle.” ↩︎

3 comments

Comments sorted by top scores.