Reason isn't magic

post by Benquo · 2019-06-18T04:04:58.390Z · score: 115 (39 votes) · LW · GW · 11 comments

This is a link post for http://benjaminrosshoffman.com/reason-isnt-magic/

Here's a story some people like to tell about the limits of reason. There's this plant, manioc, that grows easily in some places and has a lot of calories in it, so it was a staple for some indigenous South Americans since before the Europeans showed up. Traditional handling of the manioc involved some elaborate time-consuming steps that had no apparent purpose, so when the Portuguese introduced it to Africa, they didn't bother with those steps - just, grow it, cook it, eat it.

The problem is that manioc's got cyanide in it, so if you eat too much too often over a lifetime, you get sick, in a way that's not easily traceable to the plant. Somehow, over probably hundreds of years, the people living in manioc's original range figured out a way to leach out the poison, without understanding the underlying chemistry - so if you asked them why they did it that way, they wouldn't necessarily have a good answer.

Now a bunch of Africans growing and eating manioc as a staple regularly get cyanide poisoning.

This is offered as a cautionary tale against innovating through reason, since there's a lot of information embedded in your culture (via hundreds of years of selection), even if people can't explain why. The problem with this argument is that it's a nonsense comparison. 

First of all, it's not clear things got worse on net, just that a tradeoff was made. How many person-days per year were freed up by less labor-intensive manioc handling? Has anyone bothered to count the hours lost to laborious traditional manioc-processing, to compare them with the burden of consuming too much cyanide? How many of us, knowing that convenience foods probably lower our lifespans relative to slow foods, still eat them because they're ... more convenient?

How many people didn't starve because manioc was available and would grow where and when other things wouldn't?

If this is the best we can do for how poorly reason can perform, reason seems pretty great.

Second, we're not actually comparing reason to tradition - we're comparing changing things to not changing things. Change, as we know, is bad. Sometimes we change things anyway - when we think it's worth the price, or the risk. Sometimes, we're wrong.

Third, the actually existing Portuguese and Africans involved in this experiment weren't committed rationalists - they were just people trying to get by. It probably doesn't take more than a day's reasoning to figure out which steps in growing manioc are really necessary to get the calories palatably. Are we imagining that someone making a concerted effort to improve their life through reason would just stop there?

This is being compared with many generations of trial and error. Is that the standard we want to use? Reasoning isn't worth it unless a day of untrained thinking can outperform hundreds of years of accumulated tradition?

It gets worse. This isn't a randomly selected example - it's specifically selected as a case where reason would have a hard time noticing when and how it's making things worse. In this particular case, reason introduced an important problem. But life is full of risks, sometimes in ways that are worse for traditional cultures. Do we really want to say that reasoning isn't the better bet unless it outperforms literally every time, without ever making things locally worse? Even theoretically perfect Bayesian rationality will sometimes recommend changes that have an expected benefit, but turn out to be harmful. Not even tradition meets this standard! Only logical certainties do - provided, that is, we haven't made an error in one of our proofs.

We also have to count all the deaths and other problems averted by reasoning about a problem. Reasoning introduces risks - but also, risks come up even when we're not reasoning about them, just from people doing things that affect their environments. There's absolutely no reason to think that the sort of gradual iteration that accretes into tradition never enters a bad positive feedback loop. Even if you think modernity is an exceptional case of that kind of bad feedback loop, we had to have gotten there via the accretion of premodern tradition and iteration!

The only way out is through. But why did we have this exaggerated idea of what reason could do, in the first place?

11 comments

Comments sorted by top scores.

comment by Raemon · 2019-07-22T20:18:36.057Z · score: 14 (6 votes) · LW · GW

I'm curating this post alongside Scott's previous Book Review: The Secret of Our Success [LW · GW].

One object level reason to curate both is that Scott's post highlights some important details and questions about how culture and reason interface, and this one offers a concrete, non-mysterious response that I found usefully clarifying.

There's a meta level thing where I think it's sort of using for LW readers who don't keep up to date as much about how ongoing conversations played out, to have a good repository of the highlights of that conversation.

comment by Raemon · 2019-06-18T19:55:03.614Z · score: 6 (3 votes) · LW · GW

I like this post. I found it a helpful yet simple, clear-cut response to the recent suite of "how to think about reason and tradition" discussion.

comment by Raemon · 2019-06-18T04:15:48.515Z · score: 5 (2 votes) · LW · GW

I assumed this was supposed to be a linkpost and updated it, apologies if not.

comment by Stuart_Armstrong · 2019-06-18T12:01:53.046Z · score: 4 (2 votes) · LW · GW

Post was good, but I'd recommend adding an introductory paragraph to the link on LessWrong.

comment by Mary Chernyshenko (mary-chernyshenko) · 2019-06-24T19:47:47.591Z · score: 3 (2 votes) · LW · GW

I don't quite understand. Perhaps "reasoning" got it worse than "tradition" did. Then people learned what was wrong. And now they still insist on doing it not according to "tradition"? How is it different at all from setting up a new tradition and not bothering anymore?

comment by Dr_Manhattan · 2019-06-19T14:34:39.633Z · score: 3 (2 votes) · LW · GW
Second, we're not actually comparing reason to tradition - we're comparing changing things to not changing things. Change, as we know, is bad.

Request for clarification: isn't "reasonable solution" always a "change" when compared to preexisting tradition?

comment by dxu · 2019-06-19T18:52:10.883Z · score: 1 (2 votes) · LW · GW

Did you read the linked article?

comment by Dr_Manhattan · 2019-06-20T19:10:45.161Z · score: 2 (1 votes) · LW · GW

Do you mean Zvi's "Change is bad"?

comment by habryka (habryka4) · 2019-06-18T20:02:40.981Z · score: 3 (2 votes) · LW · GW

I also liked this post. Would be happy to copy its full text over to LW if you are interested, since I can imagine including it in a bunch of future sequences (and also a bunch of other things, like making it discoverable in search)

comment by Benquo · 2019-06-19T01:53:16.664Z · score: 14 (5 votes) · LW · GW

Copied it over myself, thanks for the suggestion

comment by Jimdrix_Hendri · 2019-07-24T02:35:05.252Z · score: 1 (1 votes) · LW · GW

We should select comparisons aimed at the getting the best result, not to make things easy on ourselves:

What if the Europeans had thought: "Hmm. The natives are following a procedure we don't understand with regard to casava. Their explanation doesn't make sense according to our own outlook, but it is apparent that they have a lot of experience. It may pay to be prudent rather than disregarding their rituals as superstitions."

Had the Europeans taken this attitude, they may have discovered the toxicity of yucca, experimented with imitating the leaching procedure or, at least, have introduced it slowly, since reliance on a monoculture exposes a population to other risks as well. In either case, wouldn't the Africans likely have been better off?

In case this seems like a special case, consider the impact of the introduction of potatoes to Ireland. As for the long-term, unquantifiable dangers of introducing genetically modified species into the environment on a massive scale; only time will tell.