Humans are (mostly) metarational

post by Yair Halberstadt (yair-halberstadt) · 2024-10-09T05:51:16.644Z · LW · GW · 6 comments

Contents

6 comments

At least in the early days of LessWrong this community was mostly focused on all the ways Man's reasoning failed him, the myriad ways we are irrational. What foolish humans, making mistakes even trivial mathematics can show you are wrong!

Evolution aims to maximise number of offspring. But people don't have a direct drive for that because encoding that into a human brain is too complex. Instead we like sex, which in most environments is a pretty good proxy, albeit with the occasional extremely embarrassing screwup. In a sense we have a meta-drive for offspring, which manifests into myriad actual drives which leads to a good enough approximation of the meta-aim.

In the same way, people aren't rational, but if a rational person were designing humans they might very well program us to make the mistakes we do, all in the aim of rationality. You see, we might not be rational, but we are meta-rational!

You decide to build a shed. You're a novice at carpentry but you set to it with gusto. This shouldn't take too long! 2 months in though you're still only half way through - everything seems to be taking far longer than you expected!

If you had thought from the start it would take 2 months to build the shed you wouldn't have bothered. Now you think it will still take another 2 months but you're reluctant to abandon the project. Foolish human! Have you never heard of the sunk cost fallacy?

Except it isn't!

Now I agree that there are contrived cases where the sunk cost truly is a sunk cost, and humans still make the same mistake. But in the real world the fallacy rarely ever applies, and factoring in the amount you had to pay to give you the opportunity you have right now is a good rule of thumb when considering whether to take the opportunity. Given that we are foolish humans, and will always make mistakes, committing the sunk cost fallacy is the right mistake to make.

You're walking through an empty park when a tissue falls out of your pocket and blows a few metres away. You walk over to it, bend down and pick it up, and put it in a nearby garbage can. On the way you pass dozens of other pieces of trash, but leave them where they are.

Are you a proponent of the Copenhagen interpretation of ethics? No? Then picking up any of the pieces of trash makes the same impact on the environment, so you might as well pick up the one that's closest to you.

In fact why pick it up at all? If you think cleaning the park is a worthwhile use of your time, you should clean it whether or not you dropped the tissue. If it isn't worthwhile, why clean it just because you've dropped a tissue?

Now I agree that you are acting irrationally in this particular case. But a moral principle that you fix what you broke is a pretty good principle to have, as it discourages people from breaking things. Maybe next time you'll be more careful to stop things falling out your pockets. Whereas if you just ignored whatever mess you make as not worth your time, you'll soon become lackadaisical about your impact on the environment and start dropping trash on the floor willy nilly.

Chesterton's fence states that you shouldn't remove a seemingly useless fence from a field until you understand why it was put there in the first place, and ascertained that it isn't actually necessary.

In the same way teaching people about all sorts of logical fallacies is dangerous if you don't make sure they understand why we make these logical fallacies in the first place. "Oh yes", the new ubermensch cries, "we should ignore our sunk costs", and then proceeds to divorce his wife of 10 years and have sex with the pretty girl he saw at the bar last night. "What's done is done", he says when his ex-wife can't make rent and his kids end up homeless, "there's no more reason for me to help you over any of the other homeless people - have 10 dollars, and don't spend it on alcohol or drugs".

Exploring logical fallacies should be done hand in hand with working out both why we make these mistakes, the exact boundaries where the fallacy truly applies, and what rules of thumb we should replace it with instead.

Now if you'll excuse me, I just remembered I dropped a tissue when I went to Jamaica on vacation last week, I'm off to pick it up.

6 comments

Comments sorted by top scores.

comment by AnthonyC · 2024-10-09T14:37:17.757Z · LW(p) · GW(p)

I think this post is missing the major part of what "metarational" means: acknowledging that the kinds of explicit principles and systems humans can hold in working memory and apply in real time are insufficient for capturing the full complexity of reality, having multiple such principles and systems available anyway, and skillfully switching among them in appropriate contexts.

We don't have a meta-drive for offspring (although many people do explicitly want children at certain points in their lives). We (most of us) have a general drive to like sex, which in a specific context used to pretty reliably result in children, but we do not re-examine that drive and apply it only in that context. We re-use that drive in multiple other contexts for other purposes, but those are also mostly unexamined by most people.

We don't have a meta-drive for honoring commitments like marriage. Instead we have social institutions and verbally-expressible moral principles we mutually agree to try to abide by and enforce. These evolve by a mix of unconscious selection effects and conscious reflection. The latter is usually led or spread by high-status, unusually-metarational members of the community, whose behaviors others choose to mimic without themselves possessing the metarational capability to skillfully re-derive, re-direct, and re-apply the process that generated them.

Similarly, the point about trash also ignores the larger context. Picking up my own trash has much less relationship to disgust, or germs, than picking up other people's trash. I have gone to parks to pick up trash in general, but I do it with gloves and garbage bags and pickers and hand sanitizer and disinfecting wipes. I don't fly back to Jamaica for a dropped tissue because of the larger context in which the costs, both personal and environmental, are greater to go than not. The mistake is made, a sunk cost in the true sense of "not worth continuing to invest in."

This also is not an example of Copenhagen ethics. That would be, "You dropped a tissue. If you pick up up, then all the other trash is also your fault unless you pick it up, too. Doubly true if you pick up the candy bar wrapper that blew onto the ground next to the trash can." The metarational solution to trash-in-parks would be "It's a good norm to have, to pick up trash when it's easy, because this generalizes well into a low-effort community-level solution to the large majority of trash. Then we can more skillfully direct scarce, expensive, in-depth clean-up efforts when they're really needed and the norm is proving (or will predictably prove) insufficient." Many but not all humans have this norm without understanding it - that's part of why the old SSC post on Newtonian Ethics works as humor. But not all, which is why there is still a trash problem and insufficient coordination power in society to consistently and cheaply implement better solutions.

In other words, humans can be metarational, and make effective use of rational systems to manage pre-rational parts of ourselves. But those systems are still rational, and the parts are still pre-rational, and most people don't develop these skills.

Replies from: yair-halberstadt, yair-halberstadt
↑ comment by Yair Halberstadt (yair-halberstadt) · 2024-10-09T17:00:06.721Z · LW(p) · GW(p)

Similarly, the point about trash also ignores the larger context. Picking up my own trash has much less relationship to disgust, or germs, than picking up other people's trash.

Agreed, but that's exactly the point I'm making. Once you apply insights from rationality to situations outside spherical trash in a vacuum filled park you end up with all sorts of confounding affects that make the insights less applicable. Your point about germs and my point about fixing what you break are complimentary, not contradictory.

↑ comment by Yair Halberstadt (yair-halberstadt) · 2024-10-09T16:57:06.256Z · LW(p) · GW(p)

I think this post is missing the major part of what "metarational" means: acknowledging that the kinds of explicit principles and systems humans can hold in working memory and apply in real time are insufficient for capturing the full complexity of reality, having multiple such principles and systems available anyway, and skillfully switching among them in appropriate contexts.

This sounds to me like a semantic issue? Metarational isn't exactly a standard AFAIAA, (I just made it up on the spot), and it looks like you're using it to refer to a different concept from me.

Replies from: AnthonyC
↑ comment by AnthonyC · 2024-10-09T21:39:08.407Z · LW(p) · GW(p)

I first encountered the term from David Chapman's work (as described in the blog/metabook I linked) and was under the impression he coined the term, so that's what I assumed you were referring to. If there is another definition you're using, it might be good to add a note explaining what you mean by it. So yes, there is a semantic issue here contributing to why I misunderstood the post. I don't quite know what your intended definition is, so take my comments in that context.

In any case, I highly recommend Chapman's work on metarationality if you're unfamiliar with it. I think it answers a lot of the questions you raise here. He has noticed the skulls. In light of that, I don't think there's just a semantic issue here. I think there is a natural grouping of non-rational natural selection, pre-reational human drives, rational systematic thinking, and meta-rational skillful use of systematic thinking, and that a lot of the discussion in this post goes back and forth between pre-rational and meta-rational without distinguishing between them. This is something that people do all the time, because they do look the same until you've actually succeeded in developing the meta-rational skills needed to understand the difference (or until you've had the difference pointed out to you, if you developed the skills without needing the ontology).

comment by Viliam · 2024-10-09T15:01:13.866Z · LW(p) · GW(p)

"Oh yes", the new ubermensch cries, "we should ignore our sunk costs", and then proceeds to divorce his wife of 10 years and have sex with the pretty girl he saw at the bar last night.

Is this supposed to be a good or bad example? I imagine that a man who has a few kids with his first wife, then divorces her to marry a younger one, and also has a few kids with her, would be quite successful from the evolutionary perspective.

This way we can defend both the sunk costs and ignoring the suck costs as "meta-rational".

Replies from: yair-halberstadt
↑ comment by Yair Halberstadt (yair-halberstadt) · 2024-10-09T17:01:25.091Z · LW(p) · GW(p)

I've been convinced! I'll let my wife know as soon as I'm back from Jamaica!