Some potential dangers of rationality training

Perhaps the sunk cost fallacy is useful because without it you're prone to switch projects as soon as a higher-value project comes along, leaving an ever-growing heap of abandoned projects behind you.

Sunk costs might in fact be useful heuristic for many people. But if rationality gets you in trouble in this area, use more rationality! - after all, once you know about this switching behavior, you can consciously counteract it with strategies (precommitment, etc).

What you're pointing to is that a little rationality can be a dangerous thing.

One view is that people are able to vary their biases, so that they're more biased when the bias is helpful and less when it gets in the way. For instance, Armor & Sackett (2006) found that people were more optimistic about how they'd perform on hypothetical tasks but relatively accurate in predicting how they'd perform on real tasks that they were actually about to do (when they'd have to face the reality of their actual performance). That's consistent with near/far theory - in far mode people can afford to have more biased, flattering images of themselves.

With sunk costs, one of the most relevant theories is Gollwitzer's theory about deliberative and implemental mindsets. He argues that when people are faced with a decision, they tend to go into a deliberative mindset which allows them to view the options relatively accurately so that they can make a good decision. But once they've made a decision, their focus shifts to how to implement the chosen course of action - planning how to get it done & figuring out how to overcome any obstacles. Thoughts about whether the course of action really is desirable (or feasible) are distractions and potential demotivators, so people in an implemental mindset mostly don't think about that, and are overly optimistic when they do.

If rationality training de-compartmentalizes people and makes it harder to switch between different mindsets then that could be a disadvantage. It could be harder to carry out a project if you keep putting yourself back in the mindset of deciding whether or not it's worth doing.

But it doesn't have to happen that way - rationality training could train you to respond appropriately to these changes in circumstances. For example, one alternative way of thinking in situations where people are prone to the sunk cost fallacy is to ask yourself if now is a good time to re-evaluate your course of action. Maybe the answer is no, it's not a good time - you're in the middle of doing things and it would be a distraction, or you put a lot of thought into your original decision and wouldn't have the perspective to re-evaluate it now that you're immersed in the project. In that case you can keep going with the project and trust your original decision, no need to justify it. But sometimes the answer is yes, it is a good time to re-evaluate - circumstances have changed, or you have new information which might have changed your decision if you knew it from the start, or you're considering whether to make some new commitment which you hadn't expected. If it is a good time to re-evaluate, then take a step back and take a fresh look at the decision, taking into account all that you've learned - no need to be beholden to your old decision.

Armor, D. A., & Sackett, A. M. (2006). Accuracy, error, and bias in predictions for real versus hypothetical events. Journal of Personality and Social Psychology, 91, 583-600.

Our brains obviously have things like the sunk cost fallacy and all the various heuristics built in for a reason. We don't have an explicit utility function in our brain, and so we have all kinds of hacked on solutions.

For example the sunk cost fallacy in our brain is there to prevent us from giving up on tasks prematurely. Our brains need this tendency, because we also have a boredom tendency built in. An AI would simply calculate the expected value of working on each of the things it could be doing. The boredom tendency would be countered by an understanding of diminishing marginal returns (or not at all, for an unconscious AI), and the sunk cost fallacy would be unnecessary.

You need to be explicit that all these heuristics and biases are not overall good things, but simply hacked on aspects of human psychology.

I personally feel that learning about rationality has greatly improved my life, but 1. so do New Age advocates and 2. that's just one data point.

...many people's lives aren't much improved by rationality training.

The outlook for rationality training may be brighter than you think. I've been looking at a number of materials aimed at businessmen who want greater success, and a lot of the techniques described and used by successful businessmen bear a strong resemblance to the sort of things you read on Less Wrong. For example, consider this passage in Peter Drucker's book The Effective Executive, describing Alfred P. Sloan, a legendary CEO of GM:

Alfred P. Sloan is reported to have said at a meeting of one of his top committees: "Gentlemen, I take it we are all in complete agreement on the decision here." Everyone around the table nodded assent. "Then," continued Mr. Sloan, "I propose we postpone further discussion of this matter until our next meeting to give ourselves time to develop disagreement and perhaps gain some understanding of what the decision is all about."

Sloan was anything but an "intuitive" decision-maker. He always emphasized the need to test opinions against facts and the need to make absolutely sure that one did not start out with the conclusion and then look for the facts that would support it. But he knew that the right decision demands adequate disagreement.

So here we have talk of the need to consider multiple alternatives, the importance of not just getting one side of a story, the need to test hypotheses against reality, and the importance of not starting with the bottom line... all familiar topics in our community.

I think this is a discussion about what the best order is in which debiasing should occur.

Project management seems to be an implementation of debiasing strategies and they first teach you to make more accurate predictions and then later teach you how to prevent the sunk cost fallacy from not cancelling a failing project.

Because of this i think debiasing should occur in a kind of logical order, one that prevents someone from cancelling all projects due to a good grasp of sunk cost and a bad grasp of utility calculations.

Another potential negative is a reduced capacity for strategic self-deception.

Perhaps the sunk cost fallacy is useful because without it...

This sounds like a fake justification. For every justification of a thing by pointing to the positive consequences of it, one can ask how much better that thing is than other things would be.

I expect evolution to produce beings in local optima according its criteria, which often results in a solution similar to what would be the best solution according to human criteria. But it's often significantly different, and rarely the same.

For every systematic tendency to deviate from the truth I have, I can ask myself the leading question "How does this help me?" and I should expect to find a good evolutionary answer. More than that, I would expect to actually be prone to justifying the status quo according to my personal criteria, rather than evolution's. Each time I discover that what I already do habitually is best, according to my criteria, in the modern (not evolutionary) environment, I count it as an amazing coincidence, as the algorithms that produced my behavior are not optimized for that.

Perhaps the sunk cost fallacy is useful because without it you're prone to switch projects as soon as a higher-value project comes along, leaving an ever-growing heap of abandoned projects behind you.

There's actually some literature on justifying the sunk cost fallacy, pointing to the foregone learning of switching. (I should finish my essay on the topic; one of my examples was going to be 'imagine a simple AI which avoids sunk cost fallacy by constantly switching tasks...')

EDIT: you can see my essay at http://www.gwern.net/Sunk%20cost

I would think existing irrationality's superiority to rational hacks would be covered by Algernon's Law.

It occurs to me that I have internalized the value of a group that I identify with, the value of truth from my association with the physicists and engineers guild. Interestingly, I accept this value uncritically, it is at this point in my life, an axiom.

I probably suffer as much as anybody from making myself more rational in pursuit of this value I accept uncritically. I am prone to depression, akrasia, and wonder constantly at why I should accept any of my values seriously since I knoew the truth is they are imposed on me by evolution, they are not really "mine." All except the value of the truth, which I seem to accept uncritically.

Would I be better off if I accepted a value that was more in line with human nature uncritically? The value of thinking I was sexy (and therefore working on being sexier), or the value of controlling people around me? You know, I probably would.

But that would go against my most deeply (and uncritically) held value of truth, so I am very unlikely to move in that direction. Hmmm.

Rationalists look at the winner and do what they do. Rationalists do not bemoan their new rationality because the sunk-cost fallacy leaves abandoned projects.

That said, it may be rational to remember that finishing something has huge experience value the next time you use sunk costs fallacy to defect from a project.

Perhaps the sunk cost fallacy is useful because without it you're prone to switch projects as soon as a higher-value project comes along, leaving an ever-growing heap of abandoned projects behind you.

That sounds an awful lot like irrational behavior. If only we had some group who liked thinking about ways to overcome exactly that sort of thing, and who even had enough success (by hypothesis) that they'd already eliminated some parts of human irrationality.

:D

I think there is a valid issue in our biases and irrational behavior being dependent on each other. My typical example is the planning fallacy combined with the fact that we feel loss more strongly than gain (and that pain can quickly gets higher than pleasure). Since it's much easier to feel pain than pleasure, by trying to maximize pleasure, we should be tempted to not take risks. But if in addition with underestimate risks, then we'll end up taking some risks.

More generally, I think it comes from evolution not being perfect design. It'll grow brains full of bugs. But then, another bug that lower the important of a previous bug will have a net advantage, and will spread. But once you have the two bugs, a fix for just one will have a net disadvantage, and will not spread.

It's something that occurs in computer programs too - programmers among us will remember cases in which removing a bug allowed to discover another bug, which was before compensating the first bug. Like a bug that was reverting a sort order, and another bug which another part in the program was reverting it again, leading to the correct order. Or a "off by one" in one direction at one point, compensated by an "off by one" in the other direction at another.

But from all that it comes that debiasing or rationality is not harmful in itself, it's the intermediate state where you're partly debiased which is unstable. Of course, it creates lots of issues in practically trying to debias yourself (or someone).

To quote Final Fantasy VI: "How will we know unless we try for ourselves?"