Social Impact, Effective Altruism, and Motivated Cognition

Seems kind of obvious? We've got plenty of people running around saying "Perhaps you overestimate your importance".

Well, of course everyone's going to be thinking they're doing the most effective thing, because they chose to do it based on the fact that it seemed like it'd be the most effective. (Hopefully.)

Absent specific, non-hypothetical examples and empirical evidence, I find this question hard to think or reason about. I have not noticed this problem myself, so I cannot recollect any such examples from my own experience.

I note this as another example of the "Politics is the Mind-Killer" is the Mind-Killer meta-problem. The point of the "Politics is the Mind-Killer" essay (and a correct one) is that we should avoid using tribal-loyalty triggering examples when discussing issues such as mathematics, cognitive biases, and logic that are not fundamentally about issues that touch on tribal identity. Triggering tribal loyalty unnecessarily is bad pedagogy.

However, "Politics is the Mind-Killer" is not a general excuse for avoiding discussion of politics or other matters that touch on tribal or personal identity when those matters are exactly the subject at hand. If rationality cannot come to epistemically correct and instrumentally useful results despite the blinders of tribal loyalty and personal identity, it is weak, impotent, and irrelevant.

The claim of this post is that people have cognitive biases based on personal identity that cause them to reach incorrect conclusions about the relative efficacy of different altruistic actions. If this group is truly rational, then we should be able to calmly discuss the actual issues and resolve them factually or at least work them down to the point where we realize some of us have different fundamental values. For instance, I would not expect us to resolve the question of whether to value future people equally with currently living people, but I would expect us to be able to make plausible estimates as to the number of QALYs (quality adjusted life years) per dollar of different interventions, or at the very least to figure out what information is missing and needs to be collected to answer the question. If we can't do that, if we can't even talk about that, then I have to question what the point of the entire LessWrong project actually is.

Jonah has recently been attempting to persuade Eliezer that Eliezer's comparative advantage is not the FAI research that he is currently doing but instead doing (more) evangelism. Now we have a post explaining how status signalling related motivated cognition can cause people to overestimate the value of the altruistic efforts that they happen to personally have chosen. This is almost certainly true---typical human biases work like that in all similar areas so it would be startling to find that an activity so heavily evolutionarily entangled with signalling motives was somehow immune! I feel it is important, however, to at least make passing acknowledgement of the fact that this exhortation about motivated cognition is itself subject to motive.

Jonah himself acknowledges people are more likely to suggest motivated cognition as something that the other guy might be suffering from than to apply it to themselves. While in this case there is no overt claim like "... and therefore you should believe the guy I was arguing with is biased and so agree with me instead" and I don't believe Jonah intends anything so crude, the recent context does change the meaning of any given post---at least the context and expected social influence of a post influences how I personally evaluate contributions that I encounter and I currently do not label that habit of reading a bug.

To be clear the pattern "significant argument --(short time)--> post by one participant which points out a bias that the other participant may have" isn't (always) a cause to reject the post. This one isn't particularly objectionable (a tad obvious but that's ok in discussion). Nevertheless I suggest that for the purpose of making the actual explicit point without distraction it may usually be best to keep such posts in draft form for a couple of weeks and post them later when the context loses relevance. Either that or include a lampshade or disclaimer regarding the relevance to the existing conversation. There is something about acting oblivious that invites scepticism.

For now, I would encourage effective altruist types to take pride in being self-skeptical when it comes to favorable assessments of their potential impact relative to other effective altruist types, or relative to people outside of the effective altruist community.

Yes, I find it remarkable how EAs tend to think their work is obviously vastly more important than that of "non-EAs" (as if such a thing were even well defined). There's not a lot new under the sun, and like most movements, EA is largely a recycling and recombination of things other people have been doing since the dawn of civilization. It may be a good combination, but little in EA is really unique to EA.

All of that said, I think a big reason people think their own work dominates that of others is because they have different values from other people. It's perfectly possible for lots of people to be doing lots of things that are each optimal relative to their own values. You might (perhaps correctly) point out that most EAs have values more similar to each other than my values are to theirs, so my point may apply less broadly than I suggested.

I’m not aware of any overt public discussion of [this issue].

Katja's recent post?

ETA: It's not the same issue, I didn't read either of you properly. But perhaps the same ballpark.

This feeds back into the earlier discussion about the flexibility of donations vs careers. Hot money donors who switch to apparently better alternatives face less in the way of costs to encourage rationalization. They still have some pressures along these lines, since they don't want to say their previous donations were foolish, and would probably like to be able to point to some new evidence or justification for the switch, but the problem would certainly seem to be smaller.

This seems potentially connected to Goodhart's law.

Good post.

One obvious problem with trying to overcome bias by means of "self-skepticism" is that many of the biases we try to overcome also shape our skeptical attitudes. Here, as elsewhere, adopting the outside view is probably more effective than attempting to find flaws in one's thinking "from the inside".

A possible application for the case at hand is this. Consider the reasons why you chose to work on a particular cause, instead of the many other causes you could have worked for. Are these reasons still the same ones that you currently regard as valid? If not, you should increase your credence in the hypothesis that you might be working on the wrong cause, relative to your present beliefs and values, since you might have reached this view as a result of motivated cognition.

I will give an example from my own personal life. I chose to become a vegetarian many years ago, out of concern for the animals that were suffering (in expectation) as a result of my dietary choices. However, as I read and reflected more on the issue, I came to realize that the indirect effects on other sentient beings where much more relevant than the direct effects on the animals themselves. In particular, I thought that the effects of spreading concern for all sentience by abstaining from eating animals might shape the choices made by our descendants with power to create astronomical amounts of suffering in the Universe. However, this should make me suspicious. Was I really lucky that my new reasons just so happen to vindicate the diet to which my old reasons had caused me to become deeply attached? Or is this instead the result of motivated cognition on my part? I am still a vegetarian, but for arguments of this sort I am less convinced that this is what morality requires of me.

The effective altruist movement is in early stages, and it’s important to arrive at accurate conclusions about effective philanthropy as fast as possible. At this stage in time, it may be that the biggest contribution that members of the community can make is to engender and engage in an honest and unbiased discussion of how best to make the world a better place.

This philosophy strikes me as remarkably compatible with that of Leverage Research. Are you in contact with those folks at all?