Community bias in threat evaluation

Environmentalists care about direct and specific threats. The rest of them seem to care about possible loss of something we call "Friendliness" here.

HBD people believe that different cultures encourage different behaviors, which creates different selection pressures, and the differences gradually get encoded in genes. Which means that people from other cultures are for us like 99% humans + 1% Pebblesorters. We should optimize for our values. -- In addition to this, there are also direct specific threats, e.g. violence.

Leftists point out that the Invisible Hand of Market is not Friendly, therefore it may optimize for things we consider horrible; just like evolution, and for the same reason. This part seems obvious to me; I only object connotationally that institutions in general are not Friendly, including the institutions created by leftists.

Catholics have a model of the world where Friendliness comes from God, and humans themselves are not Friendly. We are broken; we don't automatically optimize for things we actualy like to have. Azathoth is in us. Ignoring God's commands -- and sexual behavior is where doing so is most tempting -- means giving up the Friendliness and letting Azathoth optimize for its own purposes.

LessWrongians point out that an AI could be a powerful threat to Friendliness, that it is likely to be the threat, and that it may be too powerful to be stopped once this becomes obvious.

To sum up the differences: Catholics pretend they have an external source of Friendliness and try to preserve the connection with it. HBD people worry about external sources of Unfriendliness. Leftists worry about existing forces that slowly but persistently optimize away from Friendliness. LessWrongians worry about a new very fast and powerful source we may create.

My opinion is that Friendliness is fragile and in danger. Azathoth and the hypothetical Unfriendly AI will optimize away from it. There is no equivalent power optimizing towards it; we only have inertia on our side. Unless we develop the Friendly AI or become greatly more rational ourselves (unless either MIRI or CFAR reaches their goal), we are doomed in long term. But of course some other disaster can destroy us even faster; we should not forget about immediate threats.

I think all these groups except for Catholics have a valid point (and even Catholics have some useful heuristics; the problem is that their other heuristics are harmful, and their model is hopelessly wrong) and I have no idea how to evaluate which one deserves most attention. My reason to choose LW is that it is the smallest group, so its concerns are most likely to be ignored. Also, in long term only LW has a satisfying solution (although this itself does not prove that someone else shouldn't get priority in short term), the other groups are merely slowing down the inevitable.

At least some of the disagreement shrinks when you clarify "greatest risk" :

• Is it a problem for rich first-worlders (dysgenics, maybe), poor first-worlders (immigration), poor third-worlders (global warming, pollution)?

• Is it a likely but "mild" problem (dysgenics, sexual immorality, growing inequality), or an unlikely/uncertain but catastrophic problem (AI, grey goo) ?

There's probably still a lot of actual disagreement about facts left (for example, how likely is AI, whether God punishes/rewards us), but I think the bulk of the "disagreement" boils down to "bad for different people" and "bad in different ways".

Yes, it is indeed a common pattern.

People are likely to get agitated about the stuff they are actually working with, especially if it is somehow entangled with their state of knowledge, personal interests and employment. Belief that we are the ones to save the world, really helps to find motivation for continuing their pursuits (and helps fund-raising efforts, I would reckon). It is also a good excuse to push your values on others (Communism will save the world from our greed).

On the other hand, I don't think it is a bad thing. That way, we have many little small groups, each working on their small subset of problem space when also trying to save the world from the disaster, which they perceive to be the greatest danger. As long as response is proportional to actual risk, of course.

But I still agree with you that it is only prudent to treat any such claims with caution, so that we don't fall into a trap of using data taken from a small group of people working at Asteroid Defense Foundation as our only and true estimates of likelihood and effect of an asteroid impact, without verifying their claims using an unbiased source. It is certainly good to have someone looking at the sky from time to time, just in case their claims prove true, though.

Actually I don't think you're right. I don't think there's much consensus on the issue within the community, so there's not much of a conclusion to draw:

Last year's survey answer to "which disaster do you think is most likely to wipe out greater than 90% of humanity before the year 2100?" was as follows:

Pandemic (bioengineered): 272, 23% Environmental collapse: 171, 14.5% Unfriendly AI: 160, 13.5% Nuclear war: 155, 13.1% Economic/Political collapse: 137, 11.6% Pandemic (natural): 99, 8.4% Nanotech: 49, 4.1% Asteroid: 43, 3.6%

I think this is a bad thing, and that we need to try to avoid it.

I also think that we are biased not just by wanting the stuff we are good at to be the most important stuff, but also by wanting to agree with the community.

Here is a small step we can take in the right direction.

If you are one of the 15 people who down voted the following post, stop for a minute and ask yourself why. I am not saying either way whether or not the post deserved to be down voted. However, I would not be surprised if several people down voted it for a bad reason.

http://lesswrong.com/r/discussion/lw/ji0/the_first_ai_probably_wont_be_very_smart/

How much of the bias is priming? What if you provided a list of likely disasters? If you raise the issue, you can get most people to admit that a pandemic scenario is more "serious".

"Greatest risk to society/humanity" might also be interpreted different ways. One person may think of existential risk, while another might think about risk to cultural values.

It seems these groups exist, in large part, as an effect of their beliefs about the biggest risks. You're not afraid of global warming and pollution because you are an environmentalist, rather you are an evironmentalist because of your fear of global warning and pollution.

That said, I'm not sure what your point is. I'm sure there are many in each group who haven't done the math themselves and are just following like sheep. But it is the same regardless of what we are talking about...it certainly isn't specific to evalutating threats to humanity. Just groupthink and half a dozen other biases at play.

The other thing is that groups may not be focusing on the largest existential threat at any given time. Instead they might be spending time on a particular issue that has come to the forefront.

Conservative Christianity, for instance, is dealing with homosexuality right now. But that is really just a pawn in a much larger eschatological endgame. Homosexuality isn't really that big a threat to Christians. Hell is a bigger threat.

In characterizing this trend, it seems as though you are assuming that membership in these various communities is mutually exclusive. However, this doesn't have to be the case. For example, a person may be both a Catholic and a leftist. Thus, a good follow-up question might be: To what extent does leftist politics and Catholicism have an impact on a person's evaluation of risk. For example, one could compare Pope Benedict with Pope Francis, who are both Catholic, with each other and potentially conclude that income inequality/exploitation influences Francis' conceptualization of risk to society to a greater degree than abortion or sexual immorality.

Definitely brings déformation professionnelle to mind for me, I'm not sure if there is anything in there about the community aspects of it though.

This is not exactly true. Here is similar statistics ("what is a greatest danger in 21 century ") from astronomy forum :

Nuclear war - 19.8 % Resources depletion - 10 % Nothing, there is no threat - 9,4% Asteroid - 8,4 % Lack of interest for life - 6.3% Overpopulation - 5.5% AI - 4.5% Simulation shutdown - 3.9% Solar flare - 3.7% Grey goo - 2.4% Biological weapons - 3.5% Supervulcano - 2.6%

(rest goes to "other" ).