Rational Terrorism or Why shouldn't we burn down tobacco fields?

All this AI stuff is an unnecessary distraction. Why not bomb cigarette factories? If you're willing to tell people to stop smoking, you should be willing to kill a tobacco company executive if it will reduce lung cancer by the same amount, right?

This decision algorithm ("kill anyone whom I think needs killing") leads to general anarchy. There are a lot of people around who believe for one reason or another that killing various people would make things better, and most of them are wrong, for example religious fundamentalists who think killing gay people will improve society.

There are three possible equilibria - the one in which everyone kills everyone else, the one in which no one kills anyone else, and the one where everyone comes together to come up with a decision procedure to decide whom to kill - ie establishes an institution with a monopoly on using force. This third one is generally better than the other two which is why we have government and why most of us are usually willing to follow its laws.

I can conceive of extreme cases where it might be worth defecting from the equilibrium because the alternative is even worse - but bombing Intel? Come on. "A guy bombed a chip factory, guess we'll never pursue advanced computer technology again until we have the wisdom to use it."

This just puts me off being utilitarian to be honest.

Understandably so, because the outside view says that most such sacrifices for the greater good end up having been the result of bad epistemology and unrealistic assessments of the costs and benefits.

Strong rationality means that you'd be able to get away with such an act. But strong rationality also means that you generally have better methods of achieving your goals than dubious plans involving sacrifice. When you end thinking you have to do something intuitively morally objectionable 'for the greater good' then you should have tons of alarm bells going off in your head screaming out 'have you really paid attention to the outside view here?!'.

In philosophical problems, you might still have a dilemma. But in real life, such tradeoffs just don't come up on an individual level where you have to actually do the deed. Some stock traders might be actively profiting by screwing everyone over, but they don't have to do anything that would feel wrong in the EEA. The kinds of objections you hear against consequentialism are always about actions that feel wrong. Why not a more realistic example that doesn't directly feed off likely misplaced intuitions?

Imagine you're a big time banker whose firm is making tons of money off of questionably legal mortgage loans that you know will blow up in the economy's face, but you're donating all your money to a prestigious cancer research institute. You've done a very thorough analysis of the relative literature and talked to many high status doctors, and they say that with a couple billion dollars a cure to cancer is in sight. You know that when the economy blows up it will lead to lots of jobless folk without the ability to remortgage their homes. Which is sad, and you can picture all those homeless middle class people and their kids, depressed and alone, all because of you. But cancer is a huge bad ugly cause of death, and you can also picture all of those people that wouldn't have to go through dialysis and painful treatments only to die painfully anyway. Do you do the typically immoral and questionably illegal thing for the greater good?

Why isn't the above dilemma nearly as forceful an argument against consequentialism? Is it because it doesn't appeal in the same way to your evolutionarily adapted sense of justice? Then that might be evidence that your evolutionarily adapted sense of justice wasn't meant for rational moral judgement.

Here is an interesting comment related to this idea:

What I find a continuing source of amazement is that there is a subculture of people half of whom believe that AI will lead to the solving of all mankind's problems (which me might call Kurzweilian S^) and the other half of which is more or less certain (75% certain) that it will lead to annihilation. Lets call the latter the SIAI S^.

Yet you SIAI S^ invite these proponents of global suicide by AI, K-type S^, to your conferences and give them standing ovations.

And instead of waging desperate politico-military struggle to stop all this suicidal AI research you cheerlead for it, and focus your efforts on risk mitigation on discussions of how a friendly god-like AI could save us from annihilation.

You are a deeply schizophrenic little culture, which for a sociologist like me is just fascinating.

But as someone deeply concerned about these issues I find the irrationality of the S^ approach to a-life and AI threats deeply troubling. -- James J. Hughes (existential.ieet.org mailing list, 2010-07-11)

Also reminds me of this:

It is impossible for a rational person to both believe in imminent rise of sea levels and purchase ocean-front property.

It is reported that former Vice President Al Gore just purchased a villa in Montecito, California for $8.875 million. The exact address is not revealed, but Montecito is a relatively narrow strip bordering the Pacific Ocean. So its minimum elevation above sea level is 0 feet, while its overall elevation is variously reported at 50ft and 180ft. At the same time, Mr. Gore prominently sponsors a campaign and award-winning movie that warns that, due to Global Warming, we can expect to see nearby ocean-front locations, such as San Francisco, largely under water. The elevation of San Francisco is variously reported at 52ft up to high of 925ft.

I've highlighted the same idea before by the way:

Ask yourself, wouldn't you fly a plane into a tower if that was the only way to disable Skynet? The difference between religion and the risk of uFAI makes it even more dangerous. This crowd is actually highly intelligent and their incentive based on more than fairy tales told by goatherders. And if dumb people are already able to commit large-scale atrocities based on such nonsense, what are a bunch of highly-intelligent and devoted geeks who see a tangible danger able and willing to do? More so as in this case the very same people who believe it are the ones who think they must act themselves because their God doesn't even exist yet.

Ah yes, the standard argument against consequentialism: X has expected positive consequences, so consequentialism says we should do it. But clearly, if people do X, the world will be a worse place. That's why practicing consequentialism makes the world a worse place, and people shouldn't be consequentialists.

Personally I'll stick with consequentialism and say a few extra words in favor of maintaing the consistency of your abstract arguments and your real expectations.

Uhm, it's seriously egregious and needlessly harmful to suggest that SIAI supporters should maybe be engaging in terrorism. Seriously. I agree with Yvain. The example is poor and meant to be inflammatory, not to facilitate reasonable debate about what you think utilitarianism means.

Would you please rewrite it with a different example so this doesn't just dissolve into a meaningless debate about x-risk x-rationality where half of your audience is already offended at what they believe to be a bad example and a flawed understanding of utilitarianism?

This confluence of ideas made me realise that we are making life and death decisions every time we spend $1000 dollars. I'm not sure where I will go from here.

Here's a blog post I found recently that discusses that idea further

I'm surprised no one has linked to this yet. It's not a perfect match, but I think that "if killing innocent people seems like the right thing to do, you've probably made a mistake" is close enough to be relevant.

Maybe less so before the post was edited, I guess.

(Insert large amount of regret about not writing "Taking Ideas Seriously" better.)

Anyway, it's worth quoting Richard Chappell's comment on my post about virtue ethics-style consequentialism:

It's worth noting that pretty much every consequentialist since J.S. Mill has stressed the importance of inculcating generally-reliable dispositions / character traits, rather than attempting to explicitly make utility calculations in everyday life. It's certainly a good recommendation, but it seems misleading to characterize this as in any way at odds with the consequentialist tradition.

In general, the primary problem with such behavior is that if lots of people do this sort of thing society falls apart. Thus, there's a bit of a prisoner's dilemma here. So any logic favoring "cooperate" more or less applies here.

Note also that for many people they would probably see this as wrong simply because humans have a tendency to see a major distinction between action and inaction. Action that results in bad things is seen as much worse than inaction that results in bad things. Thus, the death of the guard seems "bad" to most people. This is essentially the same issue that shows up in how people answer the trolley problem.

So, let's change the question: If there's no substantial chance of killing the guard should one do it?

A topical real-life example of this is the DDoS attacks that Anonymous are making against various companies that pursue/sue people for alleged illegal file sharing.

I make no comment on the morality of this, but it seems to be effective in practise, at least some of the time, for example it may lead to the demise of the law firm ACS:law.

See also:

• http://lesswrong.com/lw/778/consequentialism_need_not_be_nearsighted/
• http://lesswrong.com/lw/apm/slowing_moores_law_why_you_might_want_to_and_how/