[Link] Algorithm aversion

Why, then, is it that people continue to mistrust algorithms, in spite of their superiority, and instead cling to human advice? A recent paper by Dietvorst, Simmons and Massey suggests it is due to a cognitive bias which they call algorithm aversion.

Why is it that opium puts people to sleep? A recent paper by Molière suggests it is due to a property which he calls its dormitive principle.

My intuition was that it was fear of edge cases, even when the person couldn't articulate exactly what an edge case is.

I would loosely model my own aversion to trusting algorithms as follows: Both human and algorithmic forecasters will have blind spots, not all of them overlapping. (I.e. there will be cases "obvious" to each which the other gets wrong.) We've been dealing with human blind spots for the entire history of civilization, and we're accustomed to them. Algorithmic blindspots, on the other hand, are terrifying: When an algorithm makes a decision that harms you, and the decision is -- to any human -- obviously stupid, the resulting situation would best be described as 'Kafkaesque'.

I suppose there's another psychological factor at work here, too: When an algorithm makes an "obviously wrong" decision, we feel helpless. By contrast, when a human does it, there's someone to be angry at. That doesn't make us any less helpless, but it makes us FEEL less so. (This makes me think of http://lesswrong.com/lw/jad/attempted_telekinesis/ .)

Real world example.

Years ago, I worked at a company which made a machine to screen pap smear slides. Granted, that much of the insanity in health care is about regulatory power, law, and money, but even so, people were just weird about algorithmic screening.

The machine was much more accurate than the great mass of labs in the country.

But no matter how accurate automated screening was when compared to manual screening, there was always such a tizzy about any algorithmic faults. The fact that manual screening produces many more such faults was simply glossed over. Human faults were invisible and accepted, machine faults were a catastrophe.

Here's an article in Harvard Business Review about algorithm aversion:

It’s not all egotism either. When the choice was between betting on the algorithm and betting on another person, participants were still more likely to avoid the algorithm if they’d seen how it performed and therefore, inevitably, had seen it err.

My emphasis.

The authors also have a forthcoming paper on this issue:

If showing results doesn’t help avoid algorithm aversion, allowing human input might. In a forthcoming paper, the same researchers found that people are significantly more willing to trust and use algorithms if they’re allowed to tweak the output a little bit. If, say, the algorithm predicted a student would perform in the top 10% of their MBA class, participants would have the chance to revise that prediction up or down by a few points. This made them more likely to bet on the algorithm, and less likely to lose confidence after seeing how it performed.

Of course, in many cases adding human input made the final forecast worse. We pride ourselves on our ability to learn, but the one thing we just can’t seem to grasp is that it’s typically best to just trust that the algorithm knows better.

Presumably another bias, the IKEA effect, which says that people prefer products they've partially created themselves, is at play here.

Possible counterexample:

My father is a professor of electrical engineering. The electronics lab courses involve using simulation software as well as using physical components. In one lab experiment, the students built a circuit that the software didn't simulate correctly (because of simplifications in the models the software used), and one of the questions the students had to answer was why they thought the computer simulation didn't match the measured values. All the students blamed experimental error, and none questioned the computer models...

Probably because humans who don't know much about algorithms basically have no way to observe or verify the procedure. The result of an algorithm has all the force of an appeal to authority, and we're far more comfortable granting authority to humans.

I think people have also had plenty of experience with machines that malfunction and have objections on those grounds. We can tell when a human goes crazy if his arguments turn into gibberish, but it's a bit harder to do with computers. If an algorithm outputs gibberish that's one thing, but there are cases when the algorithm produces a seemingly reasonable number that ends up being completely false.

It's a question of whether to trust a transparent process with a higher risk of error or a black box with a lower, but still non-negligible risk of error.

Like other commenters already pointed out, algorithms are scary because they always fail hard. Humans fail, but can recover. Hofstadter's terms seem useful here. Humans can notice or be made to notice that something isn't right and jump out of the system of the basic diagnostic procedure. All the algorithms we currently have are sphexish and will forever remain in their initial framework even when things go wrong.

If people trust human forecasters over machine forecasters, but the machine forecasts are better, just use the machine secretly, and take all the credit.

My intuition was that participants assumed the human forecasters could use some knowledge the model couldn't, but that is definitely not the case in the xperiments. The model and the forecaster have exactly the same data and that is made clear in the setup. It is really that participants prefer the human despite 'better knowledge' - esp. if the comparison is explicit:

Interestingly, participants in the model-and-human conditions, most of whom saw the model outperform the human in the first stage of the experiment [...] were, across all studies, among those least likely to choose the model.

Page 6, emphasis mine.

I admit that I am surprised and I do not understand the cause of this algorithm aversion.

I wonder if this (distrusting imperfect algorithms more than imperfect people) holds for programmers and mathematicians. Indeed, the popular perception seems to be that such folks overly trust algorithms...

Are algorithms easier to exploit than humans? Consider the case of playing a video game against a deterministic AI opponent; if the AI makes a stupid mistake once, it'll make the same stupid mistake over and over, and an AI with a known exploit is easy to beat regardless of how well it plays in "normal" circumstances.

I think it is the issue with moral responsibility, same way as with self-driving cars. People don't want a decision who may negatively affect a person be based on an algorithm because an algorithm is not a moral agent. They want some expert to stand up and say "I risk my professional prestige and blame and all that and declare this person to be prone to violence and thus needs to have a restraining order issued, and if I am wrong it is my bad". As my dad used to say it is all about who is willing to put their dick in the cigar cutter? To accept responsibility, blame, even punishment for a decision made that affects others?

Part of it is rational: decision makers having skin in the game makes decisions better. See Taleb, Anti-Fragile.

Part of it is simply we being used to or evolved to thinking without responsibility there cannot be good decisions, which is true as long as humans make them. We are not evolved to deal with algorithms.

To the blind spots/edge cases objection; the article about SPRs linked in the post shows that even when human experts are given the results of an algorithm and asked to fix their errors, they still do worse than just the algorithm on it's own.

Because even though the human may recognize an obvious mistake, they will also recognize ten good predictions as "mistakes".