The autopilot problem: driving without experience

The complacency and deskilling are a feature, not a bug. The less I have to learn to get from place to place, the more attention I have for other things that can't be automated (yet).

Attributing to a GPS faillure a woman driving 900 miles to croatia when she intended to drive 38 miles within Belgium is naive. Most likely she put the wrong address in, possibly with the help of autocomplete, possibly not. But crazy, drug-addled, and or senile people have been winding up hundreds of miles from where they thought they were for a long time before there were any GPS satellites in orbit. Actual GPS errors in my experience take you to a street behind your inou tended destination, or direct you to streets that are closed. And these errors fall off quickly as the expert system becomes, well, more expert. The GPS navigation app errors tend to be small, bringing you near where you need to go but then requiring some intelligence to realize how to fix the error the system has made. Meanwhile, I drove two hours out of my way on vacation in Florida, an error I could not have possibly made to that extent if I had had the GPS navigation systems I now use all the time.

Automated cars WILL be blamed for all sorts of problems including deaths. The unwashed innumerates will tell detailed stories about how they went wrong and be unmoved by the overall statistics of a system which will cause FEWER deaths per mile driven than do humans. Some of those deaths will occur in ways that after-the-fact innumerates, and other elements of the infotainment industry known as democracy, will tell wonderful anecdotes about them. There may even be congressional hearings and court cases. The idea that a few deaths that MIGHT have been avoided under the old regime is literally a small price to pay for an overall lower death rate will be too complex a concept to get legs in the infotainment industry.

But in the long run, the nerds will win, and economically useful automation will be broadly adopted. We don't know how to grow our own food or build our own houses anymore and we've gotten over that. We'll get over this too and the innumerate infotainment industry known as democracy will move on to its next stupidity.

I read your piece and replaced 'autopilot' with 'social structure' and it still works. When you use the autopilot of membership in a group, you get the same errors.

This post reminds me of this essay, which I enjoyed, on the topic of automation and deskilling: http://www.macroresilience.com/2011/12/29/people-make-poor-monitors-for-computers/.

Firstly, if the automation performs flawlessly, the overseers will become complacent, blindly trusting the instruments and failing to perform basic sanity checks. They will have far less procedural understanding of what's actually going on, since they have no opportunity to exercise their knowledge.

There's a related problem in manufacturing whose name I've forgotten, but basically, the less frequent defective parts are, the less likely it is human quality control people will notice defective parts, because their job is more boring and so they're less likely to be paying attention when a defective part does happen. (Conditioned on the part being defective, of course.)

Only the cruise control link is an actual comparison of automation+overseer versus just humans. The rest given are examples of automation+overseer failing but there are of course examples of just humans failing just as badly. Is there any further evidence of this phenomenon? In particular, is there evidence that the total success rate decreases as the success rate of the automation increases?

I wonder if it's possible to bring the success rate back up in QA conditions by requiring the identification of the candidate furthest from ideal within a given period, whether or not that is within tolerances. Of course, in some cases, that would completely negate the purpose of the automatic behavior.

I think that you're noticing that automation does not require that the overseer ever develop the skills required to perform the task manually; drivers don't have to learn how to maintain constant speed for hours on end, pilots don't have to develop the endurance to maintain altitude and heading. There is an element of skill atrophy, and of encouraging distractions, and the distractions are likely to result in worse immediate responses to the failure of automation; the skill (responding to emergencies) would have atrophied anyway.