[link] Cargo Cult Debugging

...except that, reading what he did, it makes perfect sense.

He recognized one potential cause of the issue, it was cheap to test, so he tested it.

The best approach to solving problems isn't to look into the most likely cause of the problem, but the most cost-effective, in terms of probability of being the issue and time involved in testing it.

I have issues with character encodings fairly regularly in my job from systems upstream or downstream giving me incorrect information about what to expect from them/send to them. I have a toolbox of preprogrammed solutions. It's cheaper for me to test every tool in my toolbox (takes less than an hour), see what fixes it, and use that, than it is for me to figure out exactly what the system upstream or downstream is doing wrong (takes many hours). On rare occasion, none of my tools work, and I need to debug the problem properly - then I modularize the solution and add it to the toolbox.

I could spend six hours every time, or one hour nine times out of ten and seven hours the tenth. One is the "proper" way to solve the problem, the other is trying random solutions (cargo cult debugging) and seeing what sticks.

This article sounds good. In practice, I don't think it measures up.

Because if it takes twenty minutes to check your dependencies, and twenty hours to learn how to read a manifest, you need to be sixty times more certain that it's a problem with the manifest to justify running that test first.

Not unlike the way some pigeons trained B. F. Skinner. Skinner gave pigeons food at random times. The pigeons repeated behaviors (turning, corner-touching, etc.) they made when the food was delivered. Skinner interpreted this as the pigeons adopting a superstition that would magically make food appear. But later research suggested Skinner's interpretation of the error was, itself, an error.

I think he's being a bit harsh on himself - if a possibility is easy to check, it can be worthwhile to check it first, even if it's not the most likely possibility.

I'd post the full error message instead of just complaining about its being unhelpful and philosophizing about cargo cults. The point about intellectual laziness is well taken, but the surrounding sermon strikes me as lacking much force.

The first words with which he describes the problem are "doesn't run". This, like its even commoner cousin "doesn't work", is overly generic. Exactly how a failure manifests is often the first vital clue to figuring out the causes.

Googling an error message is a good idea (default assumption: if it happened to me, it might have happened to someone else), but stopping there is a less effective one. Effective debugging involves squeezing all the information out of the failure that you can get: details of the error message, traces in system logs before, during and after the failure, and so on.

The failure might have to do with manifests, or that could be a red herring. Effective debugging avoids spending lots of time on potential red herrings.

The thing he tried, specifically, might not have been the best idea - as I recall Windows emits quite specific error messages when DLL dependencies are not satisfied, so a vague message that "likely has to do with a bad manifest" would not immediately send me in that direction.

Reproducing the failure can be useful, varying initial conditions - try a clean install of Windows 2003, try one that has had a bunch of stuff installed - characterizing it as a systematic or erratic error. Also trying to understand the relevant differences between Windows 2003 and other versions - it's axiomatic that the different behaviour has to originate in some difference, but the difference might be in the environment rather than in the executable being produced.

XKCD: The General Problem - http://xkcd.com/974/

Got this one on my desk. Doing things "the right way" is the Achilles heel of technical types. To combat this particular bit of insanity, I always try to start from "what's the minimum cost path that allows me to move forward?"