Diseased disciplines: the strange case of the inverted chart

Interesting. Is he (or anyone else) looking at getting something published that either picks apart the matter or encourages others to?

In trying to think of an easy experiment that might help distinguish, all that's coming to mind is adding a delay to one group and not to another. It seems unlikely this could be done ethically with humans, but animal studies may help shed some light.

An obvious alternate explanation is that fast-growing malignant cancers are likely to kill you even in the unlikely case that you are able detect them before they are large, whereas slow-growing benign cancers are likely to sit there until you get around to detecting them but are not particularly dangerous in any case.

As I have understood cancer development; benign tumors are not really cancer. But the rise of malignity is an evolutionary process, initial mutation increase division rate / inhibit apoptosis, additional mutation occur down the line + selection for maligmant cells. So one can still identify an early stage of cancer, not necessarily early in time but early in the evolutionary process.

claimed that almost all the studies on early detection of cancer confuse degree of disease at time of detection with "early detection". That is, a typical study assumes that a small cancer must have been caught early, and thus count it as a win for early detection.

But then is real early detection really what we are interested in? If study X shows that method X is able to detect smaller tumors than presciently used method Y, wouldn't we consider it a superior method since it enables us to discover cancer in an earlier stage of development when it has not metastasized?

Did you find more recent replications as well? Or was the scientific community in question content that the old, not-so-good studies had found something real?

Nicely put, overall. Your points on the demands of "management" echo this other Dijkstra screed.

However I have to disagree with your claims of "high grade reasoning" in this particular instance. You say "some of the work affects the later work in very clear ways": what ways?

The ways most people in the profession know are less than clear, otherwise we wouldn't see all the arguing back and forth about which language, which OS, which framework best serves the interests of "later work"; or about which technique is best suited to prevent defects, or any of a number of debates that have been ongoing for over four decades.

I've been doing this work for over twenty years now, and I still find the determinants of software quality elusive on the scale of whole programs. Sure, for small or "toy" examples you can always show the advantages of this or that technique: unit testing, UML models, static typing, immutability, lazy evaluation, propagation, whatever. (I've perhaps forgotten more of these tricks than some "hotshot" young programmers have ever learned.) But at larger scales of programming efforts these advantages start being swamped by the effects of other forces, that mostly have to do with the social nature of large scale software development.

I'm a regular reader of Jorge and Greg's blog, and even had a very modest contribution there. It's a wonderful effort.

"Making Software" is well worth reading overall, and I applaud the intention, but it's not the Bible. When you read it with a critical mind, you notice that parts of it are horrible, for instance the chapter on "10x software developers".

Reading that chapter was in fact largely responsible for my starting (about a year ago) to really dig into some of the most-cited studies in our field and gradually realizing that it's permeated with poorly supported folklore.

In 2009, Greg Wilson wrote that nearly all of the 10x "studies" would most likely be rejected if submitted to an academic publication. The 10x notion is another example of propagation of extremely unconvincing claims, that nevertheless have had a large influence in shaping the discipline's underlying assumptions.

But Greg had no problem including the 10x chapter which rests mostly on these studies, when he became the editor of "Making Software". As you can see from Greg's frosty tone in that link, we don't see eye to eye on this issue. I'm partially to blame for that, insofar as one of my early articles on the topic carelessly implied that the author of the chapter in question had "cheated" by using these citations. (I do think the citations are bogus, but I now believe that they were used in good faith - which worries me even more than cheating would.)

Another interesting example is the "Making Software" chapter on the NASA Software Engineering Laboratory (SEL), which trumpets the lab as "a vibrant testbed for empirical research". Vibrant? In fact by the author's own admission the SEL was all but shut down by NASA eight years earlier, but this isn't mentioned in the chapter at all.

It's well known on Less Wrong that I'm not a fan of "status" and "signaling" explanations for human behaviour. But this is one case where it's tempting... The book is one instance of a recent trend in the discipline, where people want to be seen to call for better empirical support for claimed findings, and at least pay overt homage to "evidence based software engineering". The problem is that actual behaviours belie this intention, as in the case of not providing information about the NASA SEL that no reader would fail to see as significant - the administrative failure of the SEL is definitely evidence of some kind, and Basili thought it significant enough to write an article about its "rise and fall".

Other examples of propagation of poorly supported claims, that I discuss in my ebook, are the "Cone of Uncertainty" and the "Software Crisis". I regularly stumble across more - it's sometimes a strange feeling to discover that so much that I thought was solid history or solid fact is in fact so much thin air. I sincerely hope that I'll eventually find some part of the discipline's foundations that doesn't feel like quicksand; a place to stand.

What would you suggest if asked for a major, well-supported result in software engineering?

Is it really valid to conclude that software engineering is diseased based on one propagating mistake?

How about based on the fact that the discipline relies on propagating result rather than reproducing them.

See the ebook referenced at the end of the post, for starters. Will say more later.

These days a lot of projects collect a lot of metrics.

Mostly the ones that are easy to collect: a classic case of "looking under the lamppost where there is light rather than where you actually lost your keys".

it could be the same type of "makes sense"

Now we're starting to think. Could we (I don't have a prefabricated answer to this one) think of a cheap and easy to run experiment that would help us see more clearly what's going on?

We know that late detection is sometimes much more expensive, simply because depending on the domain, some bugs can do harm (letting bad data into the database, making your customers' credit card numbers accessible to the Russian Mafia, delivering a satellite to the bottom of the Atlantic instead of into orbit) much more expensive than the cost of fixing the code itself. So it's clear that on average, cost does increase with time of detection. But are those high-profile disasters part of a smooth graph, or is it a step function where the cost of fixing the code typically doesn't increase very much, but once bugs slip past final QA all the way into production, there is suddenly the opportunity for expensive harm to be done?

In my experience, the truth is closer to the latter than the former, so that instead of constantly pushing for everything to be done as early as possible, we would be better off focusing our efforts on e.g. better automatic verification to make sure potentially costly bugs are caught no later than final QA.

But obviously there is no easy way to measure this, particularly since the profile varies greatly across domains.

The real problem with these graphs is not that they were cited wrong. After all, it does look like both are taken from different data sets, however they were collected, and support the same conclusion.

The true problem is that it is hard to say what do they measure at all.

If this true problem didn't exist, and these graphs measured something that can be measured, I'd bet that these graphs not being refuted would actually mean that they are both showing true sign of correlation. The reason is quite simple: every possible metric gets collected for a stupid presentation from time to time. If the correlation was falsifiable and wrong, we would likely see falsifications on TheDailyWTF forum as an anecdots.

Can you suggest examples, in mechanical engineering, of "tribal wisdom" claims that persist as a result of poor academic practice? (Or perhaps I am misunderstanding your analogy.)

Yes to all of that, especially the research-practice gap.

For instance around things like "test-driven development" the flow seems to be going in the "wrong" direction, by which I mean not the expected one. The academics seem to be playing catch-up to "prove" whether or not it works, which is largely irrelevant to most people who've chosen to either use TDD or to not use it.

One way to get both qualitative and quantitative evidence of the phenomenon is to look at the proceedings of the ICSE conference, taken as a historical whole (rather than one particular year, article or topic). There was a keynote speech by Ghezzi which examined some numbers, for instance it is tending to become more and more an academic conference, from beginnings more balanced between industry and academia.

Interestingly, the "peaks" of interest from ICSE in things that were more industry-relevant (design patterns and Agile) seem to correspond closely to peaks of industry involvement in the committee, which suggest that the academic-heavy trend is also a trend toward less relevance (at least on the part of this particular conference).

It's also instructive to look at the "most influential" papers from ICSE, from the point of view of "how have these papers changed the way software engineers actually work on an everyday basis". There is one that seems good from that point of view, the one on Statemate, but use of Statemate (a commercial tool) is AFAICT restricted to a subset of industries (such as auto or aerospace). And of course the Royce paper on waterfall, which isn't really from 1987, it was reprinted that year at Boehm's urging.

On the other hand, something like refactoring? That's an industry thing, and largely ignored by ICSE until 1999, seven years after being first formally described in Bill Opdyke's PhD thesis. Or design patterns - presented at OOPSLA in 1987, book published in 1994, discovered by ICSE in 1996.

Your core claim is very nearly conventional wisdom in some quarters.

Would like to hear more about that, in private if you are being intentionally vague. :)

My professional affiliation has the word "Agile" in it, a community which is known both for its disregard for the historical prescriptions arising from software engineering the formal discipline, and (more controversially) for its disdain of academic evidence.

After spending a fair bit of time on LW, though, I've become more sensitive to the ways that attachment to the labels "Agile" or "software engineering" also served as excellent ways to immunize oneself against inconvenient bits of knowledge. That's where I'm trying to make some headway; I provisionally think of software engineering as a diseased discipline and of Agile as something that may have the potential to grow into a healthy one, but which is still miles from being there.

Yup. Dijkstra was one of the early champions of Software Engineering; he is said to have almost single-handedly coined the phrase "Software Crisis" which was one of the rhetorical underpinnings of the discipline. His vision of it was as a branch of applied mathematics.

However, realizing the Software Engineering vision in practice turned out to require some compromises, for instance in order to get money from funding authorities to establish a Software Engineering Institute.

The cumulative effect of these compromises was, as Dijkstra later described, to turn away almost completely from the "applied mathematics" vision of Software Engineering, and to remake it into a sort of Taylorist conception of software development, with notions of "factories" and "productivity" and "reliability" taking center stage.

"Reliability" in particular must have been a heartache to Dijkstra: if software was mathematical, it could not break or wear out or fall apart, and the whole notion of "defects" imported from manufacturing had absolutely no place in thinking about how to write correct software. (See this paper by Priestley for interesting discussion of this dichotomy.)

Dijkstra lost, and his ideas today have little real influence on the discipline, even though he is given lip service as one of its heroes.

For my part, I don't necessarily think Dijkstra was "right". His ideas deserve careful consideration, but it's far from clear to me that software development is best seen entirely or even primarily as a branch of applied mathematics. But maybe that's only because my own mathematical ability is deficient. :)

Thanks! This work owes a substantial debt to LW, and this post (and possible subsequent ones) are my small way of trying to repay that debt.

So far, though, I've mostly been blogging about this on G+, because I'm not quite sure how to address the particular audience that LW consists of - but then I also don't know exactly how to present LW-type ideas to the community I'm most strongly identified with. I'd appreciate any advice on bridging that gap.

It doesn't really read as an inference made from the data in the post itself; it's a conclusion of the author's book-in-progress, which presumably has the warranting that you're looking for.

The conclusion doesn't directly follow from this one anecdote, but from the observation that the anecdote is typical - something like it is repeated over and over in the literature, with respect to many different claims.

You don't call a person diseased because they fail to respond to a cure: you call them diseased because they show certain symptoms.

This disease is widespread in the community, and has even been shown to cross the layman-scientist barrier.

Final 'variable' ought to read 'constant'.

Ick. Thanks!

needs to be followed up

I may mull over your points for a day or so before actually making any changes, but those are excellent points, and just the kind of feedback I was hoping for telling this story here. Much appreciated!

Well, you somewhat shift the impact. The result can hit stronger in the less sensitive point.

As I see it, the point is not "We are doing something wrong and innocent programs are dying!", more like "When we write programs, we'd better first acknowledge that we have no idea what we are doing and then try to get this idea".

Your statement can be read as if people know what they want to acomplish and then fail to accomplish it. This view is too optimistic, I think it is a good idea to remove such a misreading.

Fixed, thanks.

seems that both charts support the same conclusion: the longer problem goes undetected, the more problems it brings

Yes, there's a sort of vague consistency between the charts. That's precisely where the problem is: people already believe that "the longer a bug is undetected, the harder it is to fix" and therefore they do not look closely at the studies anymore.

In this situation, the studies themselves become suspect: this much carelessness should lead you (or at least, it leads me) to doubt the original data; and in fact the original data appears to be suspect.

As Software Engineering is too far from being a science anyway

Yes, that's where I end up. :)

"Early bugs are cheap, late bugs are expensive" suggests that you start with quick and dirty coding and gradually add quality checks (automatic/manual testing, code reviews, up to formal proofs). "Long-undetected bugs are expensive" suggests that it's best to be disciplined all the time.

Because you couldn't. In the ancestral environment, there weren't any scientific journals where you could look up the original research. The only sources of knowledge were what you personally saw and what somebody told you. In the latter case, the informant could be bullshitting, but saying so might make enemies, so the optimal strategy would be to profess belief in what people told you unless they were already declared enemies, but base your actions primarily on your own experience; which is roughly what people actually do.

In the ancestral environment you likely live more-or-less the same way your parents/elders did, so any advise they gave you was likely to have been verified for generations and hence good.

A believer in ev-psych would say something like "humans evolved language to manipulate each other, not to find truth".

Anyone want to come up with a theory about why not bothering to get things right was optimal in the ancestral environment?

Isn't sloppy science just a special case of the effect Eliezer described here?

Good catch, thanks. Edited title and text to fix.

Done! They are in the supplementary material at the bottom.

ISTM that you're making a great argument that the defects claim is in the same category as the "10% of the brain" category. Let me explain.

To a layman, not well versed in neuroanatomy, the 10% thing has surface plausibility because of association between brain size and intelligence (smaller brained animals are dumber, in general), and because of the observed fact that some humans are massively smarter than others (e.g. Einstein, the paradigmatic case). Therefore, someone with the same size brain who's only "normal" in IQ compared to Einstein must not be using all of that grey matter.

Of course, as soon as you learn more of what we actually know about how the brain works, for instance the results on modularity, the way simulated neural networks perform their functions, and so on - then the claim loses its plausibility, as you start asking which 90% we're supposed not to be using, and so on.

Similarly, someone with a poor understanding of "defects" assumes that they are essentially physical in nature: they are like a crack in cement, and software seems like layer upon layer of cement, so that if you need to reach back to repair a crack after it's been laid over, that's obviously harder to fix.

But software defects are nothing like defects in physical materials. The layers of which software is built are all equally accessible, and software doesn't crack or wear out. The problem is a lot more like writing a novel in which a heroine is dark-haired, complete with lots of subtle allusions or maybe puns referencing that hair color, and then deciding that she is blonde after all.

As you observe, the cost of fixing a defect is not a single category, but in fact decomposes in many costs which have fuzzy boundaries:

• the cost of observing the erroneous behaviour in the first place (i.e. testing, whether a tester does it or a user)
• the cost of locating the mistake in the code
• the cost of devising an appropriate modification
• the cost of changing the rest of the software to reflect the modification
• the economic consequences of having released the defect to the field
• the economic consequences of needing a new release
• all other costs (I'm sure I'm missing some)

These costs are going to vary greatly according to the particular context. The cost of testing depends on the type of testing, and each type of testing catches different types of bugs. The cost of releasing new versions is very high for embedded software, very low for Web sites. The cost of poor quality is generally low in things like games, because nobody's going to ask for their money back if Lara Croft's guns pass through a wall or two; but it can be very high in automated trading software (I've personally touched software that had cost its owners millions in bug-caused bad trades). Some huge security defects go undetected for a long time, causing zero damage until they are found (look up the 2008 Debian bug).

The one thing that we know (or strongly suspect) from experience is always monotonically increasing as we add more code is "the cost of changing the rest of the software to reflect the modification". This increase applies whatever the change being made, which is why the "cost of change curve" is plausible. (The funny part of the story is that there never was a "cost of change curve", it's all a misunderstanding; the ebook tells the whole story.)

Of course, someone who is a) sophisticated enough to understand the decomposition and b) educated enough to have read about the claim is likely to be a programmer, which means that by the availability heuristic they're likely to think that the cost they know best is what dominates the entire economic impact of defects.

In fact, this is very unlikely to be the case in general.

And in fact, the one case where I have seen a somewhat credible study with detailed data (the Hughes Aircraft study), the data went counter to the standard exponential curve: it was expensive to fix a defect during the coding phase, but the (average per defect) cost then went down.

See this paper (PDF), page 54, for one study that did this type of analysis, the only such that I'm currently aware of.

If you're interested, you might want to graph the numerical results found there: you'll find that they totally fail to match up with the standard exponential curve.

And, again, it's worth thinking about this for a moment to think about the processes that generate the data: you generally don't know that a bug has been introduced at the moment it's introduced (otherwise you'd fix it straightaway), so there is a lot of opportunity for measurement bias there.

Similarly you don't always know how much a bug has cost, because there are many activities that make up the economic cost of defects: lost customers, support calls, figuring out the bug, figuring out the fix, changing the code, documenting things, training developers to avoid the bug in future... Which of these you coun't and don't count is rarely reported in the literature.

It's not even clear that you can always tell unambiguously what counts as a "bug". The language in the industry is woefully imprecise.

Some authors have bravely tried to address what I identify as the key issue, which was briefly discussed: acknowledging that software is thought-stuff and that knowledge about the human mind is hugely relevant.

Then there's the entire topic of "Agile" which is (as of today) even more lacking than software engineering, when considered as a discipline, but which I believe has the potential to get out of the rut where software engineering is stuck, because it's less committed to its current institutions and ideologies, and more committed to getting results.

I think it's difficult to make a correct C compiler because C is more or less assembler in fancy clothes, and as such it's a language burdened with a lot of contigent history of human decisions about processor architectures. It's easier to make a Lisp interpreter, except that because of this contingent history you can't have Lisp all the way down, and at some point it becomes hard to make a verifiably correct whatever-it-is that connects the neatly mathematical and elegant formulation to the next layer of implementation down.

This is speculation, but making a "fresh start" from the level of logic gates on up might be the only way to no longer have this problem.

So if you came across this story in the field of infectious diseases, you would accept the later reporting as unproblematic, and not update at all in the direction of doubting the standard public health policy?

ETA: welcome to Less Wrong, by the way! Don't let the downvotes (neither of them mine) put you off.

They are very distinct. They intersect at one point - long-standing bugs. One addresses bugs caught earlier and earlier in the development process, and the other bugs introduced later and later in the development process.

Even as they are both live for less time, it shouldn't be too hard to see how they could wreak different amounts of havoc. Or not! We don't know! There's no data on it!