[LINK] Signalling and irrationality in Software Development

In general the article was interesting. However the failures described seem to be related to lack of perspicacity more than to irrationality. If upper management are fooled by the middle managers’ signalling, they are not perspicacious enough and probably aren’t fit to be doing their jobs – training in rationality is unlikely to change that.

In the vast majority of failed projects I’ve been called to looked at, the managers have not read one book on software engineering. They haven’t taken one class, read one article, or been to one workshop. At best, they’ve managed other failing software projects.

This is not irrationality per se, but stupidity and incompetence.

Yes, businesses are under pressure to gravitate toward bad engineering practices, but shouldn’t they be under equal market pressure to compete against companies that are using actually good software engineering practices? Why sure, in the long run. But as Keynes succinctly put it, “In the long run, we’ll all be dead.” Eventually is a long time.

An alternative perspective is that extensive government intervention substantially reduces the market pressure that large companies experience. Here is Moldbug on the subject of Dilbertization (the government intervention he implies is its propping up of the maturity transforming banking system):

The late Communist world was a world in which it was often your job to do strange, useless things badly, all day, for no good reason at all.

There is another world in which it is often your job to do strange, useless things badly, all day, for no good reason at all. This is the world of Dilbert. Many of us have experienced it.

My theory is that Dilbert and Brezhnev are the same thing. I call it Dilbert-Brezhnev syndrome, or DBS. While we are certainly not the Soviet Union, my theory is that America has contracted a rather serious case of DBS.

The Soviet Union was a world in which business bore no relation to profit. People did strange, useless things badly because, lacking the discipline of profit that enforces efficiency, they succumbed to ulterior motives. Their unprofitable enterprises, purportedly businesses, were in fact patronage structures.

America is a much more interesting case, because (aside from its endlessly burgeoning political system, including its grant-funded "nongovernmental" periphery), the industries in which we see Dilbert syndrome are private, profitable. Nothing in America today is Brezhnev bad, but it is getting there. [...]

The answer, I think, is our friend zombie finance. We do not have Gosplan, but we have Wall Street. America is Dilbertized to the extent that Wall Street is zombified.

Let's take the loans that created the housing bubble. These were zombie loans to a T. So, for example, Steve Sailer asks: where was capitalism? Why wasn't anyone betting against these loans, and driving them down to their true value?

The answer is that the free market bears very little relationship to the process that distributed these loans. [...]

My feeling is that American zombie finance is largely responsible for the appearance of DBS in the New World. Why is the country covered with hideous developments, strip-malls and chain stores? Because it has, or had, a financial system designed to finance these things. Many of us would prefer a Ritual to a Starbucks and a Joe's Diner to a Burger King, but the chains have an unbeatable advantage: it is much easier for them to get a loan. [...]

Starbucks is subject to the discipline of profit - but it is not as subject as it should be, because its sheer size gives it access to zombie money. Thus the generic can defeat the specific, blandness outcompetes character, and we drink charred cat hair rather than coffee.

In the vast majority of failed projects I’ve been called to looked at, the managers have not read one book on software engineering. They haven’t taken one class, read one article, or been to one workshop. At best, they’ve managed other failing software projects.

One might be the most dangerous number of books for a manager to have read.

My experience is that the majority of such signalling is around the time that will be needed. You look at a new project and think to yourself it's probably about 6 months work. Your manager tells you that it "has to" go live in 2 months. And then somehow you end up saying "Okay, we'll try" instead of "That seems very unlikely". Even if the previous overdue project was quite similar.

Technical people are not comfortable asserting a realistic schedule to management and management is not comfortable asserting it to shareholders.

A very relevant Dilbert strip.

Agh. Spell-checking is particularly important in the concluding paragraph.

My managers, therefore quite rationally did things that maximized their apparent value in the eyes of their bosses, even if it meant that the project (and, as a result) the entire organization was hurt.

The problems described are applicable to corporations and collective action in general.

The book Moral Mazes chronicles a sociological study in the 1980s, where a sociologist spent years living with the savages of corporate bureaucracy, learning their customs, behaviors, and morals. The irrelevancy of truth, the shifting of blame, the feudal relationship to those higher up, the obligation to keep knowledge from your bosses so that they could avoid responsibility, the situational ethics totally contradicting everything they otherwise profess in life - all of it there, and fully documented in the study.

Orwell and Hayek had it right. Organizations will get predictable results depending on how knowledge, power, and responsibility are aligned despite the clear global suboptimality (read abomination) of the solution to the individuals involved. Align those things wrong, and you get Big Brother and a boot stomping a human face, forever. Align them differently, and you get the modern corporation, or government bureaucracy, or or open source and maker movements.