The Stable State is Broken

Your list of examples is lovely and illustrative. There are enough of them to show the common thread through all of them easily, and you write them concisely and readably.

Reminds me of Scott Aaronson's Malthusianisms. Is this the article you couldn't find?

Also, I am not sure your example of science is correct: After all, plenty of very famous journals do publish retractions, and some that do not are (rightly) laughed at (parapsychology journals for example).

I agree, this class of problems is enormous, and has a hand in practically all mismanagement and misdirection of human effort. The problem is so aggravating, though, because humans seem not to expect it to happen. Why do these feel like "problems", when the underlying behavior is exactly what we'd expect given our knowledge of stable strategy?

In the case of human endeavor, I suspect this is a problem because we do not try hard enough to defend against it. We seem surprised and indignant when systems that purport to do good are "gamed." If more people were more aware that this was the consequence of strategic agents, they might watch harder for signs of destructive strategy, and more carefully design the systems that they build and manage against such strategy.

(Hmm. I notice that I'm posing a purportedly-fully-general strategy to a fully-general problem, without evidence or examples. And I'm claiming that better global understanding is a solution, when it is probably just applause lights to my imagined audience. And I still think that what I'm saying is right! Wow. You should probably ignore this comment's content, but I'll still leave it here, as a counterexample.)

I find this article to be riddled with questionable statements, judgements, and explanations. The argument schema is that "X is prominent where Y would be better" - but in most cases, it's not at all clear that Y would be better, or that the explanation for X's prominence is correct. In some cases, the triumph of Y would be in the author's self-interest, in other cases I think it's just that he's not aware of alternative considerations or perspectives.

Let's start with the claim that rule by scientists and economists would be better than rule by politicians and lawyers. I see this as self-interest, or even as the vanity of believing that one's own group - here, rationalists and quantitative thinkers - is generally superior. Politicians have people skills and lawyers know how the system works, both attributes which are actually needed to run a government well. At its worst, a government by number-crunching intellectuals would be an ineffectual technocracy, first trying to reshape the world according to the wacky theories of academics, and then helpless when a mob of pitchfork-wielding villagers show up, wanting to kill the monster.

It's funny that in one paragraph, we hear that science is frequently dysfunctional, and then a few paragraphs later, that scientists should be in charge. So maybe it's only the right sort of scientists who should be in charge. Someone else remarked that parapsychology is hardly a good example of what is prominent in science; parapsychologists are shunned like creationist biologists, as pseudoscientists. So, just as the item about politics can be read as a statement that "people in my cultural group should be running everything", perhaps the item about science can be read as a statement that "stuff like parapsychology shouldn't exist at all!" Certainly, the remarks about journals which achieve prominence because they have low quality control and so get to publish the breakthroughs, seems like pure mythmaking, a "just-so story"; I can't think of a single journal which matches that description - and there ought to be a dozen such journals, if this really is the basic explanation for the existence of bad science in print. In the real world, breakthroughs tend to be achieved by quality people and published in quality journals which everyone reads because they are known to have high standards. Bad journals may provide a space in which bad science gets to see print, but they aren't "widely read and widely cited".

The argument that having five girls for every boy would be better for the species is new to me. Obviously this is sexually appealing for males, so we can count this as another case in which the supposedly all-around-better situation conveniently happens to be better for the author's specific group. Despite the evo-psych opinion that women want quality rather than quantity in a mate and that they are more bisexual than men, the odds are that women would be personally worse off in such a world. However, what is asserted here is that this 1:5 sex ratio is evolutionarily superior to the 1:1 ratio, because the population grows and adapts faster. That is a very dubious assertion. It's often said that the 1:1 regime works through a functional division in which males are a reservoir of genetic variance and female choice of mates provides a selective filter. Maybe in reality the mate selection by both sexes is equally important, but for different traits. In any case, I don't see the knockdown argument in favor of 1:5, once you take sexual selection into account.

I could question some of the other items too.

Another example applies to memes: the ideas which survive are not the ones that are the best for people, but which are the best at replicating and perpetuating themselves. As Dawkins notes in his original meme essay:

To take a particular example, an aspect of doctrine that has been very effective in enforcing religious observance is the threat of hell fire. Many children and even some adults believe that they will suffer ghastly torments after death if they do not obey the priestly rules. This is a peculiarly nasty technique of persuasion, causing great psychological anguish throughout the middle ages and even today. But it is highly effective. It might almost have been planned deliberately by a macchiavellian priesthood trained in deep psychological indoctrination techniques. However, I doubt if the priests were that clever. Much more probably, unconscious memes have ensured their own survival by virtue of those same qualities of pseudo-ruthlessness that successful genes display. The idea of hell fire is, quite simply, self perpetuating, because of its own deep psychological impact. It has become linked with the god meme because the two reinforce each other, and assist each other's survival in the meme pool.

Another member of the religious meme complex is called faith. It means blind trust, in the absence of evidence, even in the teeth of evidence. The story of Doubting Thomas is told, not so that we shall admire Thomas, but so that we can admire the other apostles in comparison. Thomas demanded evidence. Nothing is more lethal for certain kinds of meme than a tendency to look for evidence. The other apostles, whose faith was so strong that they did not need evidence, are held up to us as worthy of imitation. The meme for blind faith secures its own perpetuation by the simple unconscious expedient of discouraging rational inquiry.

Blind faith can justify anything. If a man believes in a different god, or even if he uses a different ritual for worshipping the same god, blind faith can decree that he should die -- on the cross, at the stake, skewered on a Crusader's sword, shot in a Beirut street, or blown up in a bar in Belfast. Memes for blind faith have their own ruthless ways of propagating themselves. This is true of patriotic and political as well as religious blind faith.

http://lesswrong.com/lw/7e1/rationality_quotes_september_2011/4t5p

That's a very important and basic observation.

You missed another example : cancer. Cankerous cells are much better at replicating themselves than normal cells are. Pluricellular organisms have a multitude of systems to keep their component cells in check, yet they still fail at it from time to time. Biology has had billions of years of evolution to fine tune how it enforces cooperation within larger organisms. Can we do better, especially as the components we're considering at our scale may be as complex and clever to us as cells are to an organism? (Meaning we may not have a comparative advantage even though we're subtler than evolution).

So aside from asking what we can do next, I'd like to add : "Can we do something next?" In order to enforce a system within which you won't observe such an effect, you might need to be larger, have more resources than the sum of all you're trying to steer. Otherwise, some part of that system will eventually take over. And even then, chance events may always remain beyond your capacity to control.

One very general way we might address it is by building small, self-replicating versions of systems that have the tools to be more succesful and out compete existing systems for resources, but have built in components necessary to their running which stabilize the out-of-balance part.

If I wanted to make the united states more energy independant, for example, and I had the funds, I'd create an organization that installed for-free solar panels in neighborhoods at the price of a cut of future profits/savings from selling/using the power for a certain period of time. Then use those profits to slowly afford enough solar panels to transform another neighborhood, and so on. And I'd have the contracts for receiving income from clients include a part which specifies that if the organization ever sells solar panels rather than 'giving them away', that the contract holders were free of all payments and would simply own their equipment, I guess.

I suppose this'd cause an immune reaction of sorts to this 'viral' invasion of alternative energy by major power companies. But like all viruses, the company could probably find /some/ way to adapt. I'm assuming there are big holes in this model, and a lot of logistics problems. Like, growth might be intolerably slow for this to really take off. But its meant as more an example of the kind of thing - self reproducing, tooled to outcompete, with stability built in in some way.

I guess, addressing the internet addiction problems of people who're interested in self-help, having a tangible self-help club with group study sessions and regular meetups would be somewhat helpful- but LessWrong already has something a little like that with its meet ups, and I think internet distractions are still a problem. You almost want a self-help software that does things like block website access or redirect your attention to websites you /should/ be on at that moment, provides reminders or motivational quotes, insists on break times, and so on, to become popular or even required for some aspect of participation in some self-help community. Maybe it'd also include reporting tools for your self-help projects.

We don't know how to measure many of the things humans value, so we use proxy measures. These proxies are leaky allowing exploitation.

Seems to me all of these questions are examples of market failures.

It seems to me that in at least some of these examples you are confusing the map with the territory. Take genetics:

Genes don't proliferate by being good for the species; they win by being good for themselves.

Failing to be "good for the species" is not a fact about evolution, or genes. Thinking that evolution was supposed to be "good for the species" was just a heuristic humans used when trying to understand evolution. The "selfish gene" does not say anything meaningful about the phenomenon of evolution, it just shows that we have refined our understanding of evolution.

Now take politics:

Why do governments inevitably end up run by career lawyers and politicians instead of scientists and economists?

What does the phenomenon of government actually look like, in reality? Well, it looks like a system of human hierarchical organization in which career lawyers and politicians have a natural propensity to be on top. Thinking that the phenomenon of government has anything to do with understanding nuanced social issues is confusing the map with the territory.

The idea is this: The subset of participants which rises to prominence in any area does so because its members have traits helpful to becoming prominent, not necessarily because they have traits which are desirable.

That sounds like a special case of Universal Darwinism. Some videos on the topic from me:

• Universal Selection
• NanoDarwinism
• Degenerative Darwinism
• Natural production and natural elimination

Short summary - evolutionary processes are beyond good and evil. Maybe you get a benevolent invisible hand, maybe a boot stomping a human face forever.

Why do governments inevitably end up run by career lawyers and politicians instead of scientists and economists2?

To a second approximation the United States government is run by scientists and economists, and to a first approximation the Chinese government is also run by scientists and economists. No?

You describe a very general principle, which might be described as survival of the fittest. Darwin has explained how it applies in the context of biology. "Survival of the fittest" does not always mean survival of the biggest or strongest or fastest. Tigers may be endangered, but rats and cockroaches are not. You extend the principle from genes to what Dawkins has named as memes. Also, you describe how similar principles apply in an economic free market. This is not an entirely original observation, but reminding people of these principles is good.

Having described what, by your account, is a fundamental and universal principle, you ask, what can we do about it? If it is as fundamental as you say, we acknowledge it, and understand it, and adjust ourselves to it. What do we do about the second law of thermodynamics?

You know what has been done in the past context of economics. The Soviet Union attempted to control the entire economy from the top down. Even so, the Soviet Union could not escape competition from the outside world. Right now, North Korea is trying very hard to control the economy from the top down and shelter itself from competition from the outside world.

What do we do about disease? Lots of things, of course, including nutrition and hygiene. Also, antibiotics. But we find that some bacteria become resistant to antibiotics.

Progress is slower than we would hope. The fundamental laws of nature have not yet been repealed. Right now, that includes human nature. With that said, it would be easier for neurobiologists to change human nature than for physicists to change the second law of thermodynamics.

Why does a LessWrongian resorts to advocating eating babies instead of thinking of some ways to actually make life more rational? Because it's a cheap and easy way to signal intellectual contrarianism and spark discussion.

(/hurr durr)