Self-fulfilling correlations

The crazier, more-expensive, and more-difficult the method is, the more improvement it should show; craziness should filter out less-committed parents.

I think there'd be a U-shaped curve. Requiring some degree of effort will select for people who are willing to make good choices. At some point, craziness and expense will swamp out the good effects.

I have an example from food rather than schooling-- vegetarians might be healthier than the average person. Vegans might not be, and it's even less likely for fruitarians. Breathairians certainly won't be healthier than average.

This Genomes Unzipped post discusses the possibility that henceforth genetic sequencing will produce self-fulfilling/self-amplifying prophecies. The kid who is told that he or she has a genetic advantage in sprinting versus marathoning will then invest more effort into the advantaged activity at the margin, increasing the predictive power of the relevant genetic variants.

Any new education method will show increases in student test scores if people believe it results in increases in student test scores, because only interested parents will sign up for that method.

For example, Freakonomics tells the story of high school students in Chicago who participated in a lottery for the chance to switch schools. The students who were reassigned to new schools were more likely to graduate; but the students who applied for the lottery but lost did just as well. The explanation given is that the students (or parents) who care about education will attempt to switch to a better school, but the "better" school won't confer an advantage.

Cullen, Jacob, and Levitt. "The Impact of School Choice on Student Outcomes: An Analysis of the Chicago Public Schools". J. Public Econ. 200?.

Cullen, Jacob, and Levitt. "The Effect of School Choice on Student Outcomes: Evidence from Randomized Lotteries". National Bureau of Economic Research working paper, 2003.

Even a lie is a psychic fact. --Carl Jung, (1875-1961)

I find that helpful in reminding myself that beliefs, even false ones, can be causal.

This is one of Natures most cruel jokes, as it makes things that appear to have potential to improve your life better than they are.

Do Montessori schools or home-schooling result in better scores on standardized tests?

http://en.wikipedia.org/wiki/Homeschooling#Test_results

Back in the late 1980s, neural networks were hot; and evaluations usually indicated that they outperformed other methods of classification. In the early 1990s, genetic algorithms were hot; and evaluations usually indicated that they outperformed other methods of classification. Today, support vector machines (SVMs) are hot; and evaluations usually indicate that they outperform other methods of classifications. Neural networks and genetic algorithms no longer outperform older methods. (I write this from memory, so you shouldn't take it as gospel.)

I know you said you were writing from memory, but this paragraph above is very vague. 'Neural networks' in their many variants have been used (and still are used) in a wide variety of domains and applications, but mainly classification/recognition systems such as character recognition, speech recognition, machine vision in some cases, etc. Genetic algorithms (aka simulated annealing, evolutionary search) from what I recall are used in search and optimization applications, and I'm not aware of usage in recognition at all. Of course, using GA approaches to explore the space of NN designs is a powerful combination even today - there are numerous examples, but in the video game space, one of the most advanced and radical animation systems is driven by GA + NN approaches.

Thats one example, but in general GA-like evolutionary search is a technique that is unlikely to ever be 'trumped' in the domains it reigns. Its simple, old, but effective, like quicksort or radix sort in some sense for its domains. Think of it this way. Fully exploring a search space (for a circuit design or NN virtual circuit or whatever) with a GA/evolutionary search is expensive, but If you have the time to spare and enough population diversity, a big general evolutionary search will eventually converge to the optimal solution.

There's many potential refinements on search, and many newer classification approaches, but many (such as SVM) are very similar to past approaches with a few minor but key improvements.

In a broader sense, all of technology always uses evolutionary search all the time, it is our current global meta-algorithm.

Fads can create self-fulfilling correlations. If neural networks are hot, the smartest people tend to work on neural networks. When you compare their results to other results, it can be difficult to look at neural networks vs., say, logistic regression; and factor out the smartest people vs. pretty smart people effect.

I'm skeptical about your conclusion for the publication bias, mainly because I find that even if it does exist, its swamped out by other effects.

Science and engineering involve something like a landgrab in ideaspace. Its really, really difficult in the modern era to find an idea that is actually new. But that is exactly what you need to make your career. Scientists aren't like moths that are attracted to pretty flames, they are more like explorers and colonizers, who attempt to stake out novel claims. I believe this effect is quite strong, and this bias works in the exact opposite direction to that which you propose.

Researchers developing new techniques have a bias to over-represent their own results and under-represent older results.

All that being said, the two biases may both exist at different scales; like a gold rush effect drawing scientists into a domain combined with a strong local land-grab effect separating them into different regions of the space.

Good point! A nitpick:

Believing that Internet Explorer is safer from malicious websites than Google Chrome won't decrease your computer's infection rate.

I don't think that this is a good counterexample. If people who want to avoid malicious websites tend to believe that IE is safer from them and thus choose IE, then IE should get better results (ceteris paribus) since people who care about malicious websites are also more likely to take care to type paypal.com directly into the address bar, etc.

In other words, one's attitude to safety from malicious websites has as much effect on infection rates as one's attitude to safety from car crashes has effect on crash rates. (Or more, I would guess offhand.)

Am I missing something?

The crazier, more-expensive, and more-difficult the method is, the more improvement it should show; craziness should filter out less-committed parents.

Montessori

Your main point may well be valid; I think it probably is. But my daughter attended a Montessori kindergarten (but not a Montessori school) and I have read Maria Montessori's book. Neither seemed at all crazy to me.

The Montessori method is to engage children in activities which are challenging but not discouragingly so. Each activity produces a small increment in a skills. The children seem to become absorbed in the activities and find them very rewarding. In the adult world this would probably be something like "deliberate practice".

This idea of learning skills in small increments - in the sweet spot between "too easy and you learn nothing" and "too hard so you learn nothing and get discouraged" has wide applicability to children and adults. For example after almost a year of conventional swimming lessons and my daughter could not swim, I tried applying this method to swimming.

Swimming of course requires you to do several things at once. If you don't do them all you get a mouth full of water and learn very little.

I bought her a buoyancy vest and fins. She learned to swim with these very quickly. After a while we deflated the vest progressively and she again learned to swim that way, being now responsible for staying afloat. Then we took away the fins and she mastered that quickly. After a few lessons she was a confident swimmer. This was a very dramatic result. Back at the swim school they were surprised she could now swim, but were totally uninterested in how we achieved this.

The Montessori children seem to end up with excellent powers of concentration; that is certainly the case with my daughter. I did hear of a study that found that this was the most prominent effect of the Montessori schools. I would suggest they are worth looking at, but I would check that they are actually following the method.

A minor pedantry; this was a wonderful post:

Sometimes corr(X,Y) means X=>Y...

corr(X,Y) is a number between -1 and 1, and without specifying its value it's hard to see how it could imply anything. Perhaps it would be better stated as something like

Sometimes corr(X,Y) > 0 means that X=>Y...

I really like this post. I'd like to see a sequence of ways statistical analysis can go wrong. This article and the inverted-J one both fit that description well.

Are there any other posts on the subject?

Nice article. It seems to be the opposite of goodhart's law (self-destructing correlations) but if there's some higher model that unifies the two, I can't see it at the moment.

Presumably, you'd look at the accident rate for Volvos compared to the accident rate for similar cars driven by a similar demographic, as reflected, for instance in insurance rates. (My google-fu did not find accident rates posted on the internet, but insurance rates don't come out especially pro-Volvo.)

Tangential, but I think you may be looking at the wrong thing here; this is auto insurance. The thing it measures in addition to how often that sort of car gets into accidents, would be to what extent the car itself is resistant to damage, not how well it protects the passengers. And since other sorts of insurance aren't likely to depend on what sort of car you have, I'm not sure there's any sort of insurance you could look at for quite this info.

You didn't talk about any self-fulfilling negative correlations. A volvo doesn't prevent accidents, just makes the accidents less deadly, so it may actually be the more reckless drivers that take them so they can continue to be reckless (this effect may be much smaller though) or parents that choose a safe car for their reckless teen. Another example is when seatbelts were introduced, the owners of cars with them became more reckless because they thought they were safer, and actually ended up in more accidents (though the death rate of drivers remained about the same because the seatbelts do actually offer protection, not just self fulfilling).

Maybe someone can think of better examples. I can imagine that these are hard to perpetuate though, because the concept that something is safer is often based on scientific evidence with proper selection, or real world evidence with distorted selection, but with a negative correlation, the distorted selection would show the safety device makes people less safe. You would have to have a good advertising team to overcome both the scientific and real world examples of your safety device. Either that or it really does make you slightly safer but the negative correlation effect would have to be strong enough to overcome that. But in either of these cases it isn't self fulfilling, because the concept isn't caused by the results, but either by actual benefit or advertising.

But how would you evaluate the claim that Volvos are safer than other cars?

You would look for explanations why Volvos are safer.

Presumably, you'd look at the accident rate for Volvos compared to the accident rate for similar cars driven by a similar demographic, as reflected, for instance in insurance rates. (My google-fu did not find accident rates posted on the internet, but insurance rates don't come out especially pro-Volvo.)

No, this is not what you would do. The accident rate is consistent with many theories, including the theory that Volvos are not safer.

But suppose the results showed that Volvos had only 3/4 as many accidents as similar cars driven by similar people. Would that prove Volvos are safer?

No. Besides having a reputation for safety, Volvos also have a reputation for being overpriced and ugly. Mostly people who are concerned about safety buy Volvos. Once the reputation exists, even if it's not true, a cycle begins that feeds on itself: Cautious drivers buy Volvos, have fewer accidents, resulting in better statistics, leading more cautious drivers to buy Volvos.

Yes, the accident rate data is also consistent with this theory. So looking at accident rates alone isn't going to tell you anything about the safety of Volvos. And rational drivers would know this. They wouldn't buy a Volvo because it has a reputation for having fewer accidents; they would buy a Volvo because they have an explanation for why it is safer than other similar cars.

If a Montessori school cost the same, and was just as convenient for the parents, as every other school, and all factors other than test score were equal, and Montessori schools were believed to increase test scores, then any parent who cared at all would choose the Montessori school.

No, they wouldn't, not if they really cared. To choose an education method on the basis of test scores is irrational. A parent that really cared would try to understand our best epistemology and act according to that. Schools and parents that employ coercion and that care about test scores are flying in the face of what we know about how knowledge grows. A good parent would know that.

It seems to me that your post devalues the role of explanations.

Very good examples of perceptions driving self-selection.

It might be useful to discuss direct and indirect effects.

Suppose we want to compare fatality rates if everyone drove a Volvo versus if no one did. If the fatality rate was lower in the former scenario than in the latter, that would indicate that Volvo's (causally) decrease fatality rates.

It's possible that it is entirely through an indirect effect. For example, the decrease in the fatality rate might entirely be due to behavior changes (maybe when you get in a Volvo you think 'safety' and drive slower). On the DAG, we would have an arrow from volvo to behavior to fatality, and no arrow from volvo to fatality.

A total causal effect is much easier to estimate. We would need to assume ignorability (conditional independence of assignment given covariates). And even though safer drivers might tend to self-select into the Volvo group, it's never uniform. Safe drivers who select other vehicles would be given a lot of weight in the analysis. We would just have to have good, detailed data on predictors of driver safety.

Estimating direct and indirect effects is much harder. Typically it requires assuming ignorability of the intervention and the mediator(s). It also typically involves indexing counterfactuals with non-manipulable variables.

as an aside: a machine learning graduate student worked with me last year, and in most simulated data settings that we explored, logistic regression outperformed SVM

No. Besides having a reputation for safety, Volvos also have a reputation for being overpriced and ugly. Mostly people who are concerned about safety buy Volvos. Once the reputation exists, even if it's not true, a cycle begins that feeds on itself: Cautious drivers buy Volvos, have fewer accidents, resulting in better statistics, leading more cautious drivers to buy Volvos.

I'm having a hard time integrating what you're saying with my existing knowledge of confounders, which describes a similar phenomenon. Can anyone help me out here by describing self fulfilling phenomenon in terms of confounding?

On somewhat tagential note: due to massive mis representation of science, people put well founded logical conclusions at same level with entirely unfounded ones, and attribute overly high weight to correlation-based 'evidence', seeing the latter as more scientific. That creates a lot of demand for some correlation based 'evidence' where one would instead conduct specific experiments instead (e.g. crash-test the cars with dummies and be satisfied; it changes nothing about my personal car choice whenever people who chose some car drive safer, or whenever people on average drive less safely when they are driving a safer car. The dummy may be a poor model of me, but averaged person may be even worse model).

That could have it's roots in how science is taught at school - formulate hypothesis, test hypothesis with experiment - without the understanding that the reasoning behind hypothesis can very well be considerably stronger as evidence than many of potential 'experiments', or can be so weak as to make it worthless to even bother conducting an experiment. (For example, the reasoning (calculations) that the rocket will end up in a particular point in space after executing well controlled burns, is very solid and if the rocket ends up in the other place the chances overwhelmingly are that the experiment, rather than the reasoning, has failed)

Y; sometimes it means Y=>X; sometimes it means W=>X, W=>Y. And sometimes it's an artifact of people's beliefs about corr(X, Y). With intelligent agents, perceived causation causes correlation.> Sometimes the issue could be the choice of the appropriate Y to correlate with your X. In the Montessori example, rather than looking at test scores or even at rate of graduation from high school, it could be argued that it would be more meaningful to look at career performance. However choosing something as general as career performance requires a great deal of "narrowing down". For example: First choose your career to compare, say law. Narrow it down to one, hopefully very large, law school. Further narrow it down to one demographic - all students graduating from similar performing high schools, who themselves performed similarly in their admission tests. Continue to narrow your study down even farther - similar race, religion, family income, number of siblings, birth order, parents' careers and educations, etc. Then you can look at the pre-high school education - Montessori vs. other private educational methods. Ouch!