Schools Proliferating Without Evidence

"That's what happens to a field when it unbinds itself from the experimental evidence" - so the million dollar question for Less Wrong is: what experimental evidence can this community bind itself to, to avoid the same outcome?

NLP is an interesting parallel example. Its founders believed in testing, but the students, not so much. For example, Bandler, in "Using Your Brain For A Change", wrote:

A mathematician doesn't just get an answer and say, "OK, I'm done." He tests his answers carefully, because if he doesn't, other mathematicians will! That kind of rigor has always been missing from therapy and education. People try something and then do a two-year follow-up study to find out if it worked or not. If you test rigorously, you can find out what a technique works for and what it doesn't work for, and you can find out right away. And where you find out that it doesn't work, you need to try some other technology.

It's a transcript from a lecture demonstration where he's just shown how to extinguish someone's urge to smoke, and is testing the result. The volunteer had just claimed she no longer wished to smoke, but Bandler insists that she actually take a cigarette from him, hold it in her hands, and play around with it.

He says:

When you do change work, don't back away from testing it; push it. Events in the world are going to push it, so you may as well do it so you can find out right away. That way you can do something about it. Observing your client's nonverbal responses will give you much more information than the verbal answers to your questions.

He then points out the changed facial expression on the volunteer -- it appears that smelling the cigarette has restored her desire for one. He gives her some modified instructions, to repeat the technique being taught. Afterwards, they verify that the smell no longer acts as a compulsion trigger.

Now, the crazy thing is, Bandler's been teaching this stuff for 20 years, but hardly anybody "gets it"... about testing, or damn near anything else.

Few NLP practitioners do much testing; few NLP books even mention it. Even Bandler's own books don't say that much about it. The formal NLP trainings emphasize "outcome frame" (defining in advance what result you're trying to get), but not so much the process of testing that you've achieved that outcome.

I suspect that this is simply because Bandler is a lousy teacher in some ways. In my personal experience, the most important parts of nearly every Bandler video, audio, or book are in what seem like almost offhand remarks... that happen to reveal volumes if you already happen to be close to figuring out the same thing for yourself. Perhaps this is why he's so insistent that an NLP certification is worthless if it doesn't come from him.

You are describing old clinical psychology. Its gotten so much better. Rorschach tests are now only a very marginal measure within psychoanalytic psychology. Psychoanalytic/pscyhodynamic psychologists are themselves outcasts from mainstream clinical psychology, which is increasingly centered around evidence-based practice. For example, behaviorists are using systematic desensitization in novel and effective ways (for treating things like panic disorder), and cognitive-behavioral therapy is quite effective in treating depression: significantly more so than antidepressants.

The important thing to remember is that patients often get the treatment they want. If you're a self-absorbed neurotic, and you want to spend an hour a week for years talking about yourself, you can find someone who will take your money. If you want effective treatment, you can find that too. Most patients don't want to get better, they want to feel like they are doing something, and especially they want to talk about themselves.

This shouldn't be surprising. Medicine has a longer history than empirical science. For thousands of years it flourished without a second thought for outcome. Clearly whatever medicine is, socially speaking, it isn't reliant on the effectiveness of its methods for its survival. The same is true of education. Schools and universities existed long before there was anything to teach. Whatever social role they may play, imparting skill is a recent development, and clearly not the most central concern.

In my limited experience, the "hard problems" in philosophy are the problems which are either poorly defined and so people keep arguing about definitions without admitting it, or poorly analyzed, so people keep mixing decision theory with cognitive science, for example. While the traditional philosophy is good at asking (meta-)questions and noticing broad similarities, it is nearly useless at solving them. When a philosopher tries to honestly analyze a deep question, it usually stops being philosophy and becomes logic, linguistics, decision theory, computer science, physics or something else that qualifies as science. Hence Pearl and Kahneman and Russell, some Wittgenstein, Popper...

I also have the same kind of unease about economics, especially macroeconomics. It's damn hard to run an experiment to test a hypothesis, and you have also have people running around claiming to be Austrian economists, Keynesian economists, Chicago school economists, etc.

For example: does raising the minimum wage by a few dollars an hour really increase the percentage of people who don't have a job (the sum of the unemployed and those not in the labor force)? Can you prove it?

Today I learned that this psychotherapy criticism apparently has a name: http://en.wikipedia.org/wiki/Dodo_bird_verdict

Cognitive behavioral therapy, at least, has repeatedly been shown to be effective. I'll cite a ref from Wikipedia: Cooper, Mick (2008). Essential Research Findings in Counselling and Psychotherapy: The Facts are Friendly. SAGE Publications. ISBN 9781847870421. But I haven't read that reference.

The Rorschach test is still in use, and still being taught to students. There are studies claiming to find statistical significance in its results. They don't propose a mechanism; but they do have statistically-significant repeatable correlations between patient responses and their clinical problems. If you want to claim otherwise, get some of those papers and refute their methodology. Again, I haven't been interested enough to read the primary sources myself; but I've been told about them by someone I respect. The key questions are whether they erred by not doing something like a Bonferroni adjustment for the large number of possible hypotheses; and whether the statistically-significant correlations are clinically significant. You would have to read the papers to find out.

Another example of schools proliferating without evidence: philosophy. Consider all the different schools of ethics which have sprung up: there's utilitarian ethics, deontological ethics, and virtue ethics, with vast numbers of sub categorizations under each school.

Philosophers are more susceptible to this failure mode because on many important philosophical questions, a standard if not unanimous approach is argue that the question cannot be answered by evidence. Modal logicians trying to do metaphysics, for example.

I have a master degree in psychology. 

Other than cognitive-behavioural therapies many form of brief psychotherapies have been developed in recent years to tackle specific problems, with scientific literature showing their efficacy.

At the moment measuring the results, understanding and testing what works and why it worked, is an element of mental-health intervention that's receiving more and more attention and it's being required in many "schools" or approaches.

From what I've seen, most of the resistance to this is coming from the "long" forms of psychotherapy, arguing that it's too difficult. 

But yeah, as a field, psychotherapy is still struggling with the task of getting rid of a bunch of stuff that's about as evidence based as voodoo, and the standards required to operate in mental-health are way too lax. 

Many books written to train psychotherapist will tell you that most of the healing power comes from the relationship between the psychotherapist and the patient, and that you have to rely a lot on your personal experience and intuition, with no mentions at all of attempts to do better and to improve the methodologies.

Talking to a professional is still your best shot if you need to take care of your mental health, but you might want to find out which school or approach use the professional you are considering, and whether they have an approach of measuring results, operating on scientific evidence and so on.

Edit:

This was probably, to no small extent, responsible for the existence and continuation of psychotherapy in the first place—the promise of making yourself a Master, like Freud who'd done it first (also without the slightest scrap of experimental evidence).

I agree wholeheartedly, still Freud has the credit that the field was such a disaster when he started, that talking to people and trying to go by ear was still a huge improvement. He wanted to help people and didn't understood the scientific approach well enough to use it in such a confusing and unexplored field as mental health, or just wanted to help people now and not 50 years later.

His biggest success seems to be the radical intuition that talking to people with mental health issues yield better results than torturing them and locking them up, but it was still an improvement and I think scientific psychology would have been born a lot later without the impact his ideas had on popular culture, so I'd avoid picking on him too much.  

Lest we be too, too hard on these folks, I should add that, as Eli mentions, there is some value in talking to someone about your problems. I have seen a therapist on occasion who mainly just listens and then tries to advise based on human experience. No lying on the couch, no ink blots, no hypnosis. Just a neutral party to talk to who will keep what I say in confidence. Sort of like Catholic confession, but without all the Hail Marys.

Also, just in case anyone gets confused here (I'm sure someone will), psychotherapist are not the same thing as psychiatrist: the latter are actual doctors who can prescribe therapies, be they drug or other. They may not all be great either, but at least one that I've seen really knows his stuff.

It's my understanding that a good amount of social science does the same--I was recently in a class on organization theory that made few if any testable predictions and ended up being a bunch of just-so stories about how people in organizations think.

One of the side effects of prediciton markets of the Hansonian variety is that people would rapidly see who is doing good science and who isn't: wealth would be rapidly transferred from those who don't know how to make good evidence-centric predictions to those who do.

Remember Rorschach ink-blot tests? It's such an appealing argument: the patient looks at the ink-blot and says what he sees, the psychotherapist interprets their psychological state based on this. There've been hundreds of experiments looking for some evidence that it actually works. Since you're reading this, you can guess the answer is simply "No."

I checked. It doesn't say "No".

they are in fact more likely than anyone to assume that a dispute is a dispute about definitions.

I have to say, speaking from my experiences while I was working on a philosophy major, both reading the work of and holding discussions with professional philosophers, and comparing that to my experiences here, that this is overwhelmingly not the case.

And there's also no discernible difference between seeing a psychotherapist and spending the same amount of time talking to a randomly selected college professor from another field. It's just talking to anyone that helps you get better, apparently.

See also http://lesswrong.com/lw/94t/meta_analysis_of_writing_therapy/

Excellent analysis. I think your historical explanation is important- not the totality of the situation, but important. Problems with psychotherapy- 1) What are the goals? and 2) "I like what is happening" is a valid reason to consider it successful. Having to answer 1) above is not that unusual for a scientific endeavor. The utter subjective nature of 2) above makes it very unusual for a scientific study.

And there's also no discernible difference between seeing a psychotherapist and spending the same amount of time talking to a randomly selected college professor from another field. It's just talking to anyone that helps you get better, apparently.

Unless this has been tested for random people other than just college professors, there's a stronger case for saying that talking to a person of a certain intelligence and education level helps you get better. And I suspect that it doesn't generalise to "talking to anyone that helps you get better" but I haven't looked into it.

(I'm sure there are other factors, but I'm just going by what was said about college professors.)

Evidence-based "psychotherapy" would be a terrible thing. Who would have gotten the evidence? Professional academics pursuing their norms and their careers and with litlle daily contact with clients, who daily challenge the therapist by asking impossible questions and doing impossible things. This so called body of evidence becomes creed - and must be applied by all to Joe & Jim, regardless of what Joe and Jim says.

Evidence based therapy is a tyranny that transforms the therapist to an Eiiza (a zombie) twisting all problems into the already agreed upon solution. Talking to a bum on a park bench (or a mathematics professor) would indeed be better than this.