Feed the spinoff heuristic!

If psychology worked, I would expect marketing firms to use it to make millions of people buy tons of shit that they don't need and that won't make them happy.

Shorter version of OP's argument.

If personality psychology holds water, I would expect dating sites to use it and produce better results than Traditional Romance. Does it? From the outside looking in, it looks like it does.

It would also be useful in selecting dorm room compatibility, which I can tell you from the inside looking out does not work at all or isn't being used. I wouldn't expect it to be used in this context, though. No money in it.

There's interesting thing: Some people managed to acquire a lot of wealth via trading. That would lead you to believe their claims with regards to the methods they use being effective.

However, if you simulate the stock market with identically skilled agents, you obtain basically same wealth distribution as observed in the real world, with some few agents ending up extremely 'rich'. One can imagine that such agents, if they were people, would rationalize their undeserved wealth to feel better about themselves.

I would add to this that having a method would often (but not always) produce a clear "leader in the field" (first-mover advantage going to the discoverer). So seeing Google's share of the market is a strong indicator (even without first-hand knowledge) that "they have a serious advantage in search" vs existence of many competing diet companies does not tell me "they figured out nutrition".

Two issues with this heuristic:

1) It doesn't work well for credence goods.

2) Sometimes it takes a long time for sciences to find an application, two modern examples are astrophysics, and particle physics.

Tetlock's political judgment study was a test for macroeconomics, political science and history. Yet people with PhDs in these areas did no better on predicting macro political and economic events than those without any PhD. Maybe macro helps in producing good econometric models, but it doesn't help in making informal predictions. (Whereas one suspects that physics and chemistry would help in a test of quick predictions about a novel physical or chemical system, vs. people without a PhD in these fields).

Re: your examples successful spin-offs for psychology, to what extent did these therapies come out of well-established theory? Maybe someone can weigh in here. It seems possible that these are good therapies but ones that don't have a strong basis in theory (in contrast to technologies from physics or chemistry).

Physicist Ilya Prigogine developed his famous theory of dissipative systems which was expected to explain a lot of things from thermodynamics of living systems to the nature of the arrow of time. It is a very well-developed and deep theory. Yet, in my scientific life, I have never seen an actual numerical calculation of a measurable quantity utilizing any of Prigogine's concepts such as "rate of entropy production". Looks definitely like a missing spinoff!

Computer vision is suspiciously lacking in practical spinoffs, even though people have been studying it for 40 years.

I really like this. It emphasizes the fundamentally instrumental nature of rationality.

If I was initially lacking in the capital to do trades, I could publish my predictions online using public key cryptography and amass an impressive track record before recruiting investors.

This is a nitpick, but this protocol is at least underspecified. Aside from the need to prove that you made the predictions before the events, you also need to be able to prove that you made no other predictions before the event.

(I've always wondered why no pump-and-dump scammers use this: after ten "buy/short " mails, 1/1024 of your mailing list will have received 10/10 correct predictions from you (and another 10/1024 will have received 9/10 correct predictions.) Which should be enough to convince quite a few to buy up some penny stock (with the scammer taking the other, profitable side of the trade.) In the spirit of this post, it's probably not profitable enough. Or spammers are stupid.)

It seems to me that there are two different heuristics here and it is worth separating them.

But first I should explain why I think my initial reading of this post suggests heuristics that I think are problematic. The mere existence of CBT does not seem like strong evidence for psychology. It is no more evidence for modern mainstream psychology than freudian psychoanalysis is evidence for freudian psychology. As I understand it, CBT is gaining market share against other forms of talk therapy, but largely because of academic authority, roughly the same way that the other therapies got established. I am a fan of CBT because its proponents claim to do experiments distinguishing its efficacy from that of other talk therapies and failing to distinguish other talk therapies from talking to untrained people (which is still useful). But why do I need CBT for that? I can check that mainstream psychologists are more enthusiastic about experiments than freudian ones without resorting to the particular case of CBT. Similarly, competing nutritional theories are successful in the marketplace, sold both by large organizations with advertising budgets (Weight Watchers vs Atkins) and personal trainers working by word of mouth. But I agree that they example of CBT sheds light on psychology.

One heuristic is that experiments with every-day comprehensible goals are more useful for evaluating a field than experiments of technical claims. Most obviously, it is easier to evaluate the value of the knowledge demonstrated by such experiments than technical knowledge. Knowing that statins lower cholesterol is only useful if I trust the medical consensus on cholesterol, but knowing that they lower all-cause mortality is inherently valuable (though if the population of the experiment was chosen using cholesterol, this is also evidence that the doctors are correct about cholesterol). Similarly, the efficacy of CBT shows that psychologists know useful things, and not just trivia about what people do in weird situations. Moreover, I suspect that such experiments are more reliable than technical experiments. In particular, I suspect that they are less vulnerable to publication bias and data-mining. Certainly, I have to learn about technical measures to determine how vulnerable technical experiments are to experimenter bias.

The other heuristic is that selling a theory to someone else is a good sign. Unfortunately, this seems to me of limited value because people buy a lot of nonsense, not just competing psychological and nutritional theories, but also horoscopes. How does the military differ from academic psychologists? I'm sure it hires a lot of them. They do much larger and longer experiments than academics. They do more comprehensive experiments, with better measures of success, analogous to the advantage of all-cause mortality over number of heart attacks (let alone cholesterol). They could eliminate publication bias because they know all the studies they're doing, but only if the people in charge understand this issue; and there is still is some kind of bias in the kind of studies they let me read. These are all useful advantages, but in the end it does not look very different to me than the academic psychology we're trying to evaluate. Similarly, industry consumes a lot of biological and chemical research, which is evidence that the research is, as a whole, real, but it fails to publish attempts to replicate, so the information is indirect. On the other hand, these industries, like the military, use the knowledge internally, which is better evidence than commercial CBT and nutrition, which try to sell the knowledge directly, and mainly demonstrate the value of academic credentials to selling knowledge.

I have been unable to find any practical spinoffs of gender studies.

I think it's interesting topics for research papers, I've read something like that here: https://essays-service.com/blog/540-argumentative-essay-topics. It's great that students conduct similar studies. There are currently no qualitative content that is interesting to learn.

I think it's interesting topics for research papers, I've read something like that here: https://essays-service.com/blog/540-argumentative-essay-topics. It's great that students conduct similar studies. There are currently no qualitative content that is interesting to learn.

There's this great XKCD that totally makes this exact same point, except with more lolz.

If stock market economics worked, Noble prize winners would make money.

(This is slightly unfair, since the Black-Merton-Scholes theory does make other people money, to an extent. Additionally, while Merton and Scholes were on the board, LTCM was not strictly based on their theories. Still, surprised to see that this hasn't been mentioned.)

Did you steal this from XKCD?

Related: the generative heuristic.

Bem tells us that random undergraduate students were able to predict with 53% accuracy where an erotic image would appear in the future. If this effect was actually real, I would rerun the experiment before corporate earnings announcements, central bank interest rate changes, etc, and change the images based on the reaction of stocks and bonds to the announcements. In other words, I could easily convert "porn precognition" into "hedge fund trillionaire precognition."

This doesn't just assume that the effect is reproduceable, it assumes that the effect generalizes to things other than erotic images. Considering that erotic imagery gets special treatment in our brain's processes that finance does not, this seems like a dubious assumption even given the premise that the effect is real.

In other words, I could easily convert "porn precognition" into "hedge fund trillionaire precognition."

Not if psi is capricious, and the evidence suggests it is. (I say this to emphasize that psi is singular in this respect; your heuristic might work for other fields.) (ETA: I guess macroeconomics has similar problems.)

(ETA2: Think about it from the simulation hypothesis perspective: you're trying to manipulate the gods into doing something for you. You're dealing with transhumanly intelligent agents. It's likely not a good idea to try to be clever.

Black magic is not a myth. It is a totally unscientific and emotional form of magic, but it does get results — of an extremely temporary nature. The recoil upon those who practice it is terrific.
It is like looking for an escape of gas with a lighted candle. As far as the search goes, there is little fear of failure!
To practice black magic you have to violate every principle of science, decency, and intelligence. You must be obsessed with an insane idea of the importance of the petty object of your wretched and selfish desires.
I have been accused of being a "black magician." No more foolish statement was ever made about me. I despise the thing to such an extent that I can hardly believe in the existence of people so debased and idiotic as to practice it.

Aleister Crowley)