Generalized Efficient Markets and Academia

The mousetrap thing seems pretty analogous - If I build a better mousetrap, but can't convince people it's better, it seems similar to building a better theory and not being able to convince them it's better. Although, perhaps it's easier in the second case, as the people I have to convince in the second case are experts, who presumably it's cheaper to educate than laymen (on the other hand, I may have to spend less on uneducating laymen.)

Reminiscent of cases where it's hard to short something: you may be able to tell a theory is bunk, but not have any easy way to "short" (provide an intuitive explanation / design a simple refuting experiment)

One nice property of markets is that it only takes a small number of people to remove an inefficiency. Once the better, cheaper mousetrap hits the market, the old mousetraps become obsolete; the new mousetrap is adopted even if there are initially many more old-mousetrap-makers than new-mousetrap-makers. Same with the stock market: even if most investors don’t notice a pattern, it only takes a handful to notice the opportunity and remove it.

This is very very wrong.

Knowing that there's an inefficiency is only part of the battle. In order of importance to counter your point:

• You need to have enough volume to remove the inefficiency. For a trading strategy, this is the capital you can allocate to this strategy specifically. It's not obvious how many hedge funds, for example, are capital constrained. In the mousetrap example this means actually producing the sufficient volume of mousetraps.
• The opportunity costs matter. The people positioned best to notice the inefficiency are also positioned best to notice others. May be there is a good trading strategy but it's *less* than what you would make with your other strategies. So you wouldn't actually trade it, leaving it for others to pick up. Same with the mousetrap: may be you go build a cat trap, or may be you don't build anything but make money consulting.
• The inefficiency removal needs to stick. If you're trading a strategy and then stop, the inefficiency will likely come back. You might stop for a multiple number of reasons.
• The execution is important. Let's say you can analyze the data perfectly and come up with a clever algorithm. You'd still need to figure out how to execute on the actual trading. Part of it is not introducing too many exploitable opportunities for other trades (e.g. market makers).

Overall, I'd say even if an inefficiency is publicly known, it usually takes a non-trivial amount of people*effort to remove all of it.

Two of these things are not like the others. People can have cognitive biases or bounded rationality that prevent them from making good decisions about mouse trap purchases despite it costing money. Unlike academia there is a direct incentive for these people to change their behavior; but that's as far as it goes. There's no good way to correct their decisions for profit like there is in a financial market.

How confident would you be that a new company that produces a mouse-trap that's 10% better at catching mice would be able to articulate the advantage in a way to get their new mouse-trap design to spread?

I'm very doubtful that's the case. It's very hard to know about that advantage for regular customers who stand in front of a shelf that provides different mouse-trap's for sale.

It took the market a long time to adapt inventions as great as sliced bread. Donald Norman layed out in The Design of Everyday Things how it's easy to build doors where it's clear which side has to be pushed and which side has to be pulled. It still largely gets ignored and plenty of doors get build in ways where it's not easy for the user to know whether they have to push or pull.

The market isn't even strong enough to be able to clear other obviously bad door designs like door knobs to the extend that Australia went to forbid them in their new building codes.

Most people in the field can’t distinguish a predictive theory from phlogiston [LW · GW]

Plogiston is predictive. It's much better at predicting then thinking in terms of the four elements.

Your claim about (science) academia is that is generally unable to do some things that it clearly should be able to do, if it is doing its job correctly. Would you care to provide some evidence for it?

Efficient markets work mostly only in theory. Traders and Investors that find spots to exploit market inefficiencies hold them as secretive as possible. Those market inefficient often also don't allow high volumes of money. So its often closed private funds that truly discover and use such spots for years.

Open Investment funds often try to sell exactly this Idea, but they are more sales people that money managers.

Another problem arises if the smart money is too small to make a difference. E.g. in January many smart investors started to play the Covid-19 Spot. Longing companies like Zoom and shorting SP500. Zoom Stock went up immediately because that handful of smart money was enough to move the price. But the SP500 continued to make new All time highs since the hand full of smart investor money was laughable against the mass of auto invest strategies into the SP500.

Another distinction is that in Science participants go a long way to explain to the other party why their theory is superior, there is a direct incentive of recognition and contributing to the field. In Investing the truly smart money is doing the opposite they try to stay as secretive as possible. The public discussions about economics by Traders, Investors and Economist we often witness are either pure marketing for their fund(Traders/Investors). Handwaving reasoning (Economists).