Expert Prediction Of Experiments

Thanks for the comment. It's humbling to get a glimpse of vastly different modes of thought, optimized for radically different types of problems.

Like, I feel like I have this cognitive toolbox I've been filling up with tools for carpentry. If a mode of thought looks useful, I add it. But then I learn that there's such a thing as a shipyard, and that they use an entirely different set of tools, and I wonder how many such tools people have tried to explain to me only for me to roll my eyes and imagine how poorly it would work to drive a nail. When all you have is a hammer, everything is seen through the lens of the nail-driving paradigm.

My hindsight bias thinks it's obvious that Tetlock's questions might bias the sample toward foxes over hedghogs). He's asking people to make predictions about a wide range of domains. I predict that, if you asked leading experts in various subfields to make predictions about the results of experiments being conducted within their subfield, they would trounce outsiders, especially if the questions were open ended. (By open ended, I mean "how will the economy change if X happens" rather than "Will housing prices go up or down if X happens".)

It would be interesting if they also could make better predictions on similar types of problems. For example, are there "soft" problems, with thousands of variables and thousands of unknowns, which experts in the "soft" sciences are systematically better at solving?

It seems plausible that the "hard sciences" mental models necessary to solve problems with a couple of variables and 1 or 2 unknowns might not work well for soft problems. After all, the 2 mental architectures useful for these look radically different. The point of associative reasoning isn't to evaluate which of the first couple hypothesis is most likely. It's to make sure that the bigest factors are even on your list of potentially influential variables. Simply evaluating a couple random items on a long tailed Pareto chart of solutions will radically underpreform even the briefest seach through hypothesis space. The point is to be able to narrow down the space of hypotheses from trillions to a few dozen. It's the "explore" side of the explore/exploit tradeoff.

It would be interesting to test this, however. What happens if you take a Nobel laureate with a prize in medicine, and another with a prize in physics, give them enough funding to conduct a large number of experiments, and ask them both to spend a couple years trying to solve a wicked problem in an unrelated field? Does the physicist overestimate how applicable his or her knowledge is to the problem? Might the medical researcher take a more high-level, holistic approach, and have higher odds of success? Perhaps a less thorough but easier experiment would be to see how the two types of minds performed on candle problems.

(Also, on my second read through of your comment, I noticed an open parenthesis (which creates an unresolved tension which will stay with me all day).