Your intuitions are not magic

Thanks for the well-written article. I enjoyed the analogy between statistical tools and intuition. I'm used to questioning the former, but more often than not I still trust my intuition, though now that you point it out, I'm not sure why.

i don't know why we prefer to hold on to our intuitions. your claim, that " we persist on holding onto them exactly because we do not know how they work" has not been proven, as far as I can tell, and seems unlikely. I also don't know why our own results seem sharper than what we learn from the outside [although about this later point, i bet there's some story about lack of trust in homo hypocritus societies or something] .

As somebody who fits into the "new to the site" category, I enjoyed your article.

Intuition seems to be one of the least studied areas of cognitive science, at least until very recently. The Wikipedia entry on cognitive sciences that the post links to has no mention of "intuition", and one paper I found said that the 1999 MIT Encyclopedia of Cognitive Sciences doesn't even have a single index entry for it (while "logic" has almost 100 references).

After a bit more searching, I found a 2007 book titled Intuition in Judgment and Decision Making, which apparently represents the current state of the art in understanding the nature of intuition.

Indeed, intuitions are fallible. Though beware of the other extreme: writing off your intuitions altogether and trying to live solely based on logic. I've seen various people in the LW sphere try this, and it doesn't quite work. In some cases, like nutrition or social life, there is a bottomless pit of complexity. Trying to provably 'solve' such problems will lead to a bottomless pit of thinking, stagnation, and depression.

Logic is not a magic hammer either.

The intuitions vs reasons debates often appear misguided to me. You've eloquently pointed out that intuitions result from the same black box of spaghetti code as reasons, namely the brain.

Having just pressed "Send" on an email that estimates statistics based on my intuitions, this feels particularly salient to me.

Really well written. Great work Kaj.

Thanks for reminding me that my thoughts aren't magic.

People who know a little bit of statistics - enough to use statistical techniques, not enough to understand why or how they work - often end up horribly misusing them.

How often do people harm themselves with statistics, rather than further their goals through deception? Scientists data-mining get publications; financiers get commissions; reporters get readers.

ETA: the people who are fooled are harming themselves with statistics. But I think the people want to understand for themselves generally only use statistics that they understand.

Elegantly done - clear and informative.

Hi! I'm new here and I'd like to thank for the site and for this instructive article in particular. I'm quite convinced that the overconfidence in our own intuition, even without knowing its underlying mechanisms, is one of the main obstacles to rational thinking. Maybe this blackbox working of our intuition also plays a role in communication of our ideas to others. How we can change someone's opinions if we don't know how they got them in first place?

Thanks again and congrats for your work!

The immediately available example supporting your article for me is the relationship between dietary cholesterol and blood cholesterol. There's high general confusion around this health claim.

What no doubt compounds the confusion on the issue is that intuitively you might infer that eating zero cholesterol should lower blood cholesterol, or that eating high cholesterol should raise blood cholesterol. Evidence shows this often happens, but not always. There are enough notable outliers that the claim has been defeated in the general mind because it doesn't support the intuitive story.

That is, vegans who eat almost no cholesterol containing foods, can have high blood cholesterol. On the flip side, surely everyone has heard of that friend of a friend who eats inf eggs a day and has low blood cholesterol.

There's a reasonably interesting story that fits the evidence for the claim that if dietary cholesterol then blood cholesterol, but the nonlinearity of the relationship and also the incidence of intuition defeating cases cloud the issue.

I enjoyed your article and as a scientist, I've been interested to understand this: what seems an intuitive method to use to solve a scientific problem is not seen as an intuitive method while solving 'other' problems.

By 'other', I mean things like psychological problems or problems that arise from conflicts amongst people. It may be obvious why it is not 'intuitive' but what goes beyond my understanding is most will not even consider using the scientific method for the latter types of problem ever.

Your first idea is always in doubt seems like a nice law. But then I think is it a nice law cause my first idea was that it was nice.

Yet in general, if people are asked about the relative number of restaurants in various fast-food chains, their estimates generally bear a close relation to the truth.

The link is broken. Is it this article https://psycnet.apa.org/record/1992-18641-001 ?

I originally learned about these ideas from Thinking Fast and Slow, but I love hearing them rephrased and repeated again and again. Thinking clearly often means getting in the cognitive habit of questioning every knee-jerk intuition.

On the other hand, coming from a Bryan Caplan / Michael Huemer perspective, aren't we kind of stuck with some set of base intuitions? Intuitions like; I exist, the universe exists, other people exist, effects have causes, I'm not replaced by a new person with memory implants every time I go to sleep...

You might even call these base intuitions, "magic," in the sense that you have to have faith in them in order to do anything like rationality.

I think this article doesn't quite appreciate the full role intuitions play in science. It seems to me that intuitions help shape science in large ways. For instance, our intuitions that 'deduction works' and 'induction works' seems to stop all of us from turning into Cartesian sceptics, and preventing any science. Intuitions (and philosophical arguments) about metaphysics shape the basis of acceptable hypotheses within physics. Intuitions about what makes a scientific theory good/explanatory/falsified shape how science proceeds. Intuitions also serve to define concepts we have. If I remember correctly, in the Newtonian era, mass was not analysed in terms of anything else. It was a primitive concept in Newton's physics, and it was defined intuitively. Nowadays, modern physics has analysed concepts in terms of more and increasingly obscure concepts; but nevertheless, there is always a limit to what has been analysed in terms of what, and what remains is held, insofar as we know of it, as known primitively. That is, known intuitively.

I also have a question: Does this site in general take a negative view of heuristics humans have? I've seen various pages complaining about heuristics humans have, and not much about how helpful they are in keeping us all functioning.

This was an excellent read- I particularly enjoyed the comparison drawn between our intuition and other potentially "black box" operations such as statistical analysis. As a mathematics teacher (and recreational mathematician) I am constantly faced with, and amused by, the various ways in which my intuition can fail me when faced with a particular problem.

A wonderful example of the general failure of intuition can be seen in the classic "Monty Hall Problem." In the old TV game show Monty Hall would offer the contestant their choice of one of three doors. One door would have a large amount of cash, the other two a non-prize such as a goat. Here's where it got interesting. After the contestant makes their choice, Monty opens one of the "loosing" doors, leaving only two closed (one of which contains the prize), then offers the contestant he opportunity to switch from their original door to the other remaining door.

The question is, should they switch? Does it even matter? For most people (myself included) our intuition tells us it doesn't matter. There are two doors, so there's a 50/50 chance of winning whether you switch or not. However a quick analysis of the probabilities involved shows us that they are in fact TWICE as likely to win the prize if they switch than if they stay with their original choice.

That's a big difference- and a very counterintuitive result when first encountered (at least in my opinion)

Yet our brains assume that we hear about all those disasters [we read about in the newspaper] because we've personally witnessed them, and that the distribution of disasters in the newspapers therefore reflects the distribution of disasters in the real world.

Even if we had personally witnessed them, that wouldn't, in itself, be any reason to assume that they are representative of things in general. The representativeness of any data is always something that can be critically assessed.

"However, like with vitamin dosages and their effects on health, two variables might have a non-linear relationship."

if we limit our interval we can make a linear approximation within that interval. this is often good enough if we don't much care about data outside that interval. the easy pitfall of course is people wanting to extend the linearization beyond the bounds of the interval.

Interesting text, but I'm getting a nagging intuition here regarding what assumptions you are using for correct reasoning. Correct me if I am wrong, but is your assumption that your assumption for normative rationality (both decision making and inquiry) is one of pure statistical and logical inference, and a deviation from this norm you consider a fallacy.

Is this correct of me to assume?

Yudkowsky's intuitions are always right and thus magic.