How good is a human's gut judgement at guessing someone's IQ?

post by habryka (habryka4) · 2019-02-25T21:23:17.159Z · score: 45 (17 votes) · LW · GW · 15 comments

This is a question post.

Contents

  Answers
    33 Thrasymachus
    10 ryan_b
    6 Ruby
    1 MrNoodle
None
9 comments

In her latest post, Sarah Constantin writes:

Whatever ability IQ tests and math tests measure, I believe that lacking that ability doesn’t have any effect on one’s ability to make a good social impression or even to “seem smart” in conversation.

I remember that I made the opposite prediction a few months ago, while I was studying psychometrics more actively. In particular, I had the sense that because of the positive manifold (i.e. almost all positive mental attributes quite strongly correlate), it seems like judging someone's IQ should be relatively easy, and also quite valuable in assessing a large number of other positive qualities, which should make it quite evolutionarily advantageous to be able to assess it.

Two concrete experiments I've thought about:

1. Take a group of people whose IQ you now, then just take pictures of their faces (or short videos of them saying something), and then have a group of participants rate them on their intelligence (probably in order). See how strong they correlate.

2. Throw a bunch of participants into a group, have them talk to each other, then afterwards ask them about the relative judgement of the intelligence of the other members of the group, see how predictive they are.

These seem like very straightforward experiments, that I can imagine someone having already run, and where I would be pretty interested in the results. If only so that I can be better calibrated on how much to trust my gut judgement of someone's intelligence.

Does anyone know of any similar experiments that have been run?

Answers

answer by Thrasymachus · 2019-02-26T01:55:18.592Z · score: 33 (12 votes) · LW · GW

Googling around phrases like 'perception of intelligence' seems to be a keyword for a relevant literature. On a very cursory skim (i.e. no more than what you see here) it seems to suggest "people can estimate intelligence of strangers better than chance (but with plenty of room for error and bias), even with limited exposure". E.g.:

Perceived Intelligence Is Associated with Measured Intelligence in Men but Not Women (Note in this study the assessment was done purely on looking at a photograph of someone's face)

Accurate Intelligence Assessments in Social Interactions: Mediators and Gender Effects (Abstract starts with: "Research indicates that people can assess a stranger's measured intelligence more accurately than expected by chance, based on minimal information involving appearance and behavior.")

Thin Slices of Behavior as Cues of Personality and Intelligence. (Short 1-2min slices of behaviour in a variety of contexts leads to assessments by strangers that positively correlate with administered test scores for IQ and big 5)

answer by ryan_b · 2019-02-26T15:53:40.166Z · score: 10 (6 votes) · LW · GW

The amount of effort people invest in seeming smarter than they are, for example in situations like hiring or sales, suggests to me that people's gut judgments are easy enough to fool to be worth it.

I expect that what people are actually doing is using a few heuristics to make the judgment. I would also expect that courtesy of things like Dunning-Kruger, people towards the bottom will be as bad at estimating IQ as they are competence at any particular thing.

If we just stick with the intuition provided by fact that some people are really terrible at guessing this sort of thing and some people are not, I further expect that what people use as a heuristic changes with ability level. In example: people at the bottom might base it on whether someone agrees with them about stuff; those in the middle might focus heavily on whether someone has a smart-person job like doctor or lawyer; people towards the top might look more at the way someone does things, like some sign that they chose to think carefully about it or apply some technique known to them.

comment by ESRogs · 2019-04-03T00:09:16.628Z · score: 4 (2 votes) · LW · GW
I would also expect that courtesy of things like Dunning-Kruger, people towards the bottom will be as bad at estimating IQ as they are competence at any particular thing.

FWIW, the original Dunning-Kruger study did not show the effect that it's become known for. See: https://danluu.com/dunning-kruger/

In particular:

In two of the four cases, there's an obvious positive correlation between perceived skill and actual skill, which is the opposite of the pop-sci conception of Dunning-Kruger.
comment by ryan_b · 2019-04-03T13:29:43.626Z · score: 4 (2 votes) · LW · GW

The weaker claim is what holds in my example: in 4 out of 4 cases, the bottom quartile was very wrong in their assessment. I was speculating as to possible causal explanations; I think it is how they do the assessment.

Looking at those graphs, I can think of one rule that explains all four groups: say you are a above average when asked does the trick.

answer by Ruby · 2019-05-03T19:00:22.232Z · score: 6 (3 votes) · LW · GW

Faception is a startup which claims to have developed machine learning algorithms that can classify IQ and other things (including "academic researcher", "professional poker player", and "terrorist").

If it is the case that AI can actually do this well, then I'd take that as evidence that humans might be capable of it too.

(From a quick look it's unclear how successful they are. They seemed to be focused on military/security applications, i.e. detecting terrorists.)

answer by MrNoodle · 2019-04-09T16:06:46.384Z · score: 1 (1 votes) · LW · GW

Research on how humans attribute intelligence to each other suggests that most people attribute higher intelligence to efficient agents, i.e., agents whose actions most resemble what we believe is the best course of action in an uncertain situation. A minority attributes intelligence to agents based on the outcome of their actions.

People attribute intelligence to efficiency in proportion to their own ability to plan. Less competent planners attribute intelligence to outcome. To elaborate, a competent planner would evaluate the behavior of an agent and understand whether it acted rationally, regardless of its success. On the other hand, a bad planner cannot infer the reasoning of an agent; since he himself cannot foresee the best course of action, he would not understand why an agent made a particular decision, and would likely attribute high intelligence to an irrational but lucky agent since its behavior would result in a good outcome.

My interpretation of these results is that intelligent people are better at evaluating the intelligence of others. I believe that given enough time (and thus enough information to evaluate) an intelligent person can adequately estimate a peer's intelligence. It could be argued that "seeming" smart requires a sufficient degree of awareness and intelligence, although of a different kind; perhaps IQ tests do not have a wide enough scope to accurately measure different intelligence in its different forms.

Source:

Kryven, Marta. "Attributed Intelligence." (2018).

15 comments

Comments sorted by top scores.

comment by jimrandomh · 2019-02-26T03:38:50.509Z · score: 34 (12 votes) · LW · GW

Peoples' experiences with this are majorly confounded, and your experiment risks also being confounded, by the Conservation of Virtue effect. While it's true that mental attributes, health, height, wealth, and so on all positively correlate, within any filtered social group, those things tend to *negatively* correlate instead, because someone whose composite-trait-score is too low or high for a group will tend to end up in a different one. So, estimating IQ based on a conversation will tend to work well when the person whose IQ you're estimating comes from an unfiltered population, and poorly when it comes from a filtered one.

comment by Benito · 2019-02-26T11:37:58.452Z · score: 11 (5 votes) · LW · GW

"Conservation of Virtue effect" <- aka Berkson's paradox.

comment by habryka (habryka4) · 2019-02-27T01:03:41.937Z · score: 4 (2 votes) · LW · GW

Good connection to make. Thanks a lot for providing the more formal reference.

comment by ESRogs · 2019-04-03T00:19:17.970Z · score: 5 (3 votes) · LW · GW
Whatever ability IQ tests and math tests measure, I believe that lacking that ability doesn’t have any effect on one’s ability to make a good social impression or even to “seem smart” in conversation.

That section of Sarah's post jumped out at me too, because it seemed to be the opposite of my experience. In my (limited, subject-to-confirmation-bias) experience, how smart someone seems to me in conversation seems to match pretty well with how they did on standardized tests (or other measures of academic achievement). Obviously not perfectly, but way way better than chance.

comment by johnswentworth · 2019-02-27T01:12:31.494Z · score: 4 (2 votes) · LW · GW

It's been a few years, so I don't have sources to cite, but I remember looking into this at one point and finding that immune health during developmental years is a common major underlying cause for intelligence, attractiveness, physical fitness, and so forth. This makes a lot of sense from an evolutionary standpoint: infectious disease used to be the #1 killer, so immune health would be the thing which everything else traded off against.

One consequence is that things like e.g. attractiveness and intelligence actually do positively correlate, so peoples' halo-effect estimates actually do work, to some extent.

comment by Andile Riley (andile-riley) · 2019-04-07T16:06:01.241Z · score: 1 (1 votes) · LW · GW

Are you sure that the correlation between immune health during childhood and all of those positive attributes is causal?

Superficially, it seems more plausible that the causation starts much earlier, via mutational load or specific conditions during the gestation period. Though I suppose that it is also possible that health in the womb and health during early childhood could both effect adult outcomes. of these could have an effect as well.

comment by johnswentworth · 2019-04-07T16:09:50.997Z · score: 2 (1 votes) · LW · GW

Yeah, sorry, I should have been more clear there. The mechanisms which trade off immune strength against other things are the underlying cause. Testosterone levels in males are a good example - higher testosterone increases attractiveness, physical strength, and spatial-visual reasoning, but it's an immune suppressor.

comment by Pattern · 2019-02-26T00:03:10.399Z · score: 2 (1 votes) · LW · GW

A record of transcripts might be useful (to make sure people aren't figuring it out based on correlation (ACT/SAT scores, level of education, etc.).

comment by conjectures · 2019-04-23T08:18:24.500Z · score: 1 (1 votes) · LW · GW

I'm a little concerned that this could lead to operationalization of the fundamental attribution error. If you're in a conversation with someone who says intelligent things, does it matter if they are an intelligent person?