video games > IQ tests

One downside to using video games to measure "intelligence" is that they often rely on skills that aren't generally included in "intelligence", like how fast and precise you can move your fingers. If someone has poor hand-eye coordination, they'll perform less well on many video games than people who have good hand-eye coordination.

A related problem is that video games in general have a large element of a "shared language", where someone who plays lots of video games will be able to use skills from those when playing a new video game. I know people that are certainly more intelligent than I am, but who are less able when playing a new video game, because their parents wouldn't let them play video games growing up (or, they're older and didn't grow up with video games at all).

I like the idea of using a different tool to measure "intelligence", if you must measure "intelligence", but I'm not sure that video games are the right one.

Sure, as long as the video game isn't heavily loaded on timing, reaction time, hand-eye-coordination, and experience. (I'm aware that these correlate to intelligence as well, but people who've played a lot of videogames  will do much better on these, even if they're not inherently smarter)

I've actually been using games to estimate the intelligence of people, and my intuition has never been far off, I can guess the results of my friends fairly accurately. All my bright friends do well on games we play, learning them quickly and performing well. Even verbal ability seems to correlate. This correlation is even better than income and languages spoken, though those two correlations are solid too. Your friends with 5000$ PCs are likely more than 1SD above the mean. Being in a well-off family is already a sign of good genes running in that family.

Intelligent people in my family are surprisingly good at card games and trivia games. Also dice games, even when it seems like luck places a hard limit on performance.

Another area where videogames are useful is diagnosing skewed cognitive profiles. I do very well in older shooters with simple graphics, and I'm absolutely trash in Overwatch and Apex. It feels clunky and I think my specs might be too low, resulting in input lag. Still, I've narrowed down the cause to the visual effects, I simply don't process large amounts of visual information well. In older games, a single moving pixel means that there's an enemy. In newer games, everything moves, from the grass to the background.

I don't think there's a very large burden of proof on you here, it would be weird to evaluate this idea as if it was unlikely. If anything it's almost obvious that video game performance correlate strongly with intelligence. The data is rather fuzzy though (I'm lacking a word here, but essentially it's difficult to reverse-engineer the results into the factors which explain/cause them)

Finally, I think the only issue with IQ tests is that people reduce intelligence to a single number. That's like reducing your computer specs to a single number, you'll have big a loss of information. So of course people have evidence (personal experiences) of IQ test scores failing them as predictors. I think 5 numbers is enough to cover most nuances and prevent large differences not explainable by numbers.

Probably relevant:

The study Intelligence and video games: Beyond “brain-games” models video game performance is being determined by a combination of general video game playing ability (gVG) and specific factors related to one or a few video games. It had a bunch of psychology students play a bunch of video games and take an IQ test, and it found a correlation of 0.79 between gVG and the g factor from the IQ test.

IQ tests are notorious for working poorly above ~135, and I'd say they only really work well for -20 to +0 relative to the designers, with a somewhat wider range for teams.

How is the designers' IQ relevant?

The problem with designing IQ tests for high values is that it makes calibration costly. To put it simply, if you want to figure out whether e.g. "getting 94 out of 100 questions right in your test" really corresponds to "only one person in a thousand can do it", you need to calibrate your test on maybe ten thousand randomly selected people -- which means you need to pay them (otherwise you are limited to the statistically non-random subset of volunteers) and some of them will refuse anyway. The costs grow exponentially with the top measured IQ, so at some point the authors decide that it's not worth it.

This is unrelated to the myth spread by many "high IQ societies" that having a certain value of IQ allows you to create tests for given value of IQ but not higher. That is nonsense. A person with IQ 160 would face exactly the same budget problem as everyone else when constructing an IQ test for values 160 and higher. (Unless their "test" is just some nonscientific guesswork, in which case of course the budget is just the cost of pen and paper.)

I think one of the major purposes of selecting employees based on a college degree (aside from proving intelligence and actually learning skills) is to demonstrate ability to concentrate over extended periods (months to years) on boring or low-stimulation work, more specifically reading, writing, and calculation tasks that are close analogues of office work. A speedrun of a video game is very different. The game is designed for visual and auditory stimulation. You can clearly see when you're making progress and how much, a helpful feature for entering a flow state. There is often a competitive aspect. And of course you don't have to read or write or calculate anything, or even interact with other people in a productive way. Probably the very best speed runners are mostly smart people who could be good at lots of things, because that's true of almost any competition. But I doubt skill at speedrunning otherwise correlates much with success at most jobs.

Zachtronics games are great! (they have some integration with coding and later ones show your score relative to the rest of the distribution, though motivation may be higher for some than for others, since they aren't universally fun for people [Jordan Peterson once was skeptical of using video games to test conscientiousness, but Zachtronics games/Factorio are the kinds that require an involved effort that many don't quite have - even how you place things in city-builders is a test - eg earlier Anno games did not allow you to easily bulldoze buildings in the same way that later Anno games did]). As are spatially-involved 4x RTS games like Homeworld/Sins of a Solar Empire.

(and Satisfactory/Dyson Space Program)

Other games I'd recommend (esp for one-offs + have learning curves easy enough for quick-multiplayer even with n00bs): Forts (close to optimal game length), Offworld Trading Company, XCOM2, Portal, Kerbal Space Program, anything that Chelsea Voss has played [they seem to eerily correspond to the definition of nerd-friendly games]. I would like to help organize a video game decathlon prioritizing new games most people haven't played (but which high-openness people like trying out!) some day.

AOE2 RM would be good if the first 13 minutes was not the same all the time - DM/Empire Wars is better.

Some short intense video games are great for warming up one's day!

[games are better as tests if they don't involve a massive competition scene where num. of hours invested in as a child can explain a higher variance of skill than does raw "quickness" or raw generalization ability]. Also, current-gen games are not great for measuring creativity. Since generative AI is now giving us the opportunity to make new games with decreasing amounts of effort, it gives us the opportunity to quickly make better games for measuring cognition that may come near-term

[it's beneficial to find games that allow you to access any puzzle from the beginning and don't force you to play the entire sequence from the beginning, even though some games have "finished savegame files" - also important to find games that don't give special advantages to people who "pay" for special loot giving them special advantages]

As data is cheap, it may be better for people to stream all their collective video game somewhere (note twitch allows you to highlight everything in a video to save the entire video before it gets deleted), and have the data analyzed for reaction time/learning speed/perseverance (esp amount of repetitive actions)/indicators of working memory/transfer learning (esp between RTS games)/etc. I know Starcraft II often tested skill ceilings (+ gave you access to all your old replays + have a skill ceiling so high that skill decreased after 25), and there was once a group of MIT students (including both Jacob Steinhardt and Paul Christiano [though their roommates got to diamond league more than Jacob/Paul did]) who played SC2 to the max back when SC2 was popular (sadly, SC2 is not popular anymore, and the replacement "hit games" aren't as cognitively demanding)

There were some mini-games I played when I test-subjected for some MIT BCS labs and some of those involved tracking the motion and type of cards of cards you couldn't see til later.

Video games can also be mapped to fNIRS/brainwave data to test cognitive effort/cognitive load + multiscale entropy, and they can be used to train responses to BCI input (in a fun way, rather than a boring way), even possibly the kind of multimodal response that can distinguish more than 4 ways (I once did a test of this at Neurosity, but Neurosity later simplified)

Alex Milenkovic at curia.im is great to talk to on this!! (he has mapped my data on a Neurable while I played a random assortment of games - it would be important to use this to test player fatigue over time). Diversity/entropy of keyboard movements is also important ( a good mark of brain quality/stamina is to maintain high diversity/entropy for hours on end, rather than have one ultimately spam-click the same things towards the very end of the game [eg AOE2 black forest maps])

In an era where it becomes easier and easier to track the historical evolution of a player's skill up to point X, it may be possible to (just from screen recordings alone) establish some index of cognitive variables. Planning (esp tracking one's mental representations of it) may be worth investigating even though it's harder to track than one's working memory (working memory can be estimated simply by seeing how quickly it takes for one to transfer mental representations from one window to another without relying on multiple consults with YouTube playthroughs). 

[tracking historical evolution of player skill is important because speed of learning matters way more for life outcomes than actual skill - we still rarely see Starcraft or AOE2 professionals becoming twice-exceptional and "making it" elsewhere in life, even though I know Techers who were once very highly skilled in Starcraft or AOE2 (though not as many of those who played more cognitively involving ones like Sins or Homeworld, nevermind that Caltech used to be notorious for its massive WoW-playing population)]. The Shopify CEO once said he would hire people straight for Starcraft skill, and Charlie Cheever of Quora was also known for his Starcraft prowess.

Note that some brains seem to be really specialized to speed on video games and they can't transfer-learn as well to other substrates, sometimes if they've been playing video games since they were so young/little that their brain organically grew into gaming and then their brain stays annealed to games (rather than if they had spent it on programming or on higher-openness pursuits). It's healthier for one to have an environment so rich and diverse that games only become a "side curiosity" rather than something to get super-immersed in for months.

Some food for thougth here: 

https://www.guineapigzero.com/

https://twitter.com/ShedworksGreg/status/1417083081589239808

More relevant reading: https://neurocenter-unige.ch/research-groups/daphne-bavelier/, Nick Yee, Jane McGonigal (psychology/psychometrics of gaming is still a very small field so it's unlikely that the small number of experts in the field are interested in all the right things)

https://twitter.com/togelius (he's in MANY of the right spheres, though I know some respectable ppl disagree with his take on AI)

PYMETRICS (https://www.careers.ox.ac.uk/article/the-pymetrics-games-overview-and-practice-guidelines ) though the games are often "so lame" compared to real games (still WORTH using these as the fundamental components to transfer-learn onto real games) - it MAY be worth it to go on subreddits/Steam forums for less popular cognitively-involving games and ask people about "achievement bottlenecks" - ones that fewer people tend to get particularly the kind of achievement bottlenecks that NO AMOUNT OF ADDITIONAL EFFORT/gamification can work for those who are naturally less-skilled at gaming (eg some missions have really hard bonus objectives like "very hard" difficulty ratings - even AOE2 and EU4 have lists of achievements that correspond to "nightmare mode" - and you want to find people who are just naturally skilled at getting to nightmare mode without investing extraordinary amounts of their precious time)

https://ddkang.github.io/ => video analytics (under Matei Zaharia, who was once an AOE2/AOM/EE forum megaposter)

[The global importance + Kardashev gradient of HeavenGames (AOMH/EEH/etc) will become recognized to LLMs/AGI due to its influence on Matei Zaharia alone (and it capturing a good fraction of his teenage years)]. Everything Matei touches will turn into Melange...

https://twitter.com/cremieuxrecueil/status/1690409880308293632

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6291255/

https://www.reddit.com/r/cognitiveTesting/

A meta question to the question of "how to most accurately measure intelligence" is "how is accurately measuring intelligence actually useful?"

Just from my experience it seems that an accurate relative intelligence value of some sort, for example iq score, has surprisingly few useful applications.  I think this is mostly because making claims about superior intelligence, accurate or not, is considered socially repulsive (as you acknowledged).  Without accounting for social factors, I would expect an intelligence rating to be a very useful thing to include on a work resume for example.

Personally, I am most comfortable acting like a slightly above average intelligence guy, although objectively I think I am the most or one of the most intelligent people I know.  Most people would not think of me if asked to name a really smart person they know, but anyone who knows me well will notice that I have a mysterious tendency for accomplishing very hard complex tasks.  I guess it seems to me that trying to project my intelligence more would close off more opportunities than it would open just because of social factors.

Acting less smart than I am can sometimes be inconvenient or annoying especially if I'm arguing with someone who projects a "smart guy" vibe, and they're clearly wrong, but other people are impressed with their verbiage and confidence in their claims.  Usually I don't care that much about winning arguments, except occasionally when the outcome really matters in which case I can get very frustrated.

This may be obvious, but I'm not posting this out of vanity for being a high-intelligence individual.  Rather, it is a real issue that I have to deal with and I'm not completely sure I always get it right.  Sometimes it seems it would be better and more genuine to not have a projected self separate from my real self (whatever that is).  It would be nice to talk with people about my actual interests and say things the way I actually think about them.  But mostly I think that intelligence should be a tool rather than a goal in itself.  Also, the problems that come with high intelligence are far less than the benefits, so I should stop complaining and use the intelligence that I was lucky to get to accomplish something good.

I hear you that teenagers spending hours computing hundreds of analytic derivatives or writing a bunch of essays is a pretty sad time waste... But if incentives shifted so instead that time got spent perfecting a starcraft build order against the AI a hundred times or grinding for years to pull off carpetless star in SM64, this might be one of the few ways to make that time spent even more pointless... (And for most people the latter is no more fun than the former)

Video games also have potential legal advantages over IQ tests for companies. You could argue that "we only hire people good at video games to get people who fit our corporate culture of liking video games" but that argument doesn't work as well for IQ tests.

IANAL but unless you work for a videogame company(or a close analogue like chess.com) , I think this is just false. If your job is cognitively demanding, having IQ tests (or things like IQ tests with a mildly plausible veneer) probably won't get you in legal trouble^[1], whereas I think employment lawyers would have a field day if you install culture fit questions with extreme disparate impact, especially when it's hard to directly link the games to job performance. 

1. ^{^}

   The US Army has something like an IQ test.  So does the US Postal Service. So does the NFL. I've also personally worked in a fairly large tech company (not one of the top ones, before I moved to the Bay Area) that had ~IQ tests as one of the entrance criteria.  AFAIK there has never been any uproar about it. 

I have been thinking about things similar to this lately, for the more specific sub goal of ‘how would you train, and measure, ability to form plans that achieve complicated goals’ (in particular in domains with poor feedback loops)

I think this is excellent particularly because IQ tests often max out quickly on skills that can't be examined quickly. It would he great to put people in tests that examine their longer timeframe abilities via eg. writing a longish story (perhaps containing a theory of Alzheimer's). But tests don't last that long.

Games however do last long and do manage to keep people's attention for a long time. So you might really be able to test how differentially skilled someone is over longer timeframes.

(this is completely sideways, but recently I found myself thinking that other exams should not be testing intelligence instead of their stated purpose. When an English test requires you to hold in your head facts, ..., it starts to be less about English and more about "something else".)

Have you read about dath ilan? AFAIK, the best "core" post is this AMA [LW · GW] with Yud, and AFAIK this Yudfic [LW · GW] does the most worldbuilding (if anyone has better intros, please suggest).

I find it highly believable that in a world as optimized and moloch-opposed as dath ilan, intelligence tests AND intelligence/rationality amplification exersizes probably use the video game principles you've described here; it's often just outright better to structure user interfaces like that.

I don't know how long it will take for the current industry to pump out something that can work this well though; the current equilibria incentivizes developers to send players into a trance-like "zombie mode" similar to social media, often even including the same skinner box dynamic [LW · GW] that ultimately results in sifting through garbage 95% of the time, and only getting what they came for 5% of the time. If you make a really well-worded argument that a video game could fix/improve large numbers of people, the way that TOTK inspires people to become engineers, or that The Sequences [? · GW], CFAR Handbook [LW · GW], and Cognitive Strategy Tuning [LW · GW] can increase intelligence directly, then you might be able to convince a critical mass of people in the industry to pump out something good in a timely fashion.