Let's stop making "Intelligence scale" graphs with humans and AI

Have you seen the classic parody article “On the Impossibility of Supersized Machines”?

I think it’s possible to convey something pedagogically useful via those kinds of graphs. They can be misinterpreted, but this is true of many diagrams in life. I do dislike the “you are here” dot that Tim Urban added recently.

Try speaking aloud for precisely 269 words. You're not allowed to count or recite poetry — you have to do this while actually extemporizing something interesting to say.

Now bear in mind that an LLM doesn't output letters or words, it uses tokens. In order to count words, an LLM has to memorize which tokens contain spaces and which don't. So for it the task is comparable to asking a human to speak until they've said 269 words that begin with the letter T.

The main problem with this post is that it assumes "AI" is a monolithic thing that can't include different systems in the future. LLMs can translate from English to German. AlphaZero can beat the best human players at Go. Different systems can do different things. ChatGPT can't evaluate the result of Javascript programs accurately, but if you set it up to run Node.js, it suddenly can.

That's true, but not specific to AI. Where do you place chimp on the scale ? Low under human ? Ok, now consider an athletic man really good at brachiation, and a random chimp. It's not just that the chimp has more strength or agility in arms. I'll bet that the chimp will be definitely best at cognitive task like determining a good path across the trees, which branch to avoid etc. The chimp would also likely surpass most humans in recognition of some sorts of plants, fruits etc. 

And what about savant autists like the guy who inspired Rain Man, or the twins John and Michael mentioned by Oliver Sacks in his book 'The Man Who Mistook His Wife for a Hat'? Where are we supposed to place them on the curve ?

I've also read that bees can count up to 4 or 5, which surpasses some mammals or human children before the age of 2 or 3. Where do the bees go on the curve ?

Intelligence is likely not something that can be plotted on a simple curve. This could actually be advantageous for AI safety. Foom might be avoided if misaligned AIs have uneven cognitive capabilities and occasionally make significant errors in judgment.

How does AI being good at some tasks and worse at others make the graph you posted not a good tool at explaining FOOM or increasing AI capabilities?

I find this graph useful. I think you can agree at some point that AI will be more intelligent than humans, even if AI intelligence is quite different and lacking in a few (fewer every year) areas. If this is the case then this graph is quite effective at conveying that this point may be happening soon. 

I've generally found it much harder over time to find "examples where LLMs fail in surprising ways". If you test o3 (released the day after that post!) for the examples they chose, it does much better than previous models. And I've just tried it on your "269 words" task, which it nailed.