Why is neuron count of human brain relevant to AI timelines?

This is a great question!

Point one:

The computational capacity of the brain used to matter much more than it matters now.  The AIs we have now are near-human or superhuman at many skills, and we can measure how skill capacity varies with resources in the near-human range.  We can debate and extrapolate and argue with real data.

But we spent decades where the only intelligent system we had was the human brain, so it was the only anchor we had for timelines.  So even though it’s very hard to make good estimates from, we had to use it.

Point two:

Most information that gives rise to the human mind is learned, not evolved.

The information encoded by evolution is less than a hundred megabytes.  It’s limited by the size of the genome (1 gigabytes).  Moreover, we know that much of the genome is unimportant for mental development.  About 40% is parasitic (viruses and transposons).  Much of the remaining DNA is not under evolutionary control, varying randomly between individuals.  Of expressed genes, only about a quarter appear to be expressed in the brain.  And some of them encode things AI doesn’t need, like the high-reliability plumbing of the circle of Willis, or the mysteries of love, or the biochemical pickiness of the blood-brain barrier, or wanting to pee when you hear running water.  So the “program” contributed by evolution is no more than the size of a largish program like a compiler.  (I would claim it’s probably even less.  I think the important instincts plus the learning algorithms are only a few thousand lines of code.  But that’s debatable.)

On the other hand, the amount learned in a lifetime is on the order of one or a few gigabytes.

Point three:

Most of the information accumulated by evolution has been destroyed.  All of the information accumulated in a species is lost when that species goes extinct. And most species have gone extinct, leaving no descendants.  The world of the Permian period was (as far as we know) just as busy as today, with hundreds of thousands of animal species.  Just one of those species, a little burrowing critter among many other types of little burrowing critters, was the ancestor of all mammals.  All the other little burrowing critters lost out.  All their evolutionary innovations have been lost.

This doesn’t apply to species with horizontal transmission of genes, like bacteria.  But it applies to animals, who are the only creatures with brains.

In the strongest sense, neither the human brain analogy nor the evolution analogy really apply to AI. They only apply in a weaker sense where you are aware you're working with analogy, and should hopefully be tracking some more detailed model behind the scenes.

The best argument to consider human development a stronger analogy than evolutionary history is that present-day AIs work more like human brains than they do like evolution. See e.g. papers finding that you can use a linear function to translate some concepts between brain scans and internal layers in a LLM, or the extremely close correspondence between ConvNet feature and neurons in the visual cortex. In contrast, I predict it's extremely unlikely that you'll be able to find a nontrivial correspondence between the internals of AI and evolutionary history or the trajectory of ecosystems or similar.

Of course, just because they work more like human brains after training doesn't necessarily mean they learn similarly - and they don't learn similarly! In some ways AI's better (backpropagation is great, but it's basically impossible to implement in a brain), in other ways AI's worse (biological neurons are way smarter than artificial 'neurons'). Don't take the analogy too literally. But most of the human brain (the neocortex) already learns its 'weights' from experience over a human lifetime, in a way that's not all that different from self-supervised learning if you squint.