Many of the points you make are technically correct but aren't binding constraints. As an example, diffusion is slow over small distances but biology tends to work on µm scales where it is more than fast enough and gives quite high power densities. Tiny fractal-like microstructure is nature's secret weapon.

The points about delay (synapse delay and conduction velocity) are valid though phrasing everything in terms of diffusion speed is not ideal. In the long run, 3d silicon+ devices should beat the brain on processing latency and possibly on energy efficiency [LW · GW]

Still, pointing at diffusion as the underlying problem seems a little odd.

You're ignoring things like:

• ability to separate training and running of a model
  • spending much more on training to improve model efficiency is worthwhile since training costs are shared across all running instances
• ability to train in parallel using a lot of compute
  • current models are fully trained in <0.5 years
• ability to keep going past current human tradeoffs and do rapid iteration
  • Human brain development operates on evolutionary time scales
  • increasing human brain size by 10x won't happen anytime soon but can be done for AI models.

People like Hinton Typically point to those as advantages and that's mostly down to the nature of digital models as copy-able data, not anything related to diffusion.

Energy processing

Lungs are support equipment. Their size isn't that interesting. Normal computers, once you get off chip, have large structures for heat dissipation. Data centers can spend quite a lot of energy/equipment-mass getting rid of heat.

Highest biological power to weight ratio is bird muscle which produces around 1 w/cm³ (mechanical power). Mitochondria in this tissue produces more than 3w/cm³ of chemical ATP power. Brain power density is a lot lower. A typical human brain is 80 watts/1200cm³ = 0.067W/cm³.

synapse delay

This is a legitimate concern. Biology had to make some tradeoffs here. There are a lot of places where direct mechanical connections would be great but biology uses diffusing chemicals.

Electrical synapses exist and have negligible delay. though they are much less flexible (can't do inhibitory connections && signals can pass both ways through connection)

conduction velocity

Slow diffiusion speed of charge carriers is a valid point and is related to the 10^8 factor difference in electrical conductivity between neuron saltwater and copper. Conduction speed is an electrical problem. There's a 300x difference in conduction speed between myelinated(300m/s) and un-myelinated neurons(1m/s).

compensating disadvantages to current digital logic

The brain runs at 100-1000 Hz vs 1GHz for computers (10^6 - 10^7 x slower). It would seem at first glance that digital logic is much better.

The brain has the advantage of being 3D compared to 2D chips which means less need to move data long distances. Modern deep learning systems need to move all their synapse-weight-like data from memory into the chip during each inference cycle. You can do better by running a model across a lot of chips but this is expensive and may be inneficient [LW · GW].

In the long run, silicon (or something else) will beat brains in speed and perhaps a little in energy efficiency. If this fellow is right about lower loss interconnects [LW · GW] then you get another + 3OOM in energy efficiency.

But again, that's not what's making current models work. It's their nature as copy-able digital data that matters much more.

So I think in the long run, the only way biological brains win is if we simply do not build AGI.

Depends on how long you're talking about.  It seems plausible to me that, if we got a bunch of 250 IQ humans, then they could in fact do a major redesign of neurons.  However, I would expect all this to take at least 100 years (if not aided by superintelligent AI), which is longer than most AI timelines I've seen (unless we bring AI development to a snail's pace or a complete stop).

I agree with most of this post, but it doesn’t seem to address the possibility of whole brain emulation. However, many/(?most) would argue this is unlikely to play a major role because AGI will come first.

Although the crux of your claim that diffusion is the rate-limiting step of many biological processes may be sound, the question you actually ask, "Why hasn't evolution stumbled across a better method of doing things than passive diffusion?", is misguided.  Evolution has stumbled across such methods. Your post itself contains several examples of evolved systems which move energy and information faster than diffusion.  These include the respiratory system, which moves air into and out of the lungs much faster than diffusion would allow; the circulatory system, which moves oxygen and glucose (energy), hormones (information), and many other things, all much faster than diffusion; and neurons, which move impulses down the axon much faster than diffusion.  In none of these cases has the role of diffusion been completely removed, which is what you are looking for I suppose, but life has increased the total speed from point A to point B by several orders of magnitude by finding an alternate mechanism for most of the distance.

Beyond that type of optimization, I agree with you that this is a local maximum problem. Life's fundamental processes are based on chemical reactions in aqueous solution, which necessarily involve diffusion, and any move beyond that is a big, low-probability step.

That said, particularly in the case of the brain, I don't think evolution has thoroughly explored its possibility space, and there is room for further optimization within the current "paradigm".  As one example, the evolution of human intelligence proceeded at least in large part by scaling brain size, but we are currently limited by the constraint that the head needs to be able to fit through the mother's pelvis during birth. Birds, based on totally different evolutionary pressures involving weight reduction for flight, seem to be able to pack processing power into a much smaller volume than mammals. If humans could "only" use bird brain architecture, then we could presumably increase our intelligence substantially without increasing the size of our heads.  But despite the presumed evolutionary pressure (we suffer much higher maternal mortality than other species due to our big heads) there has not been enough time for us to convergently evolve brain miniaturization to the extent of birds.

There was a story in NewAtlas about a computer - brain tissue merge that seems to have produced some interesting features. Perhaps such a union is the type of evolutionary next step for human cognitive improvements rather than genetic screening/engineering or chemical additives.

Agreed. See also https://denovo.substack.com/p/biological-doom for an overview of different types of biological computation.

Main point: yes neurons are not the best building block for communication speed, but you shouldn’t assume that increasing it would necessarily increase fitness. Muscles are much slower, retina even more, and even the fastest punch (mantris shrimps) is several time slower than communication speed in myelinated axons. That’s said, the overall conclusion that using neurons is an evolutionary trap is probably sound, as we know most of our genetic code is for tweaking something in the brain, without much change in how most neurons work and learn.

Tangent point: you’re assuming that 250IQ is a sound concept. If you were to pass an IQ test well designed for mouses, would you expect to reach 250? If you were to measure IQ in slim molt, what kind of result would you expect, and would that IQ level help predict the behavior below?

https://www.discovermagazine.com/planet-earth/brainless-slime-mold-builds-a-replica-tokyo-subway

"We can probably raise IQ into the low or mid 200s with genetic engineering."

So you don't think 300 to 900 IQ humans are possible? You think 250 is the max of a human?

Speed of thinking isn't the only component of g-factor, it's also memory, the size of the recall-able knowledge space, and how many variables you could juggle in your head at once which imo could keep on getting larger if brains and bodies could also keep getting bigger. Do non-human brains think at a slower rate than bonobos and chimps? Young non-human primates and other animals have pretty fast reaction times. The reason they have lower IQs than humans can be explained entirely by neuron count. Human intelligence is just primate brains scaled up with more neurons. Gorillas and orangutans have brain sizes 1/3 as big as humans. Human brains have all the speed constraints as gorillas obviously. Knowledge and working memory is more important than speed.  A 500 IQ human could outsource strict numerical calculation to silicon calculators

A minor point, perhaps a nitpick: both biological systems and electronic ones depend on directed diffusion. In our bodies diffusion is often directed by chemical potentials, and in electronics it is directed by electric or vector potentials. It's the strength of the 'direction' versus the strength of the diffusion that makes the difference. (See: https://en.m.wikipedia.org/wiki/Diffusion_current)

Except in superconductors, of course.

Superintelligence is like self-replicating diamondoid machines as human industry replacement: both are plausible predictions, but not cruxes of AI risk. Serial speed advantage [LW(p) · GW(p)] alone is more than sufficient. Additionally, it by itself predicts the timeframe of only a few physical years from first AGIs to ancient AI civilizations/individuals (as hardware that removes contingent weaknesses of modern GPUs is manufactured).

Human intelligence enhancement is relevant as a path to figuring out alignment or AGI building ban coordination. With alignment, even biological brains don't have to be competitive.

There is also the problem of civilizational value drift [LW(p) · GW(p)], in the long run even a human civilization might go off the rails, and it's unclear that this problem gets addressed in time without intelligence enhancement, in a world without AGIs. The same applies to WBEs/ems if they magically appear before synthetic AGIs, except they also have the serial speed advantage. So if they remain at human level and don't work on value drift, their initial alignment with humanity isn't going to help, as in a few physical years they become an ancient civilization that has drifted uncontrollably far away from their origins (in directions not guided by moral arguments). Hanson's book explores how this process could start.

While I agree that easier speed upgrades to hardware is one advantage we should expect a digital mind to have, I wouldn't call that the primary advantage. Anithite mentions some others in their comment. I think increasing hardware size is a pretty big one. The biggest I think though is substrate independence. This unlocks many others. Easy hardware upgrades, rapid copying, travel at the speed of light between different substrate locations, easy storage of backup copies, ability to easily store oneself in a compressed form and then expand again, to add and modify parts of yourself. And many more. The beings best able to colonize the galaxy are obviously going to be ones with these sorts of advantages. Being able to send yourself in a hundred spaceships, each the size of a pea, with a compressed dormant copy of you, pushed through the interstellar spaces on laser beams operated by a copy of yourself remaining behind... No biological entity can compete with that kind of space travel, unless we are very mistaken about the laws of physics.