I feel like even under the worldview that your beliefs imply, a superintelligence will just make a brain the size of a factory, and then be in a position to outcompete or destroy humanity quite easily.

I'm going to expand on this.

Jacob's conclusion to the speed section of his post on brain efficiency [LW(p) · GW(p)] is this:

The brain is a million times slower than digital computers, but its slow speed is probably efficient for its given energy budget, as it allows for a full utilization of an enormous memory capacity and memory bandwidth. As a consequence of being very slow, brains are enormously circuit cycle efficient. Thus even some hypothetical superintelligence, running on non-exotic hardware, will not be able to think much faster than an artificial brain running on equivalent hardware at the same clock rate.

Let's accept all Jacob's analysis about the tradeoffs of clock speed, memory capacity and bandwidth.

The force of his conclusion depends on the superintelligence "running on equivalent hardware." Obviously, core to Eliezer's superintelligence argument, and habryka's comment here, is the point that the hardware underpinning AI can be made large and expanded upon in a way that is not possible for human brains.

Jacob knows this, and addresses it in comments in response to Vaniver pointing out that birds may be more efficient than jet planes in terms of calories/mile flown, but that when the relevant metric is top speed or human passengers carried, the jet wins. Jacob responds:

I agree electricity is cheap, and discuss that. But electricity is not free, and still becomes a constraint...

The rental price using enterprise GPUs is at least 4x as much, so more like $20,000/yr per agent. So the potential economic advantage is not yet multiple OOM. It's actually more like little to no advantage for low-end robotic labor, or perhaps 1 OOM advantage for programmers/researchers/ec. But if we had AGI today GPU prices would just skyrocket to arbitrage that advantage, at least until foundries could ramp up GPU production.

So the crux here appears to be about the practicality of replacing human brains with factory-sized artificial ones, in terms of physical resource limitations.

Daniel Kokotajlo disagrees that this is important:

$2,000/yr per agent is nothing, when we are talking about hypothetical AGI. This seems to be evidence against your claim that energy is a taut constraint.

Jacob doubles down that it is:

Energy is always an engineering constraint: it's a primary constraint on Moore's Law, and thus also a primary limiter on a fast takeoff with GPUs (because world power supply isn't enough to support net ANN compute much larger than current brain population net compute).

But again I already indicated it's probably not a 'taut constraint' on early AGI in terms of economic cost - at least in my model of likely requirements for early not-smarter-than-human AGI.

Also yes additionally longer term we can expect energy to become a larger fraction of economic cost - through some combination of more efficient chip production, or just the slowing of moore's law itself (which implies chips holding value for much longer, thus reducing the dominant hardware depreciation component of rental costs)

So Jacob here admits that energy is neither a 'taut constraint' for early AGI, and that at the same time it will be a larger fraction of the cost. In other words, it's not a bottleneck for AGI, and no other resource is either.

This is where Jacob's discussion ended.

So I think Jacob has at least two jobs to do to convince me. I would be very pleased and appreciative if he achieved just one of them.

First, he needs to explain why any efficiency constraints can't be overcome by just throwing a lot of material and energy resources into building and powering inefficient or as-efficient-as-human-brains GPUs. If energy is not a taut constraint for AGI, and it's also expected to be an increasing fraction of costs over time, then that sounds like an argument that we can overcome any efficiency limits with increasing expenditures to achieve superhuman performance.

Second, he needs to explain why things like energy, size, or ops/sec efficiency are the most important efficiency metrics as opposed to things like "physical tasks/second," or "brain-size intelligences produced per year," or "speed at which information can be taken in and processed via sensors positioned around the globe." There are so very many efficiency ("useful output/resource input") metrics that we can construct, and on many of them, the human brain and body are demonstrably nowhere near the physical limit.

Right now, doubling down on physics-based efficiency arguments, as he's doing here, don't feel like a winning strategy to me.

I feel like even under the worldview that your beliefs imply, a superintelligence will just make a brain the size of a factory, and then be in a position to outcompete or destroy humanity quite easily.

I am genuinely curious and confused as to what exactly you concretely imagine this supposed 'superintelligence' to be, such that is not already the size of a factory, such that you mention "size of a factory" as if that is something actually worth mentioning - at all. Please show at least your first pass fermi estimates for the compute requirements. By that I mean - what are the compute requirements for the initial SI - and then the later presumably more powerful 'factory'?

Maybe it will do that using GPUs, or maybe it will do that using some more neuromorphic design, but I really don't understand why energy density matters very much.

I would suggest reading more about advanced GPU/accelerator design, and then about datacenter design and the thermodynamic/cooling considerations therein.

The vast majority of energy that current humans produce is of course not spent on running human brains, and there are easily 10-30 OOMs of improvement lying around without going into density (just using the energy output of a single power plant under your model would produce something that would likely be easily capable of disempowering humanity).

This is so wildly ridiculous that you really need to show your work. I have already shown some calculations in these threads [LW · GW], but I'll quickly review here.

A quick google search indicates 1GW is a typical power plant output, which in theory could power roughly a million GPU datacenter. This is almost 100 times larger in power consumption than the current largest official supercomputer: Frontier - which has about 30k GPUs. The supercomputer used to train GPT4 is somewhat of a secret, but estimated to be about that size. So at 50x to 100x you are talking about scaling up to something approaching a hypothetical GPT-5 scale cluster.

Nvidia currently produces less than 100k high end enterprise GPUs per year in total, so you can't even produce this datacenter unless Nvidia grows by about 10x and TSMC grows by perhaps 2x.

The datacenter would likely cost over a hundred billion dollars, and the resulting models would be proportionally more expensive to run, such that it's unclear whether this would be a win (at least using current tech). Sure I do think there is some room for software improvement.

But no, I do not think that this hypothetical not currently achievable GPT5 - even if you were running 100k instances of it - would "likely be easily capable of disempowering humanity".

Of course if we talk longer term, the brain is obviously evidence that one human-brain power can be achieved in about 10 watts, so the 1GW power plant could support a population of 100 million uploads or neuromorphic AGIs. That's very much part of my model (and hansons, and moravecs) - eventually.

Remember this post is all about critiquing EY's specific doom model which involves fast foom on current hardware through recursive self-improvement.

Having more room at the bottom is just one of a long list of ways to end up with AIs much smarter than humans. Maybe you have rebuttals to all the other ways AIs could end up much smarter than humans

If you have read much of my writings, you should know that I believe its obvious we will end up with AIs much smarter than humans - but mainly because they will run faster [LW · GW] using much more power. In fact this prediction has already come to pass in a limited sense - GPT4 was probably trained on over 100 human lifetimes worth of virtual time/data using only about 3 months of physical time, which represents a 10000x time dilation (but thankfully only for training, not for inference).

I don't think he's at all claiming safety is trivial or that humans can expect to remain in charge. control-capture foom is very much permitted by his model and he says so directly; much bigger minds are allowed. But his model suggests that reflective algorithmic improvement is not the panacea that yudkowsky expected, nor that beating biology head to head is easy even for a very superintelligent system.

this does not change any claim I would make about safety; it should barely be an update for anyone who has already updated off of deep learning. but it should knock down yudkowsky's view of capability scaling in algorithms thoroughly. this is relevant to prediction of which kinds of system are a threat to other systems and how.

I feel like even under the worldview that your beliefs imply, a superintelligence will just make a brain the size of a factory, and then be in a position to outcompete or destroy humanity quite easily.

Presumably it takes a gigantic amount of compute to train a "brain the size of a factory"? If we assume that training a human-level AI will take 10^28 FLOP (which is quite optimistic), the Chinchilla scaling laws predict that training a model 10,000 times larger would take about 10^36 FLOP, which is far more than the total amount of compute available to humans cumulatively over our history.

By the time the world is training factory-sized brains, I expect human labor to already have been made obsolete by previous generations of AIs that were smarter than us, but not vastly so. Presumably this is Jacob's model of the future too?

Biology is incredibly efficient at certain things that happen at the cell level. To me, it seems like OP is extrapolating this observation rather too broadly. Human brains are quite inefficient at things they haven't faced selective pressure to be good at, like matrix multiplication.

Claiming that human brains are near Pareto-optimal efficiency for general intelligence seems like a huge stretch to me. Even assuming that's true, I'm much more worried about absolute levels of general intelligence rather than intelligence per Watt. Conventional nuclear bombs are dangerous even though they aren't anywhere near the efficiency of a theoretical antimatter bomb. AI "brains" need not be constrained by the size and energy constraints of a human brain.

Your instinct is right. The Landauer limit says that it takes at least $k_{b}T\ln 2$ energy to erase 1 bit of information, which is necessary to run a function which outputs 1 bit (to erase the output bit). The important thing to note is that it scales with temperature $T$ (measured in an absolute scale). Human brains operate at 310 Kelvin. Ordinary chips can already operate down to around ~230 Kelvin, and there is even a recently developed chip which operates at ~0.02 Kelvin.

So human brains being near the thermodynamic limit in this case means very little about what sort of efficiencies are possible in practice.

Your point about skull-sizes [being bounded by childbirth death risk] seems very strong for evolutionary reasons, and to which I would also add the fact that bird brains seem to do similar amounts of cognition (to smallish mammals) in a much more compact volume without having substantially higher body temperatures (~315 Kelvin).

I don't think heat dissipation is actually a limiting factor for humans as things stand right now. Looking at the heat dissipation capabilities of a human brain from three perspectives (maximum possible heat dissipation by sweat glands across the whole body, maximum actual amount of sustained power output by a human in practice, maximum heat transfer from the brain to arterial blood with current-human levels of arterial bloodflow), none of them look to me to be close to the 20w the human brain consumes.

• Based on sweat production of athletic people reaching 2L per hour, that gives an estimate of ~1kW of sustained cooling capacity for an entire human
• 5 watts per kg seems to be pretty close to the maximum power output well-trained humans can actually output in practice for a full hour, so that suggests that a 70 kg human has at least 350 watts of sustained cooling capacity (and probably more, because the limiting factor does not seem to be overheating).
• Bloodflow to the brain is about 45L / h, and brains tolerate temperature ranges of 3-4ºC, so working backwards from that we get that a 160W brain would reach temperatures of about 3ºC higher than arterial blood assuming that arterial bloodflow was the primary heat remover. Probably add in 20-100 watts to account for sweat dissipation on the head. And also the carotid artery is less than a cm in diameter, so bloodflow to the brain could probably be substantially increased if there were evolutionary pressure in that direction.

Brains in practice produce about 20W of heat, so it seems likely to me that energy consumption could probably increase by at least one order of magnitude without causing the brain to cook itself, if there was strong enough selection pressure to use that much energy (probably not two orders of magnitude though).

Getting rid of the energy constraint would help though. Proof of concept: ten humans take more energy to run than one human does, and can do more thinking than one human.

I do also find it quite likely that skull size is probably the most tightly binding constraint for humans -- we have smaller and very differently tuned neurons compared to other mammals, and I expect that the drive for smaller neurons in particular is downstream of space being very much at a premium, even more so than energy.

Further evidence for the "space, rather than energy expenditure or cooling, is the main binding constraint" hypothesis is the existence of Fontanelles -- human brains continue to grow after birth and the skull is not entirely solid in order to allow for that -- a skull that does not fully protect the brain seems like a very expensive adaptation, so it's probably buying something quite valuable.

efficient for the temperature it runs at. Jake is correct about the fundamental comparison, but he's leaving off the part where he expects reversible computing to fundamentally change the efficiency tradeoffs for intelligence eventually, which is essentially "the best way to make use of near perfect cooling" as a research field; I don't have a link to where he's said this before, since I'm remembering conversations we had out loud.

Near pareto-optimal in terms of thermodynamic efficiency as replicators and nanobots, see the discussions and links here [LW(p) · GW(p)] and here [LW(p) · GW(p)].

Jensen Huang/Nvidia is almost un-arguably one of TSMC's most important clients and probably has some insights/access to their roadmaps, and I don't particularly suspect he is lying when he claims Moore's Law is dead, it matches my own analysis of TSMC's public roadmap, as well my analysis of the industry research/chatter/gossip/analysis. Moore's Law was a long recursive miniaturization optimization process which just was always naturally destined to bottom out somewhat before new on-moore's law leading foundries cost sizable fractions of world GDP and features approach minimal sizes (well predicted in advance).

This obviously isn't the end of technological progress in computing! It's just the end of the easy era. Neuromorphic computing is much harder for comparatively small gains. Reversible computing seems almost impossibly difficult, such that many envision just jumping straight to quantum computing, which itself is no panacea and very far.

And this 2022 analysis suggests things were also going quite strong very recently,

As were chip clock frequencies under dennard scaling, until that suddenly ended. I have uncertainty over how far we are from minimal viable switch energies but it is not multiple OOM. There are more architectural tricks in the pipes in the nature of lower precision tensorcores, but not many of those left either.

Also for forecasting AI dynamics, flops/$ seems like it matters a lot more, since in the near future AI seems unlikely to have to care much about transistor density, given that there are easily 10-20 OOMs of energy and materials to be used on earth's surface for some kind of semiconductor or neuromorphic compute production.

Sure but so far the many OOM improvement in flops/$ has been driven by shrinkage, not using ever more of earth's resources to produce fabs & chips. Growing that way is very slow and absolutely in line with the smooth slow non foomy takeoff scenarios.

EY's belief distribution about NNs and early DL from over a decade ago and how that reflects on his predictive track record has already been extensively litigated in other recent threads like here [LW · GW]. I mostly agree that EY 2008 and later is somewhat cautious/circumspect about making explicitly future-disprovable predictions, but he surely did seem to exude skepticism which complements my interpretation of his actions.

That being said I also largely agree that MIRI's research path was chosen specifically to try and be more generic than any viable route to AGI. But one could consider that also as something of a failure or missed opportunity vs investing more in studying neural networks, the neuroscience of human alignment, etc.

But I've always said (perhaps not in public, but nonetheless) that I thought MIRI had a very small chance of success, but it was still a reasonable bet for at least one team to make, just in case the connectivists were all wrong about this DL thing.

One thing that I have observed, working with LLMs, is that when they're predicting the next token in a Python REPL they also make kinda similar mistakes to the ones that a human who wasn't paying that much attention would make. For example, consider the following

>>> a, b = 3, 5    # input
>>> a + b          # input
8                  # input
>>> a, b = b, a    # input
>>> a * b          # input
15                 # prediction (text-davinci-003, temperature=0, correct)
>>> a / b          # input
1.0                # prediction (text-davinci-003, temperature=0, incorrect but understandable mistake)
>>> a              # input
5                  # prediction (text-davinci-003, temperature=0, correct)
>>> a / b          # input
1.0                # prediction (text-davinci-003, temperature=0, incorrect but understandable mistake)
>>> a              # input
5                  # prediction (text-davinci-003, temperature=0, correct)
>>> a / b          # input
1.0                # prediction (text-davinci-003, temperature=0, incorrect but understandable mistake)
>>> b              # input
3                  # prediction (text-davinci-003, temperature=0, correct)
>>> a / b          # input
1.6666666666666667 # prediction (text-davinci-003, temperature=0, now correct -- looks like a humanish "oh whoops lol")

I expect that most examples in the training data of "this looks like an interaction with the Python REPL" were in fact interactions with the Python REPL. To the extent that GPT-N models do make human-like mistakes when predicting non-human-like data (instead of predicting that non-human-like data correctly, which is what their loss function wants them to do), I think that does serve as nonzero evidence that their cognition is "human-like" in the specific narrow sense of "mirroring our quirks and limitations".

More generally, I think it's particularly informative to look at the cases where the thing GPT-n does is different than the thing its loss function wants it to do. The extent to which those particular cases look like human failure modes is informative to how "human-like" the cognition of GPT-n class models (as a note, the SolidGoldMagikarp [LW · GW] class of failure modes is extremely non human-like, and so that observation caused me to update more in the "shoggoth which can exhibit human behavior among many other behaviors" view. But I haven't actually seen a whole lot of failure modes like that in normal use, and I have seen a bunch of human-not-paying-attention type failure modes in normal use).

>>> import chess
>>> board = chess.Board()
>>> board.push_san('Na3')
Move.from_uci('b1a3')
>>> print(board.piece_at(chess.parse_square('b1')))

In what sense is predicting internet text "training [LLMs] on human thoughts"?

Internet text contains the inputs and outputs of human minds in the sense that every story, post, article, essay, book, etc written by humans first went through our brains, word by word, token by token, tracing through our minds.

Training on internet text is literally training on human thoughts because text written by humans is literally an encoding of human thoughts. The fact that it is an incomplete and partial encoding is mostly irrelevant, as given enough data you can infer through any such gaps. Only a small fraction of the pixels in an image are sufficient to reconstruct it.

I am reasonably sympathetic to this argument, and I agree that the difference between EY's p(doom) > 50% and my p(doom) of perhaps 5% to 10% doesn't obviously cash out into major policy differences.

I of course fully agree with EY/bostrom/others that AI is the dominant risk, we should be appropriately cautious, etc. This is more about why I find EY's specific classic doom argument to be uncompelling.

My own doom scenario is somewhat different and more subtle, but mostly beyond scope of this (fairly quick) summary essay.

I see Eliezer as making arguments about the worst ways things could go wrong and why it’s not guaranteed that they won’t go that way.

Eliezer believes and argues that things go wrong by default, with no way he sees to avoid that. Not just "no guarantee they won't go wrong".

It may be that his arguments are sufficient to convince you of "no guarantee they won't go wrong" but not to convince you of "they go wrong by default, no apparent way to avoid that". But that's not what he's arguing.

I had the same impression at first, but in the areas where I most wanted these, I realized that Jacob linked to additional posts where he has defended specific claims at length.

Here is one example:

EY is just completely out of his depth [LW · GW] here: he doesn't seem to understand how the Landauer limit actually works, doesn't seem to understand that synapses are analog MACs which minimally require OOMs more energy than simple binary switches, doesn't seem to understand that interconnect dominates energy usage regardless, etc.

I usually find Tyler Cowenesque (and heck, Yudkowskian) phrases like this irritating, and usually they're pretty hard to interrogate, but Jacob helpfully links to an entire factpost he wrote on this specific point, elaborating on this claim in detail.
 

He does something similar here:

EY derived [LW(p) · GW(p)] much of his negative beliefs about the human mind from the cognitive biases and ev psych literature, and especially Tooby and Cosmide's influential evolved modularity hypothesis. The primary competitor to evolved modularity was/is the universal learning hypothesis and associated scaling hypothesis, and there was already sufficient evidence to rule out evolved modularity back in 2015 [LW · GW] or earlier.

He is doing an admirable job of summarizing the core arguments of Eliezer's overall model of AI risk, then providing deeply-thought-out counterarguments. The underlying posts appear to be pretty well-cited.

That doesn't mean that every claim is backed by an additional entire post of its own, or that every argument is convincingly correct.

for various reasons mostly stemming from the strong prior that biology is super effecient I expect humanity to be very difficult to kill in this way (and growing harder to kill every year as we advance prosaic AI tech)

This is more of the Cowenesque/Yudkowskianesque handwavey appeal to authority, intuition, and the implication that any informed listener ought to have mastered the same background knowledge and come naturally to the same conclusions. Here, my response would be "viruses are not optimizing to kill humans - they are optimizing for replication, which often means keeping hosts alive."

He follows this with another easy-to-argue-with claim:

killing humanity would likely not be in the best interests of even unaligned AGI, because humans will probably continue to be key components of the economy (as highly efficient general purpose robots) long after AGI running in datacenters takes most higher paying intellectual jobs

This sounds at best like an S-risk (how would the AI cause humans to behave like highly efficient general purpose robots? Probably not in ways we would enjoy from our perspective now). And we don't need to posit that an unaligned AI would be bent on keeping any semblance of an "economy" running. ChaosGPT 5.0 might specifically have the goal of destroying the entire world as efficiently as possible.

But my goal in articulating these counterarguments isn't because I'm hoping to embroil myself in a debate downthread. It's to point out that while Jacob's still very far from dealing with every facet of Eliezer's argument, he is at the same time doing what I regard as a pretty admirable job of interrogating certain specific claims in depth.

And Jacob doesn't read to me as making what I would call "personal attacks on Eliezer." He does do things like:

Accurately paraphrase arrogant-sounding things Eliezer says

Moreover he predicts that this is the default outcome, and AI alignment is so incredibly difficult that even he failed to solve it.

Accurately describe the confidence with which Eliezer makes his arguments ("brazenly").

Be very clear and honest about just how weak he finds Eliezer's argument, after putting in a very substantial amount of work in order to come to this evaluation

I have evaluated this model in detail and found it substantially incorrect and in fact brazenly naively overconfident.

Every one of his key assumptions is mostly wrong, as I and others predicted well in advance. EY seems to have been systematically overconfident [EA · GW] as an early futurist, and then perhaps updated later to avoid specific predictions, but without much updating his mental models (specifically his nanotech-woo model, as we will see).

Offer general reasons to think the epistemics are stronger on his side of the debate:

Even though the central prediction of the doom model is necessarily un-observable for anthropic reasons, alternative models (such as my own, or moravec's, or hanson's) have already made substantially better predictions, such that EY's doom model has low posterior probability.

EY has espoused this doom model for over a decade, and hasn't updated it much from what I can tell.

Note: this is one area I do think Jacob's argument is weak - I would need to see one or more specific predictions from Jacob or the others he cites from some time ago in order to evaluate this claim. But I trust it is true in his own mind and could be made legible.

Overall, I think it is straightforwaredly false to say that this post contains any "personal attacks on Eliezer."

Edit: However, I also would say that Jacob is every bit as brazen and (over?)confident as Eliezer, and has earned the same level of vociferous pushback where his arguments are weak as he has demonstrated in his arguments here.

And I want to emphasizes that in my opinion, there's nothing wrong with a full-throated debate on an important issue!

If the optimal AGI design running on GPUs takes about 10 GPUs and 10kw to rival one human-brain power, and superintelligence which kills humanity ala the foom model requires 10 billion human brain power and thus 100 billion GPUs and a 100 terrawatt power plant - that is just not something that is possible in any near term.

In EY's model there is supposedly 6 OOM improvement from nanotech, so you could get the 10 billion human brainpower with a much more feasible 100 MW power plant and 100 thousand GPUs ish (equivalent).

To put some numbers on that, USA brains directly consume 20W [LW · GW] × 330M = 6.6 GW, whereas the USA food system consumes ≈500 GW [not counting sunlight falling on crops] (≈15% of the 3300 GW total USA energy consumption).

Actually I think the shoggoth mask framing is somewhat correct, but it also applies to humans. We don't have a single fixed personality, we are also mask-wearers.

The argument that shifts me the most away from thinking of it with the shoggoth-mask analogy is the implication that a mask has a single coherent actor behind it. But if you can avoid that mental failure mode I think the shoggoth-mask analogy is basically correct.

The brain's slow speed seems mostly for energy efficiency [LW(p) · GW(p)] but it is also closely tuned to brain size such that signal delay is not a significant problem.

Agreed in advance [LW · GW].

Bird brains have higher neuron density esp in the forebrain, but I'm unsure if this also translates into higher synaptic density or just less synapses per neuron. Regardless it does look like bird brains are optimized more heavily for compactness, but its not clear what tradeoffs may being made there. But heat transport cooling scales with the surface area whereas compute and thus heat production scales with volume, so brains tend to become less dense as they grow larger absent more heroic cooling efforts.

It's more fleshed out in the 2015 ULM [LW · GW] post.

I don't understand how OpenAIs success at scaling GPT proves the universal learning model.

If I have a model which predicts "this simple architecture scales up to human intelligence with enough compute", and that is tested and indeed shown to be correct, then the model is validated.

And it helps further rule out an entire space of theories about intelligence: namely all the theories that intelligence is very complicated and requires many complex interacting innate algorithms (evolved modularity, which EY seemed to subscribe to )

Couldn't there be an as yet undiscovered algorithm for intelligence that is more efficient?

Sure there could be other algorithms for intelligence that are more efficient, and I already said I don't think we are on quite on the final scaling curve with transfomers. But over time the probability mass remaining for these undiscovered algorithms continually diminishes as we explore ever more of the algorithmic search space.

Furthemore, evolution extensively explored the search space for architectures/algorithms for intelligent agents, and essentially found common variants of universal learning on NNs in multiple unrelated lineages, substantially adding to the evidence that yes this really is as good as it gets (at least for any near term conventional computers).