Transformative AGI by 2043 is <1% likely

Thanks for this well-researched and thorough argument! I think I have a bunch of disagreements, but my main one is that it really doesn't seem like AGI will require 8-10 OOMs more inference compute than GPT-4. I am not at all convinced by your argument that it would require that much compute to accurately simulate the human brain. Maybe it would, but we aren't trying to accurately simulate a human brain, we are trying to learn circuitry that is just as capable.

Also: Could you, for posterity, list some capabilities that you are highly confident no AI system will have by 2030? Ideally capabilities that come prior to a point-of-no-return so it's not too late to act by the time we see those capabilities.

This is the multiple stages fallacy. Not only is each of the probabilities in your list too low, if you actually consider them as conditional probabilities they're double- and triple-counting the same uncertainties. And since they're all mulitplied together, and all err in the same direction, the error compounds.

Interesting that this essay gives both a 0.4% probability of transformative AI by 2043, and a 60% probability of transformative AI by 2043, for slightly different definitions of "transformative AI by 2043". One of these is higher than the highest probability given by anyone on the Open Phil panel (~45%) and the other is significantly lower than the lowest panel member probability (~10%). I guess that emphasizes the importance of being clear about what outcome we're predicting / what outcomes we care about trying to predict.

The 60% is for "We invent algorithms for transformative AGI", which I guess means that we have the tech that can be trained to do pretty much any job. And the 0.4% is the probability for the whole conjunction, which sounds like it's for pervasively implemented transformative AI: AI systems have been trained to do pretty much any job, and the infrastructure has been built (chips, robots, power) for them to be doing all of those jobs at a fairly low cost. 

It's unclear why the 0.4% number is the headline here. What's the question here, or the thing that we care about, such that this is the outcome that we're making forecasts for? e.g., I think that many paths to extinction don't route through this scenario. IIRC Eliezer has written that it's possible that AI could kill everyone before we have widespread self-driving cars. And other sorts of massive transformation don't depend on having all the infrastructure in place so that AIs/robots can be working as loggers, nurses, upholsterers, etc.

I disagree with the brain-based discussion of how much compute is required for AGI. Here’s an analogy I like (from here [LW · GW]):

Left: Suppose that I want to model a translator (specifically, a MOSFET). And suppose that my model only needs to be sufficient to emulate the calculations done by a CMOS integrated circuit. Then my model can be extremely simple—it can just treat the transistor as a cartoon switch. (image source.)

Right: Again suppose that I want to model a transistor. But this time, I want my model to accurately capture all measurable details of the transistor. Then my model needs to be mind-bogglingly complex, involving dozens of adjustable parameters, some of which are shown in this table (screenshot from here).

What’s my point? I’m suggesting an analogy between this transistor and a neuron with synapses, dendritic spikes, etc. The latter system is mind-bogglingly complex when you study it in detail—no doubt about it! But that doesn’t mean that the neuron’s essential algorithmic role is equally complicated. The latter might just amount to a little cartoon diagram with some ANDs and ORs and IF-THENs or whatever. Or maybe not, but we should at least keep that possibility in mind.

--

For example, this paper is what I consider a plausible algorithmic role of dendritic spikes and synapses in cortical pyramidal neurons, and the upshot is “it’s basically just some ANDs and ORs”. If that’s right, this little bit of brain algorithm could presumably be implemented with <<1 FLOP per spike-through-synapse. I think that’s a suggestive data point, even if (as I strongly suspect) dendritic spikes and synapses are meanwhile doing other operations too.

--

Anyway, I currently think that, based on the brain, human-speed AGI is probably possible in 1e14 FLOP/s. (This post [LW · GW] has a red-flag caveat on top, but that’s related to some issues in my discussion of memory, I stand by the compute section.) Not with current algorithms, I don’t think! But with some future algorithm.

I think that running microscopically-accurate brain simulations is many OOMs harder than running the algorithms that the brain is running. This is the same idea as the fact that running a microscopically-accurate simulation of a pocket calculator microcontroller chip, with all its thousands of transistors and capacitors and wires, stepping the simulation forward picosecond-by-picosecond, as the simulated chip multiplies two numbers, is many OOMs harder than multiplying two numbers.

Hi Ted, 

I will read the article, but there's some rather questionable assumptions here that I don't see how you could reach these conclusions while also considering them.

We invent algorithms for transformative AGI:

    - Have you considered RSI?  RSI in this context would be an algorithm that says "given a benchmark that measures objectively if an AI is transformative, propose a cognitive architecture for an AGI using a model with sufficient capabilities to make a reasonable guess".  You then train the AGI candidate from the cognitive architecture (most architectures will reuse pretrained components from prior attempts) and benchmark it.  You maintain a "league" of multiple top performing AGI candidates, and each one of them is in each cycle examining all the results, developing theories, and proposing the next candidate architecture.

   Because RSI uses criticality, it primarily depends on compute availability.  Assuming sufficient compute, it would discover a transformative AGI algorithm quickly.  Some iterations might take 1 month (training time for a llama scale model from noise), others would take less than 1 day, and many iterations would be attempted in parallel.

We invent a way for AGIs to learn faster than humans : Why is this even in the table?  This would be 1.0 because it's a known fact, AGI learns faster than humans.  Again, from the llama training run, the model went from knowing nothing to domain human level in 1 month.  That's faster.  (requiring far more data than humans isn't an issue)

AGI inference costs drop below $25/hr (per human equivalent): Well, A100s are 0.87 per hour.  A transformative AGI might use 32 A100s.  $27.84 an hour.  Looks like we're at 1.0 on this one also.  Note I didn't even bother with a method to use compute at inference time more efficiently. In short, use a low end, cheap model, and a second model that assesses, given the current input, how likely it is that the full scale model will produce a meaningfully different output.  

So for example, a robot doing a chore is using a low end, cheap model trained using supervised learning from the main model.  That model runs in real time in hardware inside the robot.  Whenever the input frame is poorly compressible from an onboard autoencoder trained on the supervised learning training set, the robot pauses what it's doing and queries the main model.  (some systems can get the latency low enough to not need to pause).  

This general approach would save main model compute, such that it could cost $240 an hour to run the main model and yet as long as it isn't used more than 10% of the time, and the local model is very small and cheap, it will hit your $25 an hour target.  

This is not cutting edge and there are papers published this week on this and related approaches.

We invent and scale cheap, quality robots: Due to recent layoffs in the sota robotics teams and how Boston Dynamics has been of low investor interest, yes this is a possible open

We massively scale production of chips and power: I will have to check the essay to see how you define "massively". 

We avoid derailment by human regulation: This is a parallel probability.  It's saying actually 1 - (probability that ALL regulatory agencies in power blocs capable of building AGI regulate it to extinction).  So it's 1 - (EU AND USA AND China AND <any other parties> block tAGI research).  The race dynamics mean that defection is rapid so the blocks fail rapidly, even under suspected defection.  If one party believes the others might be getting tAGI, they may start a national defense project to rush build their own, a project that is exempt from said regulation.

We avoid derailment by AI-caused delay: Another parallel probability.  It's the probability that an "AI chernobyl" happens and all parties respect the lesson of it.

We avoid derailment from wars (e.g., China invades Taiwan) : Another parallel probability, as Taiwan is not the only location on earth capable of manufacturing chips for AI.  

We avoid derailment from pandemics: So a continuous series of pandemics, not a 3 year dip to ~80% productivity?  Due to RSI even 80% productivity may be sufficient.

We avoid derailment from severe depressions: Another parallel probability.  

With that said : in 15 years I find the conclusion of "probably not" tAGI that can do almost everything better than humans, including robotics, and it's cheap per hour, and massive amounts of new infrastructure have been built, to probably be correct.  So many steps, and even if you're completely wrong due to the above or black swans, it doesn't mean a different step won't be rate limiting.  Worlds where we have AGI that is still 'transformative' by a plain meaning of the term, and the world is hugely different, and AGI is receiving immense financial investments and is the most valuable industry on earth, are entirely possible without it actually satisfying the end conditions this essay is aimed at.

Thanks, this was interesting.

I couldn't really follow along with my own probabilities because things started wild from the get-go. You say we need to "invent algorithms for transformative AI," when in fact we already have algorithms that are in-principle general, they're just orders of magnitude too inefficient, but we're making gradual algorithmic progress all the time. Checking the pdf, I remain confused about your picture of the world here. Do you think I'm drastically overstating the generality of current ML and the gradualness of algorithmic improvement, such that currently we are totally lacking the ability to build AGI, but after some future discovery (recognizable on its own merits and not some context-dependent "last straw") we will suddenly be able to?

And your second question is also weird! I don't really understand the epistemic state of the AI researchers in this hypothetical. They're supposed to have built something that's AGI, it just learns slower than humans. How did they get confidence in this fact? I think this question is well-posed enough that I could give a probability for it, except that I'm still confused about how to conditionalize on the first question.

The rest of the questions make plenty of sense, no complaints there.

In terms of the logical structure, I'd point out that inference costs staying low, producing chips, and producing lots of robots are all definitely things that could be routes to transformative AI, but they're not necessary. The big alternate path missing here is quality. An AI that generates high-quality designs or plans might not have a human equivalent, in which case "what's the equivalent cost at $25 per human hour" is a wrong question. Producing chips and producing robots could also happen or not happen in any combination and the world could still be transformed by high-quality AI decision-making.

I think the biggest problem with these estimates is that they rely on irrelevant comparisons to the human brain. 

What we care about is how much compute is needed to implement the high-level cognitive algorithms that run in the brain; not the amount of compute needed to simulate the low-level operations the brain carries out to perform that cognition. This is a much harder to quantity to estimate, but it's also the only thing that actually matters.

See Biology-Inspired AGI Timelines: The Trick That Never Works [LW · GW] and other extensive prior discussion [LW · GW] on this.

I think with enough algorithmic improvement, there's enough hardware lying around already to get to TAI, and once you factor this in, a bunch of other conditional events are actually unnecessary or much more likely. My own estimates:

 

Explanations:

• Inventing algorithms: this is mostly just a wild guess / gut sense, but it is upper-bounded in difficulty by the fact that evolution managed to discover algorithms for human level cognition through billions of years of pretty dumb and low bandwidth trial-and-error.
• "We invent a way for AGIs to learn faster than humans." This seems like it is mostly implied by the first conditional already and / or already satisfied: current AI systems can already be trained in less wall-clock time than it takes a human to develop during a lifetime, and much less wall clock time than it took evolution to discover the design for human brains. Even the largest AI systems are currently trained on much less data than the amount of raw sensory data that is available to a human over a lifetime. I see no reason to expect these trends to change as AI gets more advanced.
• "AGI inference costs drop below $25/hr (per human equivalent)" - Inference costs of running e.g. GPT-4 are already way, way less than the cost of humans on a per-token basis, it's just that the tokens are way, way lower quality than the best humans can produce. Conditioned on inventing the algorithms to get higher quality output, it seems like there's plausibly very little left to do to here.
• Inventing robots - doesn't seem necessary for TAI; we already have decent, if expensive, robotics and TAI can help develop more and make them cheaper / more scalable.
• Scaling production of chips and power - probably not necessary, see the point in the first part of this comment on relevant comparisons to the brain.
• Avoiding derailment - if existing hardware is sufficient to get to TAI, it's much easier to avoid derailment for any reason. You just need a few top capabilities research labs to continue to be able to push the frontier by experimenting with existing clusters.

Just to pick on the step that gets the lowest probability in your calculation, estimating that the human brain does 1e20 FLOP/s with only 20 W of power consumption requires believing that the brain is basically operating at the bitwise Landauer limit, which is around 3e20 bit erasures per watt per second at room temperature. If the FLOP we're talking about here is equivalent of operations on 8-bit floating point numbers, for example, the human brain would have an energy efficiency of around 1e20 bit erasures per watt, which is less than one order of magnitude from the Landauer limit at room temperature of 300 K.

Needless to say, I find this estimate highly unrealistic. We have no idea how to build practical densely packed devices which get anywhere close to this limit; the best we can do at the moment is perhaps 5 orders of magnitude away. Are you really thinking that the human brain is 5 OOM more energy efficient than an A100?

Still, even this estimate is much more realistic than your claim that the human brain might take 8e34 FLOP to train, which ascribes a ludicrous ~ 1e26 FLOP/s computation capacity to the human brain if this training happens over 20 years. This obviously violates the Landauer limit on computation and so is going to be simply false, unless you think the human brain loses less than one bit of information per 1e5 floating point operations it's doing. Good luck with that.

I notice that Steven Byrnes has already made the argument that these estimates are poor, but I want to hammer home the point that they are not just poor, they are crazy. Obviously, a mistake this flagrant does not inspire confidence in the rest of the argument.

I guess I just feel completely different about those conditional probabilities.

Unless we hit another AI winter the profit and national security incentives just snowball right past almost all of those. Regulation? "Severe depression"

I admit that thr loss of taiwan does innfact set back chip manufactyre by a decade or more regardless of resoyrces thrown at it but every other case just seems way off (because of the incentive structure)

So we're what , 3 months post chatgpt and customer service and drive throughs are solved or about to be solved? , so lets call that the lowrst hanging fruit. So just some quick back of the napkin google fu , the customer service by itself is a 30 billion dollar industry just in the US.

And how much more does the math break down if say , we have an AGI that can do construction work (embodied in a robot) at say 90% human efficiency for...27 dollars an hour?

In my mind every human task fully (or fully enough) automated snowballs the economic incentive and pushes more resources and man hours into solving problems with material science and things like...idk piston designs or multifunctionality or whatever.

I admit I'm impressed by the collected wisdom and apparent track records of these authors but it seems like its missing the key drivers for further improvement in the analysis.

Like would the authors have put the concept of a smartphone at 1% by 2020 if asked in 2001 based on some abnormally high conditionals about seemingly rational but actually totally orthagonal concern based on how well palm pilots did?

I also dont see how the semi conductor fab bottleneck is such a thing? , 21 million users of openai costs 700k a day to run.

So taking some liberties here but thats 30 bucks a person (so a loss with their current model but thats not my point)

If some forthcoming iteration with better cognitive architecture etc costs about that then we have , 1.25$ per hour to replace a human "thinking" job.

Im having trouble seeing how we don't rapidly advance robotics and chip manufacture and mining and energy production etc when we stumble into a world where thats the only bottleneck standing in our way to 100% replacemwnt of all useful human labor.

Again , you got the checkout clerks at grocery stores last decade. 3 months in and the entire customer service industry is on its knees. Even if you only get 95% as good as a human and have to sort of take things one at a time to start with , all that excess productivity and profit then chases the next thing. It snowballs from here.

I think this is an excellent, well-researched contribution and am confused about why it's not being upvoted more (on LW that is; it seems to be doing much better on EAF, interestingly).

Your probabilities are not independent, your estimates mostly flow from a world model which seem to me to be flatly and clearly wrong.

The plainest examples seem to be assigning

despite current models learning vastly faster than humans (training time of LLMs is not a human lifetime, and covers vastly more data) and the current nearing AGI and inference being dramatically cheaper and plummeting with algorithmic improvements. There is a general factor of progress, where progress leads to more progress, which you seem to be missing in the positive factors. For the negative, derailment that delays enough to push us out that far needs to be extreme, on the order of a full-out nuclear exchange, given more reasonable models of progress.

I'll leave you with Yud's preemptive reply:

Taking a bunch of number and multiplying them together causes errors to stack, especially when those errors are correlated.

Thanks, the parts I've read so far are really interesting! 

I would point out that the claim that we will greatly slow down, rather than scale up, electricity production capacity, is also a claim that we will utterly fail to even come anywhere close to hitting global decarbonization goals. Most major sectors will require much more electricity in a decarbonized world, as in raising total production (not just capacity) somewhere between 3x to 10x in the next few decades. This is much more than the additional power which would be needed to increase chip production as described, or to power the needed compute. The amount of silicon needed for solar panels also dwarfs that needed for wafers (yes, I know, very different quality thresholds, but still). Note that because solar power is currently the cheapest in the world, and still falling, we should also expect electricity production costs to go down over time.

I think it's also worthwhile to note that the first "we" in the list means humans, but every subsequent "we" includes the transformative AI algorithm we've learned out to build, and all the interim less capable AIs we'll be building between now and then. Not sure if/how you're trying to account for this?

Compute is not the limiting factor for mammalian intelligence.  Mammalian brains are organized to maximize communication.  The gray matter, where most compute is done, is mostly on the surface  and the white matter which dominate long range communication, fills the interior, communicating in the third dimension.

If you plot volume of white matter vs. gray matter across the various mammal brains, you find that the volume of white matter grows super linearly with volume of gray matter.   https://www.pnas.org/doi/10.1073/pnas.1716956116

As brains get larger, you need a higher ratio of communication/compute.

Your calculations, and Cotras as well, focus on FLOPs but the intelligence is created by communication.  

Does this do the thing where a bunch of related events are treated as independent events and their probability is multiplied together to achieve a low number?

edit: I see you say that each event is conditioned on the previous events being true. It doesn't seem like you took that into account when you formulated your own probabilities.

According to your probabilities, in the world where: We invent algorithms for transformative AGI, We invent a way for AGIs to learn faster than humans, AGI inference costs drop below $25/hr (per human equivalent), and We invent and scale cheap, quality robots... There's only a 46% chance that we massively scale production of chips and power. That seems totally unreasonable on its face. How could cheap, quality robots at scale not invariably lead to a massive scaling of the production of chips and power?

I think splitting the odds up this way just isn't helpful. Your brain is not too good at pondering, "What is the probability of X happening if A, B, C, ... W all happen?" Too much of "What is the probability of X independent of everything else" will creep in. Unless you have a mechanistic process for determining the probability of X

I'm going to try a better calculation on what I think are the pivotal cruxes.

Algorithms, as I mentioned in my last comment [LW(p) · GW(p)], depends on compute.  Iff enough training compute exists by 2043, then it is inevitable that a tAGI grade algorithm will be found, simply through RSI.

Learning faster than humans is just "algorithms".  Ted Sanders [LW · GW], you stated that autonomous cars not being as good as humans was because they "take time to learn".  This is completely false, this is because the current algorithms in use, especially the cohesive software and hardware systems and servers around the core driving algorithms, have bugs.  If we invent better algorithms that can generalize better, train on more examples, and using the same or a different tAGI system, seal up the software and firmware and system design stack by hunting down all the bugs that can be found with a careful review of every line (in assembly!), and seal up the chip design, so that you have a clean design with no known errata - all things humans are capable of doing already, we just can't afford the labor hours to do this - then the actual Driver agent can learn it's job in a few months.  

So compute's a big part of the low probability.  

Let's calculate this one fresh.  

Epistemic status: i have a master's in biomedical science, which included a medical school neuroscience course, a bachelor's in computer engineering, and a master's in CS, and I work as a platform engineer on an AI accelerator project.

Doing it from the bio anchors view:
There are approximately 86 billion neurons in the human brain, with approximately 1000 connections each.  Noise is extremely high, we're going to be generous and assume 8 bits of precision, which is likely more than the brain actually achieves.  We'll be generous and assuming 1 khz and that all connections are active.

Both are false, and the asynchronous calculations and noise cause a lot of error that a comparable synthetic system will not make.  This is the reason why you can safely knock 1-2 orders of magnitude right off the top, this is well justified.  Timing jitter alone from a lack of a central clock injects noise into all the synapses, synapses are using analog accumulators that change their charge levels from crosstalk and many forms of biological noise (digital accumulators reject such noise under conditions insufficient to cause a bit flip).  I am not accounting for this.

So we need 8.6e+16, or 10^17 flops in int8.  I'll have to check the essay to see where you went wrong, but 10^20 is clearly not justified.   Arguably flat out wrong.

With the current gen H100 and 2000 dense int8 tops, you would need 43 H100s.  

 Compute isn't the issue, VRAM is. 43  H100s only have 3440 gb of VRAM.
 
We need 16 bit accumulators on 8 bit weights, and there's VRAM lost from the very high sparseness of the human brain.  So for a quick estimate, if we consume 8 bytes per synapse, we need 688 terabytes of VRAM, or 400 H100s.  

So the estimate narrows to, present day, [43, 400] H100s.  I think this is a pretty hard estimate, you're going to need to produce substantial evidence to disprove.  The reason it's so firm is because you probably need 1/10 or less the above because the brain makes so many errors and randomly loses cells, so to stay alive we have a lot of redundant synapses that contain the same information.  An efficient tAGI design can obviously reliably know each weight will always be factored in every timestep, and will always be available, and ECC memory is common so undetected errors will rarely happen.
 
During inference time, how many H100s equal "one human equivalent"?  The answer may surprise you.  It's about 18.4.  The reason is that the hardest working humans, working "996" (12 hours a day, 6 days a week) are still only working 42% of the time, while H100s work 24/7.

The reason you don't need 400 is because you batch the operations.  There will be approximately 20 "human level" sessions across an array of 400 H100s, running a model that has unique weights tiled across all 400 cards, since the compute load per human level session is only about 43.  You can reduce the batch size if you want better latency, we do that for autonomous cars.

H100s now cost about $2.33 an hour.  
 

So present day, the cost is $100.19 an hour.  

I thought I was going to have to argue for better than the 38x hardware improvement you estimated, mention each H100 actually costs about $2500 to build not $25k, or describe a method that is essentially the brain's System 1 and System 2, but there's no need, compute isn't a problem.  I'm sure you can see that getting a factor of 4 reduction is fairly trivial.

Unless I made a major error, I think you may need a rewrite/retraction, as you're way off.  I look forward to hearing your response.   Ted, how do you not know this, you can just ping a colleague and check the math above.  Anyone at OpenAI that does the modeling for infrastructure will be able to check whether or not the above is correct or not.




Remaining terms:  robotics is a big open.  I will simply accept your 60% though arguably in a world where the tAGI algorithms exist, and are cheap to run, there would be total war levels of effort put into robot manufacturing.  Not to mention it will self amplify, as previously built robots contribute their labor to some of the tasks involved in building more robots.

Massively scaling chips and power, at <$25 per human equivalent, would happen inherently, being able to get a human equivalent worker that is reliable for this low a cost creates immense economic pressure to do this.  It's also coupled to cheap robotics - repetitive tasks like manufacturing and building solar panels and batteries, both of which are fundamentally a product that you 'tile', every solar cell and every battery like all it's peers, are extremely automatable.  Data centers could be air cooled and installed in large deserts, you would just accept the latency, using local hardware for the real-time control, and you would accept a less than 100% level of service, so compute availability would fluctuate with the seasons.  

All the other terms: human regulation, AI-disaster, wars, pandemics, economic depression have the issue that each is conditional on all the relevant powers fail the same way.  So all the probability estimates need to be raised to the 3rd power, conservatively.  

Conclusion: iff my calculation on the compute required is possible, this problem collapses.  Only the tAGI algorithm matters.  In worlds where the tAGI algorithm was discovered at least 5 years before 2043, tAGI is closer to certain.  Only extreme events - nuclear war, every tAGI becomes rampant in insidious and dangerous ways, coordinated worldwide efforts to restrict tAGI development - could derail it from there.  

As for thinking again about the broader picture: your "90% of tasks" requirement, below which apparently the machine is not transformative, which seems like a rather bad goalpost, is the issue.  

A reasonable tAGI algorithm is a multimodal algorithm, trained on a mixture of supervised and reinforcement learning, that is capable of learning any task to expert level iff it receives sufficient feedback on task quality to become an expert.  So there's AI scientists and AI programmers and AI grammer editors and many forms of AI doing manufacturing.

But can they write books that outsell all human authors?  Depends on how much structured data they get on human enjoyment of text samples.

Can they write movies that are better than human screenwriters?  Same limitation.

Can they go door to door and evangelize?  Depends on how much humans discriminate against robots.

There are a whole bunch of tasks that arguably do not matter that would fit into that 90%.  They don't matter because the class of tasks that do matter - self improving AI, robots, and technology - all are solvable.  "matter" defined by "did humans require the task to be done at all to live" or "does the task need to be done before robots can tear down the solar system".  If the answer is no, it doesn't matter.  Car salesman would be an example of a task that doesn't matter.  tAGI never needs to know how to do it since it can just offer fixed prices through a web page, and answer any questions about the car or transaction based on published information instead of acting in a way motivated to maximize commission.

A world where the machine can only do 50% of tasks, including all the important ones, but legal regulations means it can't watch children or argue a case in court or supervise other AI or serve in government - is still a transformed one.

Can you explain how Events #1-5 from your list are not correlated? 

For instance, I'd guess #2 (learns faster than humans) follows naturally -- or is much more likely -- if #1 (algos for transformative AI) comes to pass. Similarly, #3 (inference costs <$25/hr) seems to me a foregone conclusion if #5 (massive chip/power scale) and #2 happen.

Treating the first five as conditionally independent puts you at 1% before arriving at 0.4% with external derailments, so it's doing most of the work to make your final probability miniscule. But I suspect they are highly correlated events and would bet a decent chunk of money (at 100:1 odds, at least) that all five come to pass. 

I am confused about your use of the term "calibration". Usually it means correctly predicting the probabilities of events, as measured by frequencies. You are listing all the times you were right, without assigning your predicted probability. Do you list only high-probability predictions and conclude that you are well calibrated for, say, 95%+ predictions, since "no TAI by 2043" is estimated to be 99%+?

RE robots / “Element 4”:

IMO the relevant product category here should be human-teleoperated robots, not independent robots.

Robot-control algorithms are not an issue, since we’re already conditioning on Element 1 (algorithms for TAI). Humans can teleoperate a teleoperable robot, and therefore if Element 1 comes to pass, those future AI algorithms will be able to teleoperate a teleoperable robot too, right?

And human-teleoperated robots can already fold sheets, move boxes, get around a cluttered environment, etc., no problem. I believe this has been true for a very long time.

And when I look at teleoperated robots, I find things like Ugo which might or might not be vaporware but they mention a price point below $10/day. (ETA: see also the much more hardcore Sarcos Guardian XT. Pricing is not very transparent, but one site says you leas it for $5K/month, which isn’t bad considering how low the volumes are.)

More generally, I think “progress in robotics” is not the right thing to be looking at in this context, because “progress in robotics” has been mainly limited by algorithms (and on-board compute). I think teleoperated robots are technologically much simpler, more like the kind of thing that a wide variety of companies can crank out at scale, almost as soon as there’s demand.

If we divide the list of "10 necessary events" into two groups of five, the first five being technical achievements and the last five being ways to derail technological society... then I suppose the standard doomer view would be that once the first necessary event is achieved (AGI algorithms) then the other technical achievements become 100% possible (edit: because AI figures out how to do them); and that whether or not the derailing events occur, boils down to whether AI lets them happen. 

edit: The implication being that algorithmic progress controls everything else. 

Ted - thank you for sticking your neck out and writing this seminal piece. I do believe it has some basic fundamental missed. Allow me to explain in simpler "energy equivalence" terms. 

Let us take an elementary task say driving in moderate traffic for say 60 minutes or 30-40 km. The total task will involve say ~250-500 decisions (accelerate/ decelerate/ halt, turn) and some 100,000-10,000,000 micro-observations depending upon the external conditions, weather, etc. A human body (brain + senses + limbs) can simulate all the changes on just a grain of rice (less than 10 kJ of energy)

In order to do similar tasks (even if we achieve near free electricity), the cost in terms of energy (kJ consumed) will be far too high. 

Am I making sense?

Steve Jobs said, a human with a bicycle is more efficient than a gazelle (the most efficient animal). And thus the corollary, a human with a computer is the most efficient being. Using a computer to replace a human is also the most inefficient conquest - doomed to fail; time immaterial. 

Lots of love,

Sachin

I think the headline and abstract for this article are misleading. As I read these predictions, one of the main reasons that "transformative AGI" is unlikely by 2043 is because of severe catastrophes such as war, pandemics, and other causes. The bar is high, and humanity is fragile.

For example, the headline 30% chance of "derailment from wars" is the estimate of wars so severe that they set back AI progress by multiple years, from late 2030s to past 2043. For example, a nuclear exchange between USA and China. Presumably this would not set back progress on military AI, but it would derail civilian uses of AI such that "transformative AGI", as defined, doesn't come by 2043.

Humanity is on track to develop transformative AGI but before it does, World War III erupts, which kills off key researchers, snarls semiconductor supply chains, and reprioritizes resources to the war effort and to post-war rebuilding.

The authors squarely blame the development of AI technologies for much of the increased risk.

Progress in AGI will be destabilizing; therefore, conditional on successful progress in AGI, we expect the odds of war will rise. Conditional on being on a trajectory to transformative AGI, we forecast a 40% chance of severe war erupting by 2042.

There's also a 10% estimated risk of severe wars so destabilizing that they delay transformative AGI from late 2030s to after 2100, or perhaps forever. An analogy would be if WW2 was so severe that nobody was ever able to make tanks again.

Only if [wars] result in a durable inability to produce, or lack of interest in, transformative AGI will they matter on this timescale.

The discussion of regulation and pandemics are similarly terrifying. This is not a paper that should make anyone relax.

How has this forecast changed in the last 5 years?  Has widespread and rapid advance of non-transformative somewhat-general-purpose LLMs change any of your component predictions?

I don't actually disagree, but MUCH of the cause of this is an excessively high bar (as you point out, but it still makes the title misleading).  "perform nearly all valuable tasks at human cost or less" is really hard to put a lot of stake in, when "cost" is so hard to define at scale in an AGI era.  Money changes meaning when a large subset of human action is no longer necessary.  And even if we somehow could agree on a unit of measure, comparing the cost of creating a robot vs the cost of creating and raising a human is probably impossible.

Looking back:

> We massively scale production of chips and power

This will probably happen, actually the scale to which it's happening would probably shock the author if we went back in time 9 months ago. Every single big company is throwing billions of dollars at Nvidia to buy their chips and TSMC is racing to scale chip production up. Many startups+other companies are trying to dethrone Nvidia as well.

> AGI inference costs drop below $25/hr (per human equivalent)

This probably will happen.  It seems pretty obvious to me that inference costs fall off a cliff. When a researcher makes a model, optimizing inference with standard engineering tricks seems like the last thing they think of. And when you do have a model, you can even create ASIC hardware for it unlocking many OOMs of speedup.
> We invent and scale cheap, quality robots

This is starting to happen. 
 
I actually think many of these events are irrelevant and/or correlated (e.g. if we make an algorithm for AGI, it seems unlikely to me we also don't make it learn faster than humans, and the latter doesn't even matter much), but taking the calculation at face value, the probability should go up a lot

Have you seen Jacob Steinhardt's article https://www.lesswrong.com/posts/WZXqNYbJhtidjRXSi/what-will-gpt-2030-look-like [LW · GW] ? It seems like his prediction for a 2030 AI would already meet the threshold for being a transformative AI, at least in aspects not relating to robotics. But you put this at less than 1% likely at a much longer timescale. What do you think of that writeup, where do you disagree, and are there any places where you might consider recalibrating?

One thing I don't understand, why the robot step as necessary? AGI that is purely algorithmic but still able to do all human intellectual tasks, including science and designing better AGI, would be transformative enough. And if you had that, the odds of having the cheap robots shoot up as now the AGI can help.