Sam Altman’s Chip Ambitions Undercut OpenAI’s Safety Strategy

I think that Sam's actions increase the likelihood of a slow takeoff.

Consider Paul's description of a slow takeoff from the original takeoff speed debate [LW · GW]

right now I think hardware R&D is on the order of $100B/year, AI R&D is more like $10B/year, I guess I'm betting on something more like trillions? (limited from going higher because of accounting problems and not that much smart money)

“By doing this chip thing, Sam Altman is likely to make takeoff faster (on the margin) than it otherwise would be.”

It makes takeoff faster, but is not necessary a fast takeoff.  A fast takeoff is a specific science fiction scenario that happens many times faster than human beings can respond.  

Let's look at it with numbers:

Suppose you actually need 80 H100s per AGI.  Suppose 2 generations of Moore's law happen before AGI releases, so the number drops to 20.

Suppose they are 25k per card.  so $500,000 per "person-equivalent", though that person works 24/7 instead of at most 996, and can load multiple models.

If annual spending on chip production in 5 years is 1000 billion instead of 528 billion now, and 90%! goes to solely AI chips, that's 1.8 million "person equivalents" added to the workforce each year.  (I am pretending all the other chips are free)

If we assume at least 30% is reinvested into AI training more efficient models, and also correct for the duty cycle, that's 2.94 million people added to the workforce per year.  (Exponential gains are also slow, if you need 2-3 years per doubling thats not helping much especially since many of the instances aren't being reinvested)

I think this is what Sam means when he said “It will change the world much less than we all think and it will change jobs much less than we all think".

Update:

Suppose instead the 7T investment happens.  How much does this speed up production? 

Let's assume we get 30 percent ROI on 7T.  7T has paid for more chip making equipment, more silicon production plants, better automation of these (near future low hanging fruit only!)

And let's assume we are paying cost for the chips, and also assume the net cost is double.  So it's $3000 (current estimates) to build a GPU, and $3000 of support ICs to use it.  

So 2.1 trillion USD each year in ICs, and every $120k is another AGI instance.  

That's like adding another 572 million people per year.

Hmm.  Yeah that's kind of a singularity,.  ICs stops being a limiting factor and then obviously you are adding another China to the Earth every 2 years.  Especially if you factor in exponential growth.  Or how you functionally have 245 million test dummies/automated AI researchers each year...

Update: well it's kind of a fast takeoff actually. 

That's not a fast takeoff, that's a slow takeoff, and it's not an impulse.  

A fast takeoff would be a scenario where say, enough GPUs to host 1 billion extra humans (or the cognitive equivalent) was already "out in the world", and also most of the hardware was already in clusters that can host big AI models, and it was secured poorly and monitored poorly.   Some of the original AI timeline calculations were for when $1000 of compute would equal a human brain.  Such a world would be at risk for a fast takeoff, where every random computer in a gas station can host an AGI.

What exactly do you think GPT5 could do in that scenario?

At some model capabilities level (gpt-8?), the overall capabilities will be subhuman, but the model will be able to control robots to do at least 90 percent of the manufacturing and resource gathering steps needed to build more robots. This reduces the cost of building robots by 10 times (ok not actually 10 times but pretending land and IP is free...), increases the total number of robots humans can build by 10 times, and assuming the robots can be ordered to build anything that is similar to the steps needed to build robots (a rocket, a car, a house all have similar techniques to build) , it increases human resiliency.

Any problems humans have, they have 10 times the resources to deal with those problems.*

They have 10 times the resources to make bunkers (future nuclear wars and engineered pandemics), manufacture weapons (future wars), can afford every camera to have a classifier gpu (terrorism and mass shooting), can afford more than 10 times the spacecraft (off planet colonies maybe, more telescopes to see aliens arriving), can produce 10 times as much food, housing, can afford to use solely clean energy for everything, and so on.

You likely objection will be that humans could not maintain control of a bunch of general purpose robotics, an ASI would just "hack in" and take over them all. However, if for the sake of argument, you think there might be a way to secure the equipment using methods that are not hackable, it would help humans a lot.

This also helps humans with rogue AI - it gives them 10 times the resources for monitoring systems and 10 times the military hardware to deal with rebels. The "aliens" case is a superset of the "rogue AI" case, and it's the same thing - assuming interstellar spacecraft are tiny, humans at least have a fighting chance if they have robots capable of exponential growth and several years of warning.

So it's possible to come to the conclusion that humans have their best chance of survival "getting strapped" with near future armies of robots, carefully locked down with many layers of cybersecurity and isolation, to survive the challenges that are to come.

And that "pausing everything" or "extremely slow AI progress with massive amounts of review and red tape" will have the outcome of China during the Opium wars, China again in WW2, France during WW2, Ukraine right now, USA automakers during the 80s...

Failing to adopt new weapons and tech I believe has not yet paid off in human history. AI could be the exception to the trend.

• Actually more than 10 times, since humans need a constant amount of food, shelter, medicine, and transport. Also if the automation is 99 percent that's 100 times and so on.

No, I will not agree that GPT5 will not destroy the world, cause I have no idea what it will be capable of.

Great, this appears to be an empirical question that we disagree on!

I (and the Manifold prediction market I linked) think there is a tiny chance that GPT-5 will destroy the world.

You appear to disagree.  I hope you are buying "yes" shares on Manifold and that one of us can "update our priors" a year from now when GPT-5 has/has-not destroyed the world.