LessWrong 2.0 Reader

I mean, yes, likely? But it doesn't make it easy to evalute whether model is going to have world-ending capabilities without getting the world ended.

I think that you can probably put a lot inside a 1.5B model, but I just think that such a model is going to be very dissimilar to GPT-2 and will likely utilize much more training compute and will probably be the result of pruning (pruned networks can be small, but it’s notoriously difficult to train equivalent networks without pruning).

Also, I'm not sure that the training of o1 can be called "COT fine-tuning" without asterisks, because we don’t know how much compute actually went into this training. It could easily be comparable to the compute necessary to train a model of the same size.

I haven’t seen a direct comparison between o1 and GPT-4. OpenAI only told us about GPT-4o, which itself seems to be a distilled mini-model. The comparison can also be unclear because o1 seems to be deliberately trained on coding/math tasks, unlike GPT-4o.

(I think that "making predictions about the future based on what OpenAI says about their models in public" should generally be treated as naive, because we are getting an intentionally obfuscated picture from them.)

What I am saying is that if you take the original GPT-2, COT prompt it, and fine-tune on outputs using some sort of RL, using less than 50% of the compute for training GPT-2, you are unlikely (<5%) to get GPT-4 level performance (because otherwise somebody would already do that.

Very funny that we had this conversation a couple of weeks prior to transparently deciding that we should retaliate with p=.7!

I think you're getting this exactly wrong (and this invalidates most of the OP). If you find a model that has a constant factor of 100 in the asymptotics that's a huge deal if everything else has log scaling. That would already present discontinuous progress and potentially put you at ASI right away.

Basically the current scaling laws, if they keep holding are a lower bound on the expected progress and can't really give you any information to upper bound it.

(maybe the part that seems unrealistic is the difficulty of eliciting values for the power set of possible coalitions, as generating a value for any one coalition feels like an expensive process, and the size of a power set grows exponentially with the number of players)

This is extremely well produced, I think it's the best introduction to Shapley values I've ever seen. Kudos for the simple explanation and approachable designs!

(Not an indictment of this site, but with this as with other explainers, I still struggle to see how to apply Shapley values to any real world problems haha - unlike something like quadratic funding, which also sports fancy mechanism math but is much more obvious how to use)

[Epistemic status: somewhat informed speculation] TLDR: I do not believe China was a major threat source, recession makes it slightly less likely they will be one too. Conventional wars are more likely to happen, and their effect on AI development is uncertain.

I generally do not think China is a big of a threat in the AGI race as some others (notably Aschenbrenner) think. I think for AGI to be first developed in China, several factors need to be true: China has more centralized compute available than other countries, open models are near the frontier but not over the AGI limit, and China's attitude towards developing AGI shifts (possibly due to race dynamics). I think for compute they are currently not on track, for frontier models there is a lag, and attitude is towards trying not to develop AGI, at least publicly and it seems also privately as far as we can glimpse. While the Chinese public is more techno-optimistic than the US, the CCP is leaning towards engineers rather than politicians, and senior advisors in AI are AI-pilled.

The current recession in China is due to a set of complex causes, but it's a mix of politics and economics, and politics are quite slow to budge. I don't want to get too much into it, but the banking sector is stretched thin with a lot of workers unable to pay back mortgages on apartments which were not completed due to real-estate developers building too much real estate and ending up holding the bag with many unsold apartments - with most of them being second apartments, so not necessities but "investments". This is causing a loop of bankruptcies which is hard to stop, and has led to overall pessimism over the future. Lowering of the interest rates and making money available to banks has caused loans to be available, but people are skeptical to take them due to what they perceive as an uncertain future. CCP is likely to work on things which make the future more certain, large infrastructure projects such as bridges and dams as they have historically done, at least for some time. Nuclear power plants and hydroelectric dams definitely will qualify, but enormous compute clusters (using which chips? overpriced smuggled ones?) will likely not.

That is not to say that, if it seems like US is racing towards AGI and is reaping benefits from advanced AI, China will not put all the resources of a centralized government into catching up - and that can be quite a few resources since they can comandeer private enterprise or property to do so. If countries of the world play it sane, actually negotiate international limits, and meet China where they want to be met (CCP has many reasons not to want AGI) I do not expect China to be a threat to existence directly.

Recession is also more likely to make China want to blame bad economic results on foreign influence, and perhaps more likely to stoke international conflicts directly. I am personally not likely to want to live in a country bordering China in the next 10 years. How this will influence AGI is tough to predict - more resources spent on war means less on AI development, unless AI development is essential for a warfare edge, in which case we should expect a boom in AI development. The earlier the conflict happens, the less likely AI is to play a major role in warfare.

A thing that didn't appear on your list, and which I think is pretty important (cruxy for a lot of discussions; closest to what Hanson meant in the FOOM debate), is "human-relative discontinuity/speed". Here the question is something like: "how much faster does AI get smarter, compared to humans?". There's conceptual confusion / talking past each other in part because one aspect of the debate is:

• how much locking force there is between AI and humans (e.g. humans can learn from AIs teaching them, can learn from AI's internals, can use AIs, and humans share ideas with other humans about AI (this was what Hanson argued))

and other aspect is

• how fast does an intelligence explosion go, by the stars (sidereal).

If you think there's not much coupling, then sidereal speed is the crux about whether takeoff will look discontinuous. But if you think there's a lot of coupling, then you might think something else is a crux about continuity, e.g. "how big are the biggest atomic jumps in capability".

Gradual/Sudden

I don't think contemporary theory has ignored this - see recent theories of density-dependent selection here: (article making the same point), (review). The fundamental issue you're hinging on is that absolute population growth (most effective exploitation of resources) is an ecological concept, not an evolutionary one, and population ecology theory is less well-known outside its field than population genetic theory.