LessWrong 2.0 Reader

(maybe the part that seems unrealistic is the difficulty of eliciting values for the power set of possible coalitions, as generating a value for any one coalition feels like an expensive process, and the size of a power set grows exponentially with the number of players)

This is extremely well produced, I think it's the best introduction to Shapley values I've ever seen. Kudos for the simple explanation and approachable designs!

(Not an indictment of this site, but with this as with other explainers, I still struggle to see how to apply Shapley values to any real world problems haha - unlike something like quadratic funding, which also sports fancy mechanism math but is much more obvious how to use)

[Epistemic status: somewhat informed speculation] TLDR: I do not believe China was a major threat source, recession makes it slightly less likely they will be one too. Conventional wars are more likely to happen, and their effect on AI development is uncertain.

I generally do not think China is a big of a threat in the AGI race as some others (notably Aschenbrenner) think. I think for AGI to be first developed in China, several factors need to be true: China has more centralized compute available than other countries, open models are near the frontier but not over the AGI limit, and China's attitude towards developing AGI shifts (possibly due to race dynamics). I think for compute they are currently not on track, for frontier models there is a lag, and attitude is towards trying not to develop AGI, at least publicly and it seems also privately as far as we can glimpse. While the Chinese public is more techno-optimistic than the US, the CCP is leaning towards engineers rather than politicians, and senior advisors in AI are AI-pilled.

The current recession in China is due to a set of complex causes, but it's a mix of politics and economics, and politics are quite slow to budge. I don't want to get too much into it, but the banking sector is stretched thin with a lot of workers unable to pay back mortgages on apartments which were not completed due to real-estate developers building too much real estate and ending up holding the bag with many unsold apartments - with most of them being second apartments, so not necessities but "investments". This is causing a loop of bankruptcies which is hard to stop, and has led to overall pessimism over the future. Lowering of the interest rates and making money available to banks has caused loans to be available, but people are skeptical to take them due to what they perceive as an uncertain future. CCP is likely to work on things which make the future more certain, large infrastructure projects such as bridges and dams as they have historically done, at least for some time. Nuclear power plants and hydroelectric dams definitely will qualify, but enormous compute clusters (using which chips? overpriced smuggled ones?) will likely not.

That is not to say that, if it seems like US is racing towards AGI and is reaping benefits from advanced AI, China will not put all the resources of a centralized government into catching up - and that can be quite a few resources since they can comandeer private enterprise or property to do so. If countries of the world play it sane, actually negotiate international limits, and meet China where they want to be met (CCP has many reasons not to want AGI) I do not expect China to be a threat to existence directly.

Recession is also more likely to make China want to blame bad economic results on foreign influence, and perhaps more likely to stoke international conflicts directly. I am personally not likely to want to live in a country bordering China in the next 10 years. How this will influence AGI is tough to predict - more resources spent on war means less on AI development, unless AI development is essential for a warfare edge, in which case we should expect a boom in AI development. The earlier the conflict happens, the less likely AI is to play a major role in warfare.

A thing that didn't appear on your list, and which I think is pretty important (cruxy for a lot of discussions; closest to what Hanson meant in the FOOM debate), is "human-relative discontinuity/speed". Here the question is something like: "how much faster does AI get smarter, compared to humans?". There's conceptual confusion / talking past each other in part because one aspect of the debate is:

• how much locking force there is between AI and humans (e.g. humans can learn from AIs teaching them, can learn from AI's internals, can use AIs, and humans share ideas with other humans about AI (this was what Hanson argued))

and other aspect is

• how fast does an intelligence explosion go, by the stars (sidereal).

If you think there's not much coupling, then sidereal speed is the crux about whether takeoff will look discontinuous. But if you think there's a lot of coupling, then you might think something else is a crux about continuity, e.g. "how big are the biggest atomic jumps in capability".

Gradual/Sudden

I don't think contemporary theory has ignored this - see recent theories of density-dependent selection here: (article making the same point), (review). The fundamental issue you're hinging on is that absolute population growth (most effective exploitation of resources) is an ecological concept, not an evolutionary one, and population ecology theory is less well-known outside its field than population genetic theory.

I'm kind of confused which unilateralist got to design the game. You say:

The first person to click the Unilateral Virtue Link was a proponent of "Avoiding actions that noticeably increase the chance that the world will end." But, this virtue was actually in the majority. The first unilateralist of a Virtue Minority was a proponent of "Accurately reporting your epistemic state."

A year later, as we decided what to do for Petrov Day, we decided to lure the first unilateralist into a surprise meeting, where I then said "Here's a reminder of what happened in Petrov Day last year. You now have one hour to design this year's Petrov game. Go."

So it sounds like the unilateralist who wanted to avoid actions that noticeably increase the chance the world will end got picked. But then it sounds like the winner made a game that was supposed to be about accurately reporting epistemic state:

The designer had (I think?) initially noticed the "focus on accurately reporting epistemic state" aspect, but said that during the stressful hour of designing the game had eventually forgotten that. The version they handed off wasn't particularly optimized for that, but the framework of a social deception game seemed to me to be a good substrate for "accurate epistemic reporting." [...] It seemed important to me that Petrov's payoff specifically be about reporting his beliefs

Relevant: https://andymatuschak.org/hmwl/

I agree with the spirit of what you are saying but I want to register a desire for "long timelines" to mean ">50 years" or "after 2100". In public discourse, heading Yann LeCunn say something like "I have long timelines, by which I mean, no crazy event in the next 5 years" - it's simply not what people think when they think long timelines, outside of the AI sphere.

Ah, that does make it almost impossible then. Such a utility function when paused must have constant value for all outcomes, or it will have incentive to do something. Then in the non-paused state the otherwise reachable utility is either greater than that (in which case it has incentive to prevent being paused) or less than or equal (in which case its best outcome it to make itself paused).