LessWrong 2.0 Reader

The four questions you ask are excellent, since they get away from general differences of culture or political system, and address the processes that are actually producing Chinese AI. 

The best reference I have so far is a May 2024 report from Concordia AI on "The State of AI Safety in China". I haven't even gone through it yet, but let me reproduce the executive summary here: 

The relevance and quality of Chinese technical research for frontier AI safety has increased substantially, with growing work on frontier issues such as LLM unlearning, misuse risks of AI in biology and chemistry, and evaluating "power-seeking" and "self-awareness" risks of LLMs. 

There have been nearly 15 Chinese technical papers on frontier AI safety per month on average over the past 6 months. The report identifies 11 key research groups who have written a substantial portion of these papers. 

China’s decision to sign the Bletchley Declaration, issue a joint statement on AI governance with France, and pursue an intergovernmental AI dialogue with the US indicates a growing convergence of views on AI safety among major powers compared to early 2023. 

Since 2022, 8 Track 1.5 or 2 dialogues focused on AI have taken place between China and Western countries, with 2 focused on frontier AI safety and governance. 

Chinese national policy and leadership show growing interest in developing large models while balancing risk prevention. 

Unofficial expert drafts of China’s forthcoming national AI law contain provisions on AI safety, such as specialized oversight for foundation models and stipulating value alignment of AGI. 

Local governments in China’s 3 biggest AI hubs have issued policies on AGI or large models, primarily aimed at accelerating development while also including provisions on topics such as international cooperation, ethics, and testing and evaluation. 

Several influential industry associations established projects or committees to research AI safety and security problems, but their focus is primarily on content and data security rather than frontier AI safety. 

In recent months, Chinese experts have discussed several focused AI safety topics, including “red lines” that AI must not cross to avoid “existential risks,” minimum funding levels for AI safety research, and AI’s impact on biosecurity.

So clearly there is a discourse about AI safety there, that does sometimes extend even as far as the risk of extinction. It's nowhere near as prominent or dramatic as it has been in the USA, but it's there. 

This is really well put! 

This post made me reflect on how my working style has changed. This bounded cognition and best-case story is the main thing that I've changed in my working style over the last two years and it yields me a lot of relaxation but also a lot more creative results. I like how you mention meditation in the essay as well, it is like going into a sit, setting an intention and sticking to that during the sit, not changing it and then reflecting after it. You've set the intention, stick to it and relax.

I'm sharing this with the people I'm working with, thanks!

It's important to be precise about the specific claim we're discussing here. 

The claim that R&D is less valuable than broad automation is not equivalent to the claim that technological progress itself is less important than other forms of value. This is because technological progress is sustained not just by explicit R&D but by large-scale economic forces that complement the R&D process, such as general infrastructure, tools, and labor used to invent, implement, and deploy various technologies. These complementary factors make it possible to both run experiments that enable the development of technologies and diffuse these technologies widely after they are developed in a laboratory environment—providing straightforwardly large value.

To provide a specific operationalization of our thesis, we can examine the elasticity of economic output with respect to different inputs—that is, how much economic value increases when a particular input to economic output is scaled. The thesis here is that automating R&D alone would, by itself, raise output by significantly less than automating labor broadly (separately from R&D). This is effectively what we mean when we say R&D has "less value" than broad automation.

We seem to be misunderstanding each other a little... I am saying that given existing alignment practices (which I think mostly boil down to different applications of reinforcement learning), you can try to align an AI with anything, any verbally specifiable goal or values. Some will be less successful than others because of the cognitive limitations of current AIs (e.g. they are inherently better at being glibly persuasive than at producing long precise deductions). But in particular, there's no technical barrier that would prevent the creation of an AI that is meant e.g. to be a master criminal strategist, from the beginning. 

In the link above, one starts with models that have already been aligned in the direction of being helpful assistants that nonetheless refuse to do certain things, etc. The discovery is that if they are further finetuned to produce shoddy code full of security holes, they start becoming misaligned. To say it again: they are aligned to be helpful and ethical, then they are narrowly finetuned to produce irresponsible code, and as a result they become broadly misaligned. 

This shows a vulnerability of current alignment practices. But remember, when these AIs are first produced - when they start life as "foundation models" - they have no disposition to good or evil at all, or even towards presenting a unified personality to the world. They start out as "egoless" sequence predictors, engines of language rather than of intelligence per se, that will speak with several voices as easily as with one voice, or with no voice at all except impersonal narration. 

It's only when they are prompted to produce the responses of an intelligent agent with particular characteristics, that the underlying linguistic generativity is harnessed in the direction of creating an agent with particular values and goals. So what I'm emphasizing is that when it comes to turning a genuine language model into an intelligent agent, the agent may be given any values and goals at all. And if it had been created by the same methods used to create our current friendly agents, the hypothetical "criminal mastermind AI" would presumably also be vulnerable to emergent misalignment, if finetuned on the right narrow class of "good actions". 

Is this relevant to your question? I'm not sure that I have understood its scope correctly. 

Yeah, what would be my alternative true rejection? I don't think the normalization effect is weak, indeed I expect even just within my social circle for this whole situation to come up regularly as justification for threatening people with libel suits.

Well, that was an interesting top-down processing error.

Here's some hopefully-funny hallucination from o3. If this satisfies the bounty, then I promise I'll spend half on future subscriptions and API usage, I guess...

I've been doing this for about 10 years. This January I needed to get some new socks but my brand was discontinued so I decided to buy a few different brands and compare them. I will take this opportunity to write a public sock review.

1. CS CELERSPORT Ankle Athletic Running Socks Low Cut Sports Tab Socks (the black version of the brand you linked): I did not like the wrinkle in the back, and the texture was a bit weird. 4/5.
2. Hanes Men's Max Cushioned Ankle Socks: Cozy and nice texture, but they made my feet too hot. I might buy these if I lived somewhere colder. 4/5.
3. Hanes Men's Socks, X-Temp Cushioned No Show Socks: Nice texture, and not too hot. A little tight on the toes which makes it harder for me to wiggle them. These are the ones I decided to go with. 4.5/5.

Do you truly not believe that for your own ljfe - to use the examples there - solving aging, curing all disease, solving energy isn't even more valuable? To you? Perhaps you don't believe those possible but then that's where the whole disagreement lies. 

And if you are talking about Superintelligent AGI and automation why even talk about output per person? I thought you at least believe people are automated out and thus decoupled?

Does he not believe in AGI and Superintelligence at all? Why not just say that?

As one of the authors, I'll answer for myself. Unfortunately, I'm not exactly sure what these terms mean precisely, so I'll answer a different question instead. If your question is whether I believe that AIs will eventually match or surpass human performance—either collectively or individually—across the full range of tasks that humans are capable of performing, then my answer is yes. I do believe that, in the long run, AI systems will reach or exceed human-level performance across virtually all domains of ability. 

However, I fail to see how this belief directly supports the argument you are making in your comment. Even if we accept that AIs will eventually be highly competent across essentially all important tasks, that fact alone does not straightforwardly imply that our core thesis—that the main value from AI will come from broad automation rather than the automation of R&D—is incorrect.