My Assessment of the Chinese AI Safety Community

I'm a Westerner, but did business in China, have quite a few Chinese friends and acquaintances, and have studied a fair amount of classical and modern Chinese culture, governance, law, etc.

Most of what you're saying makes sense with my experience, and a lot of Western ideas are generally regarded as either "sounds nice but is hypocritical and not what Westerns actually do" (a common viewpoint until ~10 years ago) with a later idea of "actually no, many young Westerners are sincere about their ideas - they're actually just crazy in an ideological way about things that can't and won't work" that is a somewhat newer idea. (白左, etc)

The one place I might disagree with you is that I think mainland Chinese leadership tends to have two qualities that might be favorable towards understanding and mitigating AI risk:

(1) The majority of senior Chinese political leadership are engineers and seem intrinsically more open to having conversations along science and engineering lines than the majority of Western leadership. Pathos-based arguments, especially emerging from Western intellectuals, do not get much uptake in China and aren't persuasive. But concerns around safety, second-order effects, third-order effects, complex system dynamics, causality, etc, grounded in scientific, mathematical, and engineering principles seem to be engaged with easily at face value in private conversations, and with a level of technical sophistication that there doesn't need to be as much direct reliance on asking for industry leaders and specialists to explain and contextualize diagrams, concepts, technologies, etc. Senior Chinese leadership also seem to be better - this is just my opinion - at identifying credible and non-credible sources of technical information and identifying experts who make sound arguments grounded in causality. This is a very large advantage.

(2) In recent decades, it seems like mainland Chinese leadership are able to both operate on longer timescales - credibly making and implementing multi-decade plans and running them - as well as making rapid changes in technology adoption, regulation, and economic markets once a decision has been made in an area. The most common examples we see in the West are videos of skyscrapers being constructed very rapidly, but my personal example is I remember needing to go pay my rent with shoeboxes full of 100 renminbi notes during the era of Hu Jintao's chairmanship and being quite shocked when China went to near cashless almost overnight. 

I think those two factors - genuine understanding of engineering and technical causality, combined with greater viability for engaging in both longer timescale and short-timescale action, seem like important points worth mentioning.

Thanks for writing this.

Overall, I don't like the post much under it's current form. There's ~0 evidence (e.g. from Chinese newspapers) and there is very little actual argumentation. I like that you give us a local view but putting a few links to back your claims would be very very appreciated. Right now it's hard to update on your post given that the claims are very empirical and without any external sources.

More minorly: "A domestic regulation framework for nuclear power is not a strong signal for a willingness to engage in nuclear arms reduction" I also disagree with this statement. I think it's definitely a signal.

A "moonshot idea" I saw brought up is getting Yudkowsky's Harry Potter fanfiction translated into Chinese (please never ever do this).

Can you expand on this? Why would it be a bad idea? I have interacted with mainland chinese people (outside of china) and I'm not really making the connection. 

Thanks for writing this post! I want to note a different perspective. Although unlike OP, I have not lived in China since 2015 and am certainly more out of touch with how the country is today.

I do observe some of the same dynamics that OP describes, but I want to point out that China is a really big country with inherently diverse perspectives, even in the current political environment. I don’t see the dynamics described in this post as necessarily the dominant one, and certainly not the only one. I know a lot of young people, both in my social circle and online, that share many of the Western progressive values such as the pursuit of equality, freedom, and altruism. I see many people trying their best to live a meaningful life and do good for the world. (Of course, many people are not thinking about this at all, but that is the same everywhere. It's not like these concerns are that mainstream in the West.) As a small piece of evidence, 三联生活周刊 did an interview with me about AI safety recently, and it got 100k+ views on WeChat and only positive comments. I’ve also had a few people reach out to me expressing interest in EA/AI safety since the interview came out.

You can't just hope an entire field into being in China. Chinese EAs have been doing field-building for the past 5+ years, and I see no field.

Implying that they are simply "hoping the field into being" is really unfair to the Chinese EAs doing field building. Even in the US, EA was much less mainstream 5 years ago.

The main reason I could find is the lack of interfaces, people who can navigate both the Western EA sphere and the Chinese technical sphere.

I agree this is a major bottleneck.

A "moonshot idea" I saw brought up is getting Yudkowsky's Harry Potter fanfiction translated into Chinese (please never ever do this).

This has already been done, and has pretty good reviews and some discussions. 

I've looked through the EA/Rationalist/AI Safety forums in China

If these are public, could you post the links to them?

there is only one group doing technical alignment work in China

Do you know the name of the group, and what kinds of approaches they are taking toward technical alignment?

This seems to be arguing against a starry-eyed idealist case for an "AI disarmament treaty", but not really against a cynical/realistic case. (At first I was going to say "arguing against a strawman", but no, there are in fact lots of starry-eyed idealists in alignment.)

Here's my cynical/realistic case for an "AI disarmament treaty" (or something vaguely in that cluster) with China. As the post notes, the regulations mostly provide evidence that Beijing sees near-term AI as a potential threat to stability that needs to be addressed with regulation. For purposes of an AI treaty, that's plausibly all we need. Near-term AI is a potential threat to stability from the CCP's perspective. That's true whether the AI is built in China or somewhere else; American-built AI is still a threat to stability. So presumably the CCP would love for the US to adopt rules limiting new LLMs. If the US comes along and says "we'll block training of big new AIs if you do", the CCP's answer is plausibly "yes definitely that sounds excellent".

And sure, China won't be working much on AI safety. That's fine; the point of an "AI disarmament treaty" is not to get literally everyone working on safety. They don't even need to be motivated by X-risk. If they're willing to commit to not build big new AI, then it doesn't really matter whether they're doing that for the same reasons we want it.

If only we could spread the meme of irresponsible Western powers charging head-first into building AGI without thinking through the consequences and how wise the Chinese regulation is in contrast.

I think it'll be easier in a few years to demonstrate the foolishness of open-source AI.

Since we are talking regulation and/or treaties, both government concerns, is there anything to the idea of working in the context of party slogans (by which I mean tifa [提法])?

What I have in mind is something like this:

• Review the arguments about AGI in English
• Look for conceptual overlap with slogans on various topics (I expect these to be national development, national security, and the socialist conception of historical laws, by way of example).
• See if any can be updated gracefully to talk about AGI specifically.

I think this might work because:

• It will sever AGI from EA/Rationalism
• It will put the AGI arguments in a context more familiar to Chinese officials and party members at least

It will not work if:

• Making obvious adjustments to slogans in order to associate a specific issue with it is culturally nonsensical, or viewed as presumptuous or otherwise low-status
• It literally does not work linguistically in a "For make benefit glorious nation of Kazakstan" sort of way.

That being said, I think it might be a good exercise at least from the English side as a direct attempt at working with some Chinese conceptualization of the motivations for the problem of alignment. A few examples of what I mean, with A being the English idea we want to communicate and B being a modified slogan that points at the same problem:

A) We do not know how to build an AI that knows human values.

B) We do not know how to build an AI with Chinese characteristics.

A) An AGI will be more capable than all of humanity put together.

B) The backwards will be beaten, and all nations will be backwards compared to AGI.

A) Unaligned AGI will cause catastrophe or human extinction by default.

B) Unaligned AGI is a member of the hostile forces by default.

Me and @Roman_Yampolskiy [LW · GW] published a piece on AI xrisk in a Chinese academic newspaper: http://www.cssn.cn/skgz/bwyc/202303/t20230306_5601326.shtml

We were approached after our piece in Time and asked to write for them (we also gave quotes for another provincial newspaper). I have the impression (I've also lived and worked in China) that leading Chinese decision makers and intellectuals (or perhaps their children) read Western news sources like Time, NYTimes, Economist, etc. AI xrisk is currently probably mostly unknown in China, and if stumbled upon people might have trouble believing it (as they have in the west). But if/when we're going to have a real conversation about AI xrisk in the west, I think the information will seep into China as well, and I'm somewhat hopeful that if this happens, it could perhaps prepare China for cooperation to reduce xrisk. In the end, no one wants to die.

Curious about your takes though, I'm of course not Chinese. Thanks for the write-up!

I've heard people be somewhat optimistic about this AI guideline from China. They think that this means Beijing is willing to participate in an AI disarmament treaty due to concerns over AI risk.

I'm curious where you've seen this.  My impression from reading the takes of people working on the governance side of things is that this is mostly being interpreted as a positive sign because it (hopefully) relaxes race dynamics in the US. "Oh, look, we don't even need to try all that hard, no need to rush to the finish line."  I haven't seen anyone serious making a claim that this is downstream of any awareness of x-risk concerns, let alone intent to mitigate them.

Thanks, I'm now more curious about chinese viewpoints on AI and the singularity in general.

Hi! I run a reasonably popular podcast (ChinaTalk) and am also very frustrated with the misconceptions people have around this topic! Pls reach out I’m at jordan@chinatalk.media

Thanks very much for this dose of reality. So maybe a western analogy for the attitude to "AI safety" at Chinese companies, is that at first, it will be comparable to the attitude at Meta. What I mean by this: Microsoft works with OpenAI, and Google works with Anthropic, so they both work with organizations that at least talk about the danger of AI takeover, alongside more mundane concerns. But as far as I can tell, Meta does not officially acknowledge AI takeover as a real risk at all. The closest thing to an official Meta policy on the risk of AI takeover, is Yann LeCun on Twitter, telling Eliezer that it's a fantasy. 

I found an article from the "National Business Daily", from the start of this April, in which a few Chinese academics comment on the petition calling for a 6-month pause on advanced AI. They say things like: GPT-4 just does what humans tell it to do; self-awareness is the crucial threshold, and GPT-4 is nowhere near self-awareness; and even one who says that the human level of intelligence is like "the speed of light" for Turing-machine IQ, a threshold that cannot be passed... Again, I actually find that rather similar to how western researchers react to questions like, can AI surpass us, and what will happen if it does? Unlike more mundane questions of computer science, there's no consensus of verified knowledge that can reliably answer those questions for us, so people just fall back on their individual intuitions and private theories. 

Nonetheless, AI is fast-moving and I think that, as Chinese systems approach the performance of GPT-3 and GPT-4, it will unavoidably dawn on some of the people involved, that this is potentially creating autonomous intelligent beings that can equal or surpass any human; and so they will start to take public positions on what that means, the desirability of it, the inevitability of it, and so on. I can, for example, imagine a vigorous public debate over the creation of artificial persons, with some saying, just don't do it at all, others saying, they should have the rights of a citizen, still others saying, we should enforce Asimov laws and not feel bad about doing so, and so on. 

Relevant: China-related AI safety and governance paths - Career review (80000hours.org)

Do you really think that a scientist is going to walk up to his friend from the Politburo and say "Hey, I know AI is a central priority of ours, but there are a few fringe scientists in the US asking for treaties limiting AI, right as they are doing their hardest to cripple our own AI development. Yes, I believe they are acting in good faith"

This part focused me, what happens if you do this is, your politburo friend answers "that's obviously dangerous propaganda, so I'm going to generate counterarguments against it so that no one entertains it ever again, because we can't afford to have those sorts of doubts around when we're so far behind."

But of course, that reaction is not healthy. An international alignment treaty should be seen as a way for China to convince the US to let them catch up, in a sense, by getting an equal seat at the table, which they're otherwise not going to get. If they want to survive this, they'd do well to note that.

It is far more likely that Beijing sees near-term AI as a potential threat to stability that needs to be addressed with regulation.

This suggests that Beijing is making regulations that are driven by short-term thinking and not careful thinking over longer timelines. 

Generally, my perspective of Chinese decision-making is that government policy is written by people who are able to think over longer timelines. 

To me "AGI has to be aligned with CEV" and the sentence of "AGI has to be aligned with socialist values" sound very similar to me. Anything that's an existential threat to humanity is also an existential threat to the CCP.

Even if they all become law and are strictly enforced, they are simply regulations on AI data usage and training.

If the US would create similar rules and enforce them it would do a lot more than the six-month training pause. The rules prevent the training of large language models based on random internet data and allow only training it on high-quality data. 

If you want to slow down dangerous AI development but not slow down AI alignment work, those policies sound a lot smarter than what the letter calling for the AI training pause calls for. 

Do you really think that a scientist is going to walk up to his friend from the Politburo and say "Hey, I know AI is a central priority of ours, but there are a few fringe scientists in the US asking for treaties limiting AI, right as they are doing their hardest to cripple our own AI development. Yes, I believe they are acting in good faith, they're even promising to not widen the current AI gap they have with us!"

No, but they might say "There's a call for US labs to stop training larger AI models, can we take actions to support that proposal to slow down US AI development? Our analysts at the Cyberspace Administration of China believe that AI could be very dangerous if its development isn't well regulated. Can we push for an international treaty that makes the recommendations of the Cyberspace Administration of China also binding on other countries?"

Let us know when the podcast with you goes up!

When it comes to having an international treaty, whether or not China will agree depends a lot on the content of the treaty.

If the Chinese put their guidelines in practice, they might be open to have a treaty that basically says that all companies have to abide by the guidelines. 

The proposed requirements on training data quality can prevent models like GPT-3 and GPT-4 from being trained. 

Legal liability for anything the model does is also an effective way to slow down AI development.

Old lurker new account! Need to go work soon so very very quick high-level thoughts:

I feel like we feel some of the same frustration at mainstream/western EA, or others alignment speculaters without a deep understanding of China. I can crudely gesture at their wrong inferences sourcing from assumptions orthogonal to the truth, implying fundamental misconceptions in their model, but in the end it comes down to context: much of this is sub-verbal subtle differences that I can succinctly identify without putting in a few months of effort first. Often when I talk to even ABC EAs I want to shout JUST LIVE IN CHINA FOR TWENTY YEARS AND I WON'T NEED TO CONVINCE YOU.

Having said that, I disagree with some of these conclusions: HPMOR is cringe, yes, but also HPMOR is glorious, c'mon. It's polarizing in Chinese, yes, but it's also polarizing in English. Good selector for the target audience. Translating it to build the Alignment field is not exactly zero expected utility but pretty close.

My impression of Chinese EA field-building is that it's basically still severely under-resourced, both funding and talent (esp talent). A team of like 10 people (of various degrees of Chinese background) has been plugging away at it for a few years, but they need much more help. Marginal impact through the roof, for the handful of people that are capable of helping.

我浏览了中国的EA/Rationalist/AI安全论坛

Hi, do you know Concordia AI in Beijing? they are pretty active - 
https://concordia-ai.com/
https://www.zhihu.com/org/an-yuan-ai-6/posts
 

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

Does Anyuan(安远) have a website? I haven't heard of them and am curious. (I've heard of Concordia Consulting https://concordia-consulting.com/ and Tianxia https://www.tian-xia.com/.)

Good post. 

For Westerners looking to get a palatable foothold on the priorities and viewpoints of the CCP (and Xi), I endorse "The Avoidable War" written last year by Kevin Rudd (former Prime Minister of Australia, speaks mandarin, has worked in China, loads of diplomatic experience - imo about as good of a person that exists to interpret Chinese grand strategy and explain it from a Western viewpoint). The book is (imo, for a politician), impressively objective in its analysis.

Some good stuff in there explaining the nature of Chinese cynicism about foreign motivations that echoes some of what is written in this post, but with a bit more historical background and strategic context. 

I did a masters of data science at Tsinghua University in Beijing. Maybe it's a little biased, but I thought they knew their stuff. Very math heavy. At the time (2020), the entire department seemed to think graph networks, with graph based convolutions and attention was the way forward towards advance AI. I still think this is a reasonable thought. No mention of AI safety, though I did not know about the community (or concern) then.

The main thing seems to be money. How do we get more money into alignment in a way that is not detrimental or repulsive to people we want to attract to working there?

I'm trying to think of ideas here. As a recap of what I think the post says:

• China is still very much in the race
• There is very little AI safety activities there, possibly due to a lack of reception of EA ideas.

^let me know if I am understanding correctly.

Some ideas/thoughts:

• it seems to me that many in AI Safety or in other specific "cause areas" are already dissociating from EA, though not much from LW.
• I am not sure if we should expect mainstream adoption of AI Safety ideas (its not really mainstream in the west, nor is EA, or LW).
• It seems like there are some communication issues (the Org looking for funding) that this post can help with
• to me it is super interesting to hear that there is less resistance to the ideas of AI Safety in China. Though I don't want to fully believe that yet. Though, im not sure that the AI Safety field is people bottlenecked right now, it seems we currently don't know what to do with more people really.
• still, it's clear that we need to have a strong field in China. Perhaps less alignment focused, and more governance? Though my impression from your post is that governance is less doable, but maybe I am misunderstanding.

I might have more thoughts later on.

(for context, I am recently involved in governance work for the EU AI Act)

It sounds like you're skeptical about EA field building because most Chinese people find "changing the world" childishly naive and impractical. Do you think object-level x-risk field building is equally doomed?

For example, if 看理想 (an offshoot of publishing house 理想国 that produces audio programs about culture and society) came out with an audio program about x-risks (say, 1-2 episodes about each of nuclear war, climate change, AI safety, biosecurity), I think the audience would be fairly receptive to it. 梁文道, one of the leaders of 看理想, has shared on his podcast 八分 (at the end of episode 114) that a big part of his worldview is “先天下之忧而忧，后天下之乐而乐” (“worry before the people fear something will happen, and be happy only after the people are happy”), a well-known quote describing ideals of Confucian governance, and which has similarities with EA ideals.

In general, I guess I would have expected Chinese people to be pretty receptive to altruism given the emphasis on contributing (贡献) to the greater good in the party line (e.g. studying the good example of Lei Feng), which gets reflected a lot in media/textbooks/etc. But maybe I spend too much time consuming Chinese media and not enough time talking to actual Chinese people.