中国 AI——他们也和我们一样？对话总部位于北京的 Concordia AI CEO Brian Tse

Today I'm excited to share my conversation with Brian Tse, founder and CEO of Concordia AI, a Beijing-based social enterprise working to advance global AI safety and governance. I've been wanting to have this conversation since I first started the show, and it feels especially timely right now.

Recently, at C/urve in Berkeley, I participated in conversations about AI policy and the options we have available as we attempt to safely navigate the development and deployment of powerful AI systems. I was once again struck by how many of these discussions, including with members of technical staff at frontier model developers, are explicitly premised on assumptions like, “China will never slow down its AI development,” or, in some cases, “China doesn't care about AI safety.”

These notions are doing a lot of work in shaping American AI policy and corporate strategy. They're being used to justify everything from aggressive scaling timelines to military partnerships to export controls that cut China off from advanced chips. And yet, I'm not sure how well they hold up to scrutiny.

I often say that AI defies all binaries. Even American politics, which is usually so rigidly polarized and partisan, often has its coalitions scrambled by AI questions. This conversation with Brian suggests that, at least to a first approximation, the US and Chinese AI ecosystems are not opposing forces, but similarly structured communities pursuing remarkably familiar goals.

Both countries have mission-oriented startups using, for now at least, functionally the same architectures and training techniques to relentlessly push the AI capabilities frontier. Both have big tech companies investing huge resources in data center buildouts. Both governments are gradually waking up to the possibility of economic transformation and using industrial policy to secure their supply chains and export controls to gain leverage.

Both societies are concerned about rapid, disruptive change and have safety research institutes like Brian's Concordia AI emerging to study the risks and promote best practices. The differences, on the other hand, are comparatively subtle.

The Chinese AI community, per Brian's telling, is less focused on AGI timelines or achieving some sort of recursive self-improvement loop or intelligence explosion, and more concerned with building practical applications that serve society's needs. This view is supported by the fact that Chinese companies are, in some important ways, much more open than their American counterparts.

In addition to making their model weights freely available to businesses and researchers worldwide, they are publishing all sorts of methods at international conferences, generally in English. And then there's China's recent refusal to purchase NVIDIA's H20 chips, even after the Trump administration made them available. If China were truly in an all-out race to AGI at any cost, as many assume, wouldn't they be buying every chip they could get their hands on?

Of course, none of this rules out a hidden agenda on the Chinese side. But it should at least give us pause before assuming the worst about Chinese intentions. This conversation goes much deeper than that, with Brian offering one of the most thorough overviews of China's AI governance landscape that you'll find anywhere.

We discussed China's pre-deployment testing requirements for consumer-facing AI products, which are notably stronger than any corresponding US requirement, though roughly on par with, or perhaps still a bit behind, what American leaders are doing voluntarily on their own. We also discussed China's national AI safety standards, which explicitly address catastrophic risks, including loss of control.

We discussed the elevation of AI safety to a top national security concern and the growing ecosystem of AI safety researchers across Chinese universities and institutions, who are naturally grappling with the same safety concerns that we are: CBRN risks, loss of control, and alignment. They are developing, you guessed it, broadly similar defense-in-depth solutions. There are links in the show notes to deeply researched reports that Brian, along with the Concordia team and co-authors, have produced that detail all of this activity, again in English.

Overall, to me, this doesn't sound like a country that's AGI-pilled and racing toward superintelligence with reckless abandon. If anything, as you might expect from a country run by engineers and with a living memory of revolutions, it sounds like a government that's taking a more grounded, cautious, sober approach.

Of course, I do want to be very clear about my own limitations here. I've still never even been to China. I'm certainly not claiming any expertise in Chinese culture, politics, or society, and I completely understand why many in the American political establishment are worried about losing a competition for technological leadership.

The stakes are indeed incredibly high. The challenges of building trust between rival great powers are immense, and, as I've said many times, I wouldn't want to live in Xi's China. I value the freedoms of speech and otherwise that I have the opportunity to exercise here.

But the question at hand isn't really whether China's political system is one that we would choose. It's whether China is engaged in a reckless race to AGI that forces American companies and the US government to throw caution to the wind in response. On that question, the evidence Brian presents suggests a very different picture than many in the West assume.

Of course, this conversation won't be the final word on these questions, but I do hope it's an important counterbalance to some dangerous assumptions that are currently shaping policy. Considering Brian's observation that there doesn't seem to be as much fear and paranoia about the US in Chinese AI circles as there is about China in the American AI discourse, at a minimum, I think this report from an AI safety leader on the ground in China should give us the courage to place the burden of proof on anyone who would use claims about China's approach to justify their own apparently reckless plans.

I also hope it inspires much more direct dialogue between the two countries' AI research communities. This is an illuminating conversation about Chinese AI development, safety, and governance with Brian Tse, founder and CEO of Concordia AI.

Hi, Nathan. Thank you so much for having me. I'm a big fan of your podcast.

Thank you very much. Well, that's kind and surprising to hear because we're 12 time zones away. This is a Beijing-to-Detroit conversation happening at 9:00 a.m. for me and 9:00 p.m. for you, so I appreciate you staying up late for me.

I'm really excited about this conversation. As regular listeners will know, I had dreams of getting to China this summer and attending a big AI conference there. There was a satellite event focused on emerging risks from AI, including deception and scheming behaviors. Unfortunately, I had too much going on at home and I wasn't able to make it, so I've been hotly anticipating this conversation as a way to fill in what I missed.

In terms of experiential learning, I'm going to have to get it secondhand from you. One point of view I think we share is that there's little in the world going on today that's more important than the way that the US—or the West broadly, but certainly the US and China—relate to each other on AI issues.

I definitely want to get your perspective from the other side of the world on how that's going and, hopefully, how we can steer the future in a positive direction. To start, do you want to tell us a little bit about yourself? I know your educational background spans East and West, and I'd love to hear a little bit of your personal history.

Absolutely. I started my journey in AI at Tsinghua University and at a deep learning hardware startup, just as the deep learning revolution was taking off in China. That was about 10 years ago. The AlphaGo moment was very exciting, but what really captivated me were the big-picture AI governance questions that it raised.

That curiosity led me to the UK, where I became a policy affiliate with the Centre for the Governance of AI, initially founded at the University of Oxford. I looked into the governance of dual-use technologies in the past and had the incredible opportunity to visit and present my work at Google DeepMind in London.

From there, I also served as a senior adviser to the Partnership on AI and consulted with OpenAI around 2019. I provided external input on the societal implications and release strategies of early large language models like GPT-2, which was just a fascinating experience right in the heart of Silicon Valley.

Then I found my way back to Beijing, where I did some work with the Beijing Academy of Artificial Intelligence, including developing one of the first AI ethics principles from Chinese institutions. It was around that time that I founded Concordia AI, with the goal of bridging these different worlds and building a more cohesive global conversation on AI safety.

I love it. I wish I had spent more time going back and forth myself. So, tell us about Concordia. You're in Beijing—what do you do?

Concordia AI is an independent social enterprise with a clear mission since day 1: to advance global AI safety and governance. Our work is roughly structured around 3 pillars.

First, we drive the development of AI safety standards and policy by participating in the relevant national standard-setting committees and also providing expert consultations to government bodies.

Second, we collaborate directly with some of the leading AI labs and companies, helping them develop safety frameworks and implement safety best practices. For example, we recently co-authored the first comprehensive framework for managing the critical risks from general-purpose AI models, in collaboration with Shanghai AI Lab.

Third, we also foster global dialogue on AI safety. This includes convening forums that bring together some of the top experts from academia, industry, and policy at some of the biggest AI conferences in China and Singapore. We also participate in global policy forums such as the United Nations and global AI summits, and conduct research to help international stakeholders understand AI safety and governance developments in the region.

Is there an organization in the United States, or more broadly in the West, that you would say is the closest analog to your organization? Just to give an intuition or point of reference—depending on who you mention, there's a decent chance we've done an episode with them in the past.

I don't think there is one perfect analog, but there are certainly a lot of civil society organizations that do work on AI safety and governance. What is unique about Concordia is perhaps the Asian perspective on the global landscape, as well as blending work across academia, industry, and policy.

The one that comes to mind, I guess—and maybe this is recency bias, because I just had a conversation and put out an episode with him—is FAR.AI. I just had Adam Gleave on the podcast, and it seems like, certainly in terms of supporting these international convenings, there's some overlap there. From what I understand so far of your overall worldview, there seems to be significant alignment or overlap between you and them.

I don't know if you want to react to that, or if I said, “Concordia is like the FAR.AI of China,” what would you say that misses?

I do think there's a lot of common ground in terms of our mission. I think Concordia is also involved in a lot of national policy and standard-setting work, and I can't speak to whether that's true for some of the other groups.

I think they're moving in that direction as well, although I think they're coming to that a little bit later, after more of a research focus in their earlier years. They're also realizing how important that's going to be, so they're shifting energy and activity in that direction, too.

How about just a general sketch of the AI landscape in China? In the U.S., we have a handful of what I sometimes call “live players.” For me, I define that qualitatively as: Who do I think is really in a position to shape what the future of AI looks like? Depending on exactly where you want to draw that line, there's probably 5 live players in the U.S. Obviously, you'd put OpenAI, Google, and Anthropic in that category. I think xAI has to be included, and probably Meta also has to be included. You could go a little farther and say, “Well, maybe Microsoft,” or add a couple of other organizations to that list as well.

Notably, there's a very high concentration in the Bay Area, with a little bit in the Seattle area as well. It's a geographically dense cluster in the U.S. It's also, broadly speaking, a culturally dense cluster. The people doing this work have known each other for years. They've all read their Eliezer, agree or disagree, and they've all developed a remarkable amount of shared cultural context and history, even before the current AI paradigm really started to take off.

I couldn't do a similar sketch in China, and I don't know to what degree things have developed as expected there or whether there have been surprises. Here, some of the companies I've mentioned are giants that you would definitely have anticipated being in that spot. Google was clearly an AI leader the whole time, so no surprise there. We've got companies like Alibaba in China that are in a similar role: They were massive tech giants with huge resources, naturally investing in AI, and they continue to be leaders.

But then we've also got these DeepSeek- and Kimi-type companies—Kimi is Moonshot AI's model—which, for many in the West, kind of came up by surprise. How would you sketch the Chinese landscape in terms of who the live players are? To what degree has that shaped up in a very expected way? I know there's also a little more geographic distribution, but I'm not sure how you would describe that. I'm also not sure whether you were surprised by any of these other companies hitting the top tier in China, or whether, from the inside view, that was more expected.

China has multiple hubs for AI development and startups. You have Beijing, which is home to the big companies like Baidu and ByteDance. Moonshot AI and Zhipu AI were founded by alumni of Tsinghua University, which has a very high density of AI researchers and entrepreneurs.

You have Shanghai, which is home to MiniMax and other very strong LLM startups with a focus on multimodality. Shanghai is also hosting the World Artificial Intelligence Conference, which draws a lot of attention across the investment and industry ecosystem. Then you have Hangzhou, where DeepSeek and Alibaba have their headquarters.

Actually, in 2025, there was a term that got popularized: “the 6 little dragons of Hangzhou.” That included DeepSeek, as well as 2 robotics companies like Unitree. Then you have Shenzhen, in the south of China, which has major players like Huawei and Tencent.

I think that sort of geographical distribution also applies to AI safety. In our report on the state of AI safety in China, we documented around 31 groups in China that have published papers on AI safety. The top 5 cities are Beijing, Shanghai, Hangzhou, Shenzhen, and Hong Kong.

Is there any cultural constant across these hubs? Again, with the Bay Area, it's definitely a distinct culture. I won't try to fully characterize it in 30 seconds, but it's clear that there is a science-fiction-inspired intellectual tradition and a lot of shared cultural touchstones that span the companies. Even as they're competing fiercely with each other in the market and for research breakthroughs, they certainly have some commonalities.

Is there anything like that in China? How would you describe the cultural foundation of AI work in China?

I think one notable difference is that across all of these Chinese cities, the tech ecosystem and the policy ecosystem are quite intertwined. Whereas in the U.S., my sense is that there's a pretty big difference between Silicon Valley culture and the culture in D.C. Perhaps there are just more coherent views on policy questions across these Chinese AI hubs.

How about where they are taking us—their vision for the future? One of my common refrains is that the scarcest resource is a positive vision for the future. I'm always struck by what we hear from the leaders of AI companies in the United States. It's usually very much like, “It's going to be so amazing. We're going to solve all the problems and cure all the diseases.” It rarely goes much deeper than that.

We don't hear too much about what life is supposed to be like when the AIs get as good as these leaders say they're going to get. I feel like that's quite a notable gap. There is this sort of science-fiction-inspired but very vague sketch, and then it's just, “We have to develop this technology. Trust us, it's going to be amazing.”

Is there a similar lack of concrete vision in China? You also see survey results that suggest Chinese people are more optimistic about AI than Americans. I wonder if there's a more concrete vision that the government and companies together are offering the public, or if this is just a dispositional thing. The last 25 years in China have gone comparatively really well, so maybe people are just more optimistic in general.

How would you describe the sense that exists in China about what's going to be good about this and why people are relatively optimistic?

That sounds like a great question. Maybe let's start with Chinese society in general, and then I can comment on the perspective of Chinese policymakers.

In our reports, we've looked at some of the existing public opinion surveys on AI in China. The limited samples suggest that the Chinese public generally view the benefits from AI as outweighing the harms. One survey after ChatGPT suggested that the Chinese public do think there could be existential risks from AGI, but still think that AGI should be developed because they think the risks are largely controllable.

Another recent survey focused on Chinese students and also found that they are pretty optimistic about the benefits of AI outweighing the potential harms. Around 80% of respondents agreed that AI would do more good than harm for society.

To put this into a broader context, this is not very surprising if you consider the recent past. Let's take my parents, for example. They were born around the 1960s in a southern province of mainland China, in Fujian. Within their lifetime, the country's per capita GDP has increased by more than 140 times, and the rate of extreme poverty has decreased from approximately 88% in the early 1980s to close to zero today.

I think when you have witnessed that level of transformative progress firsthand, or through your parents, you tend to have a more optimistic view of what technology, as well as proper governance, could deliver.

Yeah, that's incredible. Obviously, it started from a low point after some bad times, but still, it's an incredible rebound. In many ways, you look at Chinese cities today, and it's hard to argue that they're not world-leading in many respects.

I guess maybe just one more question on the culture of AI development in China. Would you say that there is a single culture of AI development in China, or because it's spread across these different regional hubs and different companies, would it be overly simplistic to say that there is a culture of AI development in China?

I think there are different approaches to thinking about frontier AI, but it's not necessarily geographically based. The mainstream approach in the Chinese commercial AI market is scaling large foundation models and deploying them. They follow developments in the West closely, particularly after the success of GPT-3.

The Chinese AI ecosystem is rapidly advancing in areas that your audience would be familiar with: increasingly multimodal and capable AI, advanced reasoning, increasingly capable AI agents, and scaling across training, post-training, and inference-time stages.

Alongside this mainstream trend, there are also a couple of directions that receive significant emphasis in China. There are prominent AI policy documents and scientists who believe embodied AI—training and deploying AI in the physical world—is the most important approach for achieving powerful AI in the future.

There is an institute called the Beijing Institute for General Artificial Intelligence that focuses on developing AI that can reason and plan toward complex goals with minimal initial input. The director framed this approach as a “small data, big task” approach, in contrast to the “big data” path for training large-scale foundation models. So, yes, I do think there are different flavors of thinking about frontier AI.

Yeah, both of those definitely have resonance, or analogues, in Western AI culture as well. “Culture” is maybe not the right word, but certainly there is a school of thought that embodied AI is going to be critical. There are also things like the ARC-AGI prize, and people are very focused on getting AI systems to adapt to new tasks and become more sample-efficient.

Overall, it seems more similar than different, which I think is a big theme of all the reading of your reports and all the preparation I've done for this conversation. It just kept coming back again and again: this sounds a lot more similar than different.

How do you think Chinese AI companies look at the West these days? Maybe we should speak about companies, or about the intersection of companies and government, which are more intertwined in China than in the US.

In the West, we hear all the time—and it's very frustrating to me—so many conversations about what we should do, what we can do, and what the option set looks like ending with, “China's going to do this. China's going to do that. It's inevitable. We can't stop them, so we've got to do it ourselves.”

This adversarial relationship, and the race to AGI between the 2 civilizations, is taken as a given. Do you hear that coming from the Chinese side as well? Is that also something like a mirror image? If not, what is the narrative in terms of how Chinese AI relates to Western AI that you hear there?

One thing that has significantly less discussion among Chinese companies and policymakers is specific AGI timelines or a clear finish line for AGI. I think this contrasts with the public discourse in the West, where speculation about when AGI might be achieved is very prominent.

I think Chinese entrepreneurs have a stronger focus on achieving technological self-sufficiency, getting very useful and practical applications, being profitable, and enjoying commercial success. There is also some idealistic goal of open-sourcing AI to benefit everyone, especially countries and communities that lack AI capacity and access.

Yeah, interesting. I wonder what that comes from. One would be tempted to say that the AGI notion of this super-powerful thing that can run the world, or even become a singleton or something, seems to echo Western religion in some ways—the single God, Abrahamic concept that is very familiar here. Maybe that just isn't such a background force in China, and so that whole notion resonates less.

You say there's less emphasis on a finish line, less emphasis on AGI, and less emphasis on timelines. Is there any—what the Western story is very confusing, right? It's sort of like we're going to get over this event horizon of AGI, and I just heard the other day that Sam Altman had been asked, “How will you know when you've got AGI?” An answer he offered was basically, “When the AI researchers are using more of our research compute than the human researchers.” That's how we'll know that we've tipped over into AGI, because the AIs will be, if they deserve more of our compute allocation than the human researchers in some clear sense, at a tipping point.

We have this very weird story where we sort of have this event horizon beyond which we can't see. Is there something similar in the Chinese narrative, or is it just that things will hopefully get better and daily life will get better? I'm still struggling to fill what feels like a blank space in the Chinese imagination around the big-picture future of AI, if anything.

Mhm. Yeah. Let's talk about the recent high-level directive on AI Plus released by the Chinese central government. This is a pretty big deal that represents a comprehensive blueprint for how China plans to develop and deploy AI domestically, and it was released in August 2025. Overall, the goal is for AI to serve as a new economic engine—a general-purpose technology similar to the internet in the past few decades or electricity during the Industrial Revolution. It is to be fully integrated into the whole economy.

The plan has 6 pillars, with the first being AI as a tool for scientific discovery and innovation. I think this is listed first because Beijing sees it as a force multiplier for everything else. I was at an AI for science forum in Shanghai last year, and even then it was clear to scientists, policymakers, and entrepreneurs that it's really inspiring to use AI to solve scientific challenges, with examples like AlphaFold and others. Interestingly, the Chinese document takes this a step further, calling for the use of AI to advance even the social sciences and philosophy.

Then there are other domains, like AI as a tool for industrial transformation. Think about fully autonomous factories, smart logistics, and the use of AI-powered drones and robotics in areas like agriculture. Third, there is AI as a consumption booster, using AI to create new consumer experiences, even including things like the metaverse and brain-computer interfaces.

Then there is AI as a human collaborator as the fourth area, and a key theme is to use AI to support human labor, not fully replace it. It's about helping workers collaborate more closely with machines and leveraging AI agents as copilots to boost productivity. Then there is AI as a tool for government efficiency and proper governance.

Finally, I believe the sixth area is using AI for international cooperation as well. The leadership of China views AI as an international public good that should benefit humanity, while also making open-source tools and models widely available, particularly for the Global South. Notably, throughout this comprehensive national blueprint for AI, it doesn't mention AGI, superintelligence, or any other such concept.

It seems to me that the real AI race for China isn't about beating the US to AGI supremacy. It's about prioritizing the integration of AI into, and also boosting, the real economy in the coming years and decades.

Yeah, that's good to know. That's a more grounded conception. I'm remembering one article—I can find the link—but it dates back a few years now, probably 5-plus years. It was in The Washington Post, and it was from a Chinese government official. Basically, the idea was that with AI, we might be entering a world where central planning can really work. It was making the case, interestingly, to a Western audience in English.

I have some questions about that phenomenon in general because it isn't something I take for granted. I think we in the United States, broadly and culturally, take for granted that Chinese researchers are going to the trouble to publish their work in English and make it really quite readable for us, which is something we probably shouldn't take for granted. But we'll circle back to that in a second.

The op-ed was basically saying that markets have been great, but they're undirected. They were the best thing we could have had, but now, with really powerful AI that can crunch all this data, there is an opportunity for central planning and government direction of society to really work. That was a few years ago, and it was just one op-ed. I don't really know how big of a deal it was, although it did come from someone who was official enough that it was not nothing.

I didn't hear that in your characterization just now. It did not sound like there was this idea that with AI, the government will finally be able to get it all right. Is there any impulse like that that you see, where AI is supposed to be a magic solution that will make state planning and centralized economic direction really work in a way that it hasn't historically?

I think that sounds a little bit utopian for Chinese policy discourse, so no. I don't think people are betting on using AI to solve everything, including central planning of society. On the margin, I think when I talk about using AI for proper governance and government efficiency, there are examples of using AI to create better early-warning systems for, for example, natural disasters. That means getting real-time data early enough for the government to act and prepare for potential crises.

So, on the margin, yes, there are valuable applications, but probably not as a unifying silver bullet to societal issues.

Yeah, interesting. I'll have to dig up that link. I just Googled it and couldn't find it immediately, but I can track it down.

How about on this English thing? You mentioned self-sufficiency. One worry that I have broadly is that, right now, it does seem like a very fortunate situation we're in. For as much decoupling as there has been—and the trend obviously seems to be continuing in that direction—the AIs coming out of the 2 countries are much more similar than different, right? We've got language models, the attention mechanism, large-scale pretraining and post-training, and now reasoning and multimodality. It's very much following the same path of development.

That’s again something I don’t take for granted. How do you think the Chinese research and development ecosystem broadly thinks about that? Do they want to be doing the same thing as what’s happening in the West? Why are they publishing in English? Why not just do all this stuff in Chinese?

Do you think there could be a divergence where, because of whatever cultural factors—or perhaps, I worry about this sometimes in the context of chips, if the chips become too different or availability becomes too different—that could be a forcing function that takes R&D efforts in somewhat different directions? There’s a lot there, so sound off on it, if you would, please.

Yes, I think for several decades, especially since the deep learning revolution around 2012, Chinese and Western AI researchers have been part of the same ecosystem. They attend and present at the same top-tier machine learning conferences, like NeurIPS and ICML. They publish papers on arXiv to share findings quickly.

Publishing at top-tier international conferences is seen as more prestigious than publishing in Chinese-language journals, and I would say the communities have been deeply integrated. I remember seeing a study showing that, for the UK, the most common partner for AI papers is actually China, not the US. Another study in Nature in 2024 analyzed over 5 million AI papers, and they found that collaboration between researchers in China and the US produced more impactful and novel AI research than when either country works alone.

It is deeply integrated and collaborative, and this practice isn’t unique to AI. English serves as the lingua franca for most modern scientific fields. There is obviously a deeper and darker historical reason for this. We have lived in a world dominated by the English language for the past 2 centuries, largely a product of the Western-led Industrial Revolution and the era of colonialism. I think that has created a kind of cultural gravity as well.

I’m happy to talk about the tech stack later on, but on the language point, I do think it’s something that we shouldn’t be taking for granted.

Yeah. Are there Chinese versions of all these papers as well, or are they literally just putting them out in English as the canonical version?

I think for most of the papers, it would just be in English, and then there could be summaries of these papers in blog posts in Chinese.

Yeah. Fascinating. How strong is the English on these teams? If I go to DeepSeek and show up not speaking more than 2 words of Chinese, can I expect to have a comfortable conversation in English?

I thought the DeepSeek-R1 paper—in addition to all the other things that have been discussed, and this has been true of the Kimi papers, too—was just very well written. Clear, compelling, and better written than the typical paper that I read from native English-speaking authors.

Is that something where English is like that? Do the rank-and-file researchers at these companies have that level of command of English, or is it a specialized skill within the company? If I showed up at the lunchroom, would I be able to speak very comfortably in English to everyone? What does the English production process look like at these companies?

The vast majority of people at these companies would be able to understand English, both in terms of hearing and reading. I think a good number of them would also be able to have a conversation with you, particularly among the scientists. But obviously, that’s not the case for everyone, especially at companies that have hired more from local universities.

Interesting. Okay. How about if we go outside the companies and just look at the public for a minute? Obviously, there’s been this whole chip wrangling. I wonder, has that impacted the public’s access to AI at all?

If I’m just an average Chinese computer or internet user, do I feel like there’s any scarcity of access to AI, or is it similar to being in the US, where I have 4 or 5 different products I can choose from? They all have a free version and a relatively affordable paid version, and I can basically get all the AI I want at a not-crazy monthly fee.

What does access look like at the retail level in China? How would you describe how much of an everyday thing it has become for normal people who aren’t part of the AI community itself?

I think it’s pretty similar and pretty accessible, with people able to download multiple chatbots and video-generation apps on their phones. Many of them are free, so I would say it’s pretty similar and accessible.

And these are cloud services, too, right? When you say “download,” obviously you download the app, but you’re not generally downloading a model and running inference locally on your device. The inference model is similar to what we experience in the West, right? The models are being run in the cloud, and you’re getting streamed tokens. That’s the typical pattern.

I know I’ve asked this before, and it’s a little less obvious in China because so many of the models have been open-sourced, but I’m assuming the delivery mechanism is still basically via the cloud.

For the everyday consumer, you would download the applications through the cloud or just use the websites in your browser. Obviously, open weights are also very much a thing for developers, so all of these options exist.

Okay, cool. Let’s talk about your op-ed in Time. It’s been fascinating to see Time magazine become a flagship outlet for AI policy thought leadership.

Before getting into the specifics of your op-ed—which is basically making the case that China is taking AI safety seriously, and that, by extension, we should be less confident in the “China’s going to do it, so what else can we say?” get-out-of-jail-free card that happens in so many American conversations—how did you actually get connected with Time magazine? What was your experience of publishing an op-ed in Time magazine from China?

We had the report “State of AI Safety in China,” and we wanted to communicate the message that there is a lot of activity and nuance going on in the landscape. We also wanted to connect with an international audience, especially on recommendations for AI policy.

We wrote up the op-ed and pitched it to the editor at Time magazine, which has been quite interested in AI safety as a topic over the last few years. I think that was a very straightforward process. We appreciate how professional and open they are in terms of these topics, so it was a really good experience.

Cool. I’m hoping to get Marc Benioff on the show. He owns Time magazine now. It’s definitely something I did not have on my AI bingo card: Eliezer Yudkowsky publishing there, and you publishing there. It’s quite an impressive reinvention of an older media institution, I would say.

Okay, take us through the argument that China is taking AI safety seriously. You can walk me through it probably better than I can guide you through it. What are the key reasons to believe this?

AI safety has been elevated to a top national security and public safety concern over the last 1 to 2 years. One of the most important political meetings in China is the 3rd plenum, which happened in 2024, where the leadership, including President Xi Jinping, called for an oversight system to ensure the safety of AI. That meeting classified AI safety as a major public safety concern.

Following that meeting, in February this year, China also published an update to the national emergency response plan. The new version includes AI risk alongside concerns such as cybersecurity, biological security, and natural disasters. That suggests AI is not viewed simply as a content-control issue, as some might think.

There is also a study session on AI by the Chinese leadership, involving the top 24 officials in the Politburo. This is a channel for senior policymakers to be informed by experts in different fields, including AI. In the session in April this year, the Chinese president noted that AI brings unprecedented development opportunities but could also bring unprecedented risks and challenges.

If we compare the meeting readout with previous high-level statements by the government, it was much more detailed on safety. It described distinct stages of AI risk management, from technology monitoring to early warning and then to emergency response. I would start with this very high-level national prioritization of AI safety that we have seen in the last year.

Could you give a little more context on how these things should be understood? What I mean is that, when the Politburo has a study group, you could at least superficially look at a similar meeting at the White House, where people talked about AI and various things were said. You could say, “That seems like it’s happening pretty much the same way in both countries.” And maybe that’s right.

Maybe that is the right way to understand it. But I at least have some sense that there's something more official, meaningful, and substantive when a Chinese body does something like this versus when it happens in the United States. I think a lot of these things in the United States are like, “Yeah, well, we had that one meeting at the White House, and we said a few things, and then we moved on.” In a couple of cases, we got voluntary commitments. Mostly, they've been honored by the companies.

But at the governmental level, we've totally turned over the government, and there's not a lot of continuity. Just because a meeting was held doesn't mean anything is going to happen downstream of that. So how would you characterize what these meetings or statements really mean, so to speak, in the Chinese context?

Absolutely. One of the key functions of these statements and study sessions is to signal the priorities of the leadership and of the country to a domestic audience, right? Local officials would study them to understand the priorities, and other stakeholders in society, including investors and private companies, would also pay close attention. This allows the government to mobilize significant resources toward specific goals, creating a whole-of-society approach to achieve these policy objectives.

Of course, this type of top-down mobilization doesn't guarantee a perfectly coordinated process. Actors might respond dynamically and might have different interpretations. But I do think there are some differences in AI policy between the United States and China.

The first major difference is continuity. In China, there's strong, long-term continuity in national plans, including for AI. Once the central leadership sets a direction, it could be a multiyear or even multidecade commitment that will be carried out in the government and across society. Whereas in the United States, it seems like a different story, with an executive order on AI by one administration that might not survive the next administration.

The second difference is coherence. In China, the Politburo sets the overall tone, and then there's a more unified approach to implementing policy. Whereas in the United States, you have different opinions between the White House, Congress, and other stakeholders. So I think in terms of the strengths of the Chinese policymaking system, it allows for a more unified focus on longer-term planning.

In terms of the AI safety culture differences between China and the United States, I have a number of observations. The first one is that China appears more willing to consider regulations. Perhaps that comes from a higher level of trust by the public toward the government.

Second, we also see that academics have a really significant influence on policymaking in China. For example, several Chinese academics who are deeply concerned about the potential catastrophic risks from AI have briefed the Politburo leadership directly through either study sessions or other channels. I think that is partly due to a historical tradition where, for almost 2,000 years, scholars have held almost the highest status in Chinese society—more prestigious than entrepreneurs and businesspeople.

It also means that China might be less susceptible to capture by commercial interests and tech lobbying than what we might see in some other countries. Finally, it also seems to me that the AI safety discourse in China is less polarizing, for lack of a better word. Looking at the headlines, it seems like in the United States, the views on AI safety have a huge spectrum, with people calling for an immediate pause on AI development on one end and effective accelerationists who want to speed up at all costs on the other. I think the discourse in China is a bit more in the middle.

Yeah, that's interesting. How about the question of accepting performance compromises for the sake of safety? This is something that I feel like we might really need to be willing to do longer term. We could maybe have a little more powerful AI, or maybe it could just respond faster or whatever, if we didn't have certain guardrails in place.

We haven't seen too much willingness to do that in the United States yet. Arguably, it hasn't really been needed yet. But I wonder if you could characterize the attitude: Are these things seen as something that people are quite ready and willing to trade off against each other?

I think in China, overall, people do not view capabilities and safety as zero-sum. It's possible to increase the effort in both directions. For example, one quote I remember is that Chinese Vice Premier Ding Xuexiang went to the World Economic Forum in Davos in early 2024, and he gave a metaphor: If the braking system isn't under control, you can't really step on the accelerator with confidence.

For context, he is the most senior official in charge of science and technology planning in China. At least conceptually, across all of the Chinese policy documents that we have analyzed, we always view safety and capabilities as going hand in hand, and not necessarily as a zero-sum dynamic.

Yeah, there are certainly many ways in which that has proven to be true over time, and maybe it's the healthier attitude in general versus viewing them as being at odds. There probably will be some areas where they are at odds. One very simple one is that if you really want to control sensitive systems, you might want to put a filter on outputs.

Now you have this question of, should I stream the outputs token by token and run the filter in parallel with that, or should I wait for the generation to be done, do a classification, and then return the result? There are some areas where you will have, if not necessarily a power trade-off, at least a user-experience trade-off against safety and control.

But maybe I shouldn't emphasize those too much, because it is probably easy for people to fall into the frame that it's a trade-off, when in fact, in many, many situations, better control allows you to get better use of AI, and it's all quite to the good. This is the 45-degree line too, right? I really like the metaphor. I'm not a big analogy guy in general, but I do like the metaphor of, “You can't go fast without good brakes.” Is the 45-degree line basically the same idea, or would you unpack that one slightly differently?

I think it's basically the same idea: We need to advance safety and control alongside the development of AI capabilities. For example, if we have more dangerous AI capabilities in specific domains, then we should also have stronger safeguards at the same time.

One really interesting characterization that I heard of the difference—of course, there are many differences, but one particularly relevant difference—between the Chinese and American systems came from the Dwarkesh Podcast. It described the role of mayors and governors and the incentives that mayors and governors have.

In the United States, they are obviously elected by local populations, so they're accountable to local populations, and their incentive is to be popular with the local population. In the Chinese system, as it was described, the incentive is to do a good job according to the priorities that the central leadership has set. There's still a lot of opportunity to be inventive and come up with good local solutions, but the way you get promoted, become a higher-ranking official, and advance in your career is not by being exactly popular with the local population. It's by effectively hitting the objectives that the national leadership has set.

I guess, first of all, I would wonder: Do you agree with that assessment of the incentives that people have? And if that is true, are we starting to see mayors or governors do things locally and entrepreneurially to try to advance the AI safety priorities that the central leadership has set? Are there any examples of people who have come up with cool new ideas that are bubbling up in response to these nationally prescribed priorities?

That's a good question. There is the concept of the mayor economy in China, and for a while, local and provincial governors were very much focused on optimizing GDP and economic performance. They were building a lot of infrastructure and real estate, boosting investment in those local areas, and then there was an increasing level of environmental concern over the last decade.

In the assessment of those governors' performance, environmental goals and metrics were added, and that changed a lot. For example, it improved the air quality in Beijing. I think that is an example where you need to identify and improve the local incentives in order to achieve national policy.

For AI specifically, over the last 2 years, we have seen a number of provincial AI plans, from Shenzhen to Shanghai and other areas. I would say one prominent example is creating experimental zones for self-driving cars to look at different levels of automation and how they could be properly regulated.

In general, there's a tradition of doing experiments locally before institutionalizing them at the national level. We saw that with experimental zones in the era of economic reform, with Shenzhen being a prominent example, and I think we're also seeing the same strategy being played out for some of these AI implementations.

How—this is a bit of a digression—but how would you describe the state of self-driving vehicles in China? Have you had a chance to ride in any? How would they compare to a Waymo that people might have experienced in San Francisco?

I think the state of the art in China is pretty comparable to Waymo.

I have seen and experienced some of the self-driving cars in Baidu with their Apollo project. I think it's getting pretty good. I think one of the main bottlenecks is regulation. With self-driving cars, it can't be just a little bit safer than humans, right? You want them to be maybe 10 times safer for society and the public to feel comfortable with it.

Yeah, that's funny. That's almost exactly the same as what it seems like we have in the United States, too. I'm always marveling at the fact that even as we're now hitting literally 10 times safer, people are still kind of not comfortable with it, and we're unsure if we should really adopt it or whatever. But boy, the latest data from Waymo is like—

Really, really hard to read any other way, in my view, other than that we really should make this a national priority.

We'll see if that happens. How about on the AI safety dimension specifically, though? Is there anything from mayors or, you know, more local or regional officials that you would highlight as interesting? Maybe it's just too early to have anything there. Obviously, this is all developing very quickly.

One national piece of regulation that came out in 2025 was around labeling AI-generated outputs. Starting from September this year, all developers and social media platforms are required to have both explicit labels and implicit metadata for AI-generated outputs, ranging from text, audio, and video to even simulated environments. I've seen Shanghai trying to pioneer a consortium of AI companies like MiniMax, but also social media platforms like RedNote, to make sure that there is a unified and functional standard around watermarking and content provenance.

Interesting. One worry that we have in the United States is that we're going to get a bunch of state-level rules that are at odds with each other and contradictory in different ways. There are obviously just 50 states, so there's a lot of possibility for all kinds of small differences that create a lot of overhead and headache for AI developers.

A lot of people are saying, “Well, we need to have a national standard.” The federal government doesn't seem too inclined to create one, so currently we kind of have nothing. There are a few state rules popping up, and people are sort of worried about this chaos. What does the usual trajectory look like in China for harmonizing those kinds of ideas as they mature? If you have multiple different regional initiatives trying to figure things out, does the central government come in at some point and say, “Okay, you guys got the best version of this, so we're just going to make that the standard”? Or do those sorts of things continue to persist in different regions for a long time? I'm wondering if there's a similar issue in China that Chinese AI companies would have to contend with going forward.

So far, AI regulations in China are mostly at the national level. You have specific regulations for recommendation systems, deep synthesis, and generative AI. There are also AI ethics review regulations. At least so far, it doesn't seem like a major issue for China, and I can speak to those specific regulations if there's interest.

Yeah, that's a perfect segue. I was just going to ask next: how does this all cash out for Chinese AI companies today? In the United States, basically, you can just launch a new model. We've seen what was, in my view, the craziest example of that when xAI launched Grok 4 at the exact same time that Grok 3 was online identifying itself as MechaHitler. They basically went online and said, “Here's Grok 4. Here's all the great things about it.” They didn't even comment on their Grok 3 identifying as Hitler.

Obviously, other companies are taking a much more responsible and deliberate approach, but xAI at least shows that in the United States today, you can take something screaming hot straight off the GPUs and throw it into the public with basically nobody to tell you that you can and no requirements that you do any meaningful testing. You can just test it in prod, so to speak, as they say.

What's that look like in China? Is there a structured process that everybody has to go through? I've heard that in the past people have told me, “Well, they sort of let people launch, but if you mess something up, they come down on you hard.” So it's more of an incentive to stay on the right side of things. How is that shaping up today?

If you are a deployer of a publicly available chatbot, then you have to preregister with the government. The process is quite involved. In particular, it requires developers to do safety testing for various risks. There is a national standard with 31 risks that you have to test for, and the Chinese regulators—specifically, the Cyberspace Administration of China—will be given pre-deployment access to the model. They then have to approve it before it can be released to the public.

Now, over 500 such systems and different versions of models have been filed over the last 2 years. I think this basically introduces a filing and registration process that looks like a licensing regime. The Interim Measures for the Management of Generative Artificial Intelligence Services, I think, is one of the most important regulations for governing general-purpose AI today.

Interesting. That sounds like that's changed. Maybe it was a year ago that I did my last episode with someone who was on the ground in China. Do I have that timeline right? Has that changed within the last year, such that the Chinese system has gone from “You can kind of go ahead and do your thing, but we're watching you” to “Now you have to do these 31 different dimensions, preregister, and give early access”?

The system of registration and pre-deployment testing already existed in 2024. It was being developed a few months after the release of ChatGPT and as Baidu and other Chinese companies started developing their own large models. So it's not new.

Okay. Are there explicit thresholds, and is all this data public? If I've got 500 companies that have been through this 31-dimensional analysis, is there a place where I can go and look at their scores on all those different dimensions? Are there rates of—because obviously nobody's been able to drive all the unwanted behavior to zero, right?—are there explicit thresholds that you have to stay below on a particular test?

Is there also a red-teaming component? One worry that people would have if the standards became too well known is that they would become very easy to game. You could say, “Oh, yeah, we could definitely hit that. We could stay under those thresholds.” That doesn't necessarily mean your AI is broadly going to be well behaved or under control. How are they dealing with that? On the one hand, you might want transparency; on the other hand, you might want to have some tests that are not disclosed or are more dynamic, because you don't want people to overly game those finite test sets.

No, that's a good point. First of all, there is a publicly available database of the models being registered, so people can check it out. In terms of the threshold, there are specific numerical targets. For example, there is a set of questions that the model would have to be queried on, and at least 96% of the time, the output should be considered acceptable by the authorities.

96%?

96%, yeah. There is also a red-teaming component. Apart from internal testing by the companies, companies would also need to create test accounts for the Cyberspace Administration authorities in their local provinces, basically granting them access to test the model pre-deployment. The process could involve multiple rounds of feedback, review, fine-tuning, and then conversation until the regulators are satisfied with how the model behaves.

That 96% is quite interesting. Obviously, the flip side of that is that it's up to a 4% rate of responses on sensitive topics that would be allowed, even though the outputs would not be pleasing to the government. That strikes me as remarkably high in some sense. We don't have anything like that in the United States at all.

But I guess when I think about what I think I know about the Chinese internet, and at what rate you can say something online that is out of line with what the government wants to permit in the public sphere and have that stay up on the internet, my sense is that it's not 4%. If you went online and sent something out of line like that, you would expect it to be taken down at a significantly higher rate than 96%. Is that sort of a concession, in some sense, to the inherent unwieldiness of the technology, or am I maybe wrong in my background assumptions? How would you help me understand that better?

In the initial draft of the regulations for generative AI, there was some language around ensuring that the outputs of large language models are very reliable and truthful, almost to a perfect degree. Then there was a window of 3 months in which the industry and the wider public were allowed to provide feedback. I think there were some concerns in the industry that ensuring the outputs of large language models were perfectly truthful and reliable was just technically infeasible, and so the final piece of the regulation was relaxed a little bit.

So I do think there is some trade-off between this policy goal and the inherent limitations of the technology.

But then I suppose there’s another layer, perhaps, of governance, where it’s one thing if the chatbot says something to me that is out of line with what the government would want it to say. A separate question, in some sense, is whether I then post that on the internet or share it with other people. So I guess there are multiple layers of controlling the information flow, and the model outputs are only one layer, right? Ultimately, that is operative to make sure people are engaging in the way the government wants them to be engaging—sharing and seeing the sorts of information that the government wants them to be sharing and seeing.

Yeah, I think so. To give a concrete example, the government of Beijing passed a regulation for AI chatbots in the medical setting. Hospitals and medical institutions are not allowed to use AI chatbots to produce medical prescriptions without a human in the loop. I think that’s very sensible, given the rate of hallucinations from AI. So even if AI chatbots produce bad and unreliable medical recommendations, the institutions cannot use them, right? That’s another layer of governance.

Yeah, interesting. It’s a good reminder, too, that there are a lot of practical aspects to this stuff that are not purely about politics or the historical memory of famous events or whatever. People in the United States tend to frame these questions in terms of the canonical things that people ask, and I’m just phrasing my own description of it sensitively, given the fact that you’re there and I’m here.

It’s useful to highlight that medical chatbots are probably much more of what people are going to be getting day-to-day value from, and could also potentially pose real risks, as opposed to what happened at a particular historical episode or whatever.

How far down the scale of entrepreneurial activity does this registration exist? Here, I can take a model and spin it up into an app. Obviously, I can go publish it, just as I can create whatever random experience I want to create. If I’m a very small-time app developer in China and I go grab a Qwen model, a DeepSeek model, or whatever, and I want to make a new product experience with it, do I have any sort of fast lane to get that stuff online and test it out, or do I have to go through the same governmental approval process?

Could I say, “Well, look, DeepSeek already did this, and I’m just a wrapper app around it. Therefore, I don’t have to go through all this pain”? Or do I still have to go through all the same steps as the bigger companies do?

If you are only deploying the AI model for internal research purposes or only in a business-to-business setting, then the same set of regulations would not apply. But as long as you are doing a chatbot or a recommendation system that is broadly available, then a series of binding regulations would apply.

Yeah. Okay, interesting. Are there things like AI friends in the Chinese app ecosystem? In the United States, we have a growing category of AI friends, AI boyfriends, and AI girlfriends. Sometimes there’s now explicitly an AI therapist, and by and large these things are just rolled out to see what happens.

I have no idea if something like an AI friend or an AI girlfriend would even be approved in the Chinese context, or if there are other restrictions. I remember that there was a crackdown on video games where—correct me if I’m wrong—I believe at a national level it was like, “Video games will only be played during these hours going forward. Focus on your studies, kids.”

How much of an editorial role does the Chinese government play in terms of what kinds of AI experiences are going to be allowed? Do we have AI boyfriends and AI girlfriends in China today?

I’m not too familiar, but there are certainly apps for AI companions—being friends, being collaborators, and also different characters. I have seen an app store, a virtual app store, where you can pick different characters and choose their different personality traits. So that definitely exists.

I think you’re right that Tencent and other gaming platforms would have to control the amount of time that kids and teenagers could play on an everyday basis, to focus on their studies and the real world. There is certainly that sense of paternalism from Chinese society and the government.

How about at the training level? To the degree that we’ve seen attempts at regulation, I guess we’ve seen everything, at least at the state level in the United States. But some of the things that have gotten the most heat in terms of debate are at the training level. Before you go and do some big training, you have to tell the government you’re going to, or you have to commit to some sort of testing.

That’s quite distinct from when your model is done and you then have to register it or prove that you’ve hit certain thresholds. Is there any sort of pre-training approval process happening on the Chinese side?

There are national recommendations on different stages of the AI R&D pipeline. For example, there is a national standards-setting body called TC260 that recently published the AI Safety Governance Framework. They published it last year, but they have updated it to Version 2.0, and I think this is very relevant to some of the conversation around frontier AI safety.

The document highlights concerns around catastrophic risk, including a lot of focus on loss of control and also misuse in dual-use domains like biological weapons and cyberattacks. In terms of risk prevention, it highlights measures such as data curation and filtering in the pre-training phase, especially removing some of the hazardous knowledge that might be relevant for CBRN risk. But as far as I know, this is not part of binding regulations yet.

Gotcha. Okay. I guess how would we try to summarize all this? In many ways, it seems like there are more similarities than differences. Obviously, the core technology that’s being developed thus far looks very similar, which means it has all the same strengths and weaknesses.

The list of concerns is also similar. Reading through the safety framework that you helped author, I was very struck by the fact that if you took the names off it, I would not be very confident about where this came from in the world. The taxonomies of risk, and the way of understanding and conceptualizing them, seem much more similar than different.

I’m not sure if that quite extends to loss of control. Certainly, with things like biosecurity, CBRN-type risks, and cybersecurity, it seems much more similar. Does the Chinese side have a similar imagination when it comes to loss-of-control risks?

Yeah. Concordia AI worked with the Shanghai AI Lab to co-author the Frontier AI Risk Management Framework. For context, Shanghai AI Lab is an advanced AI research institute with significant leadership roles in national standards-setting and other areas in China.

The framework that we put out is designed to help developers of general-purpose AI models, especially those at the frontier, manage the critical risks that the systems pose, particularly for public safety and national security. We built on some of the best practices and standards from safety-critical industries and from around the international community.

We tried to define a set of red lines as an unacceptable threshold that no one should cross, and also yellow lines as early-warning indicators that would trigger more stringent safety and security measures. Those include critical areas like cyber offense, biological risk, large-scale manipulation and persuasion, and loss-of-control risk.

On loss of control, which you asked about, we also released a technical report alongside the framework. The report evaluated more than 20 open-weight and proprietary models across different domains. Under loss of control, we broke it down into several dangerous capabilities and propensities, including the ability for AI models to self-replicate, to deceive and scheme, and the possibility of producing uncontrolled AI R&D.

So I do think some of the conversations are quite similar to those in other parts of the world, which is a theme that we have constantly come back to during this conversation.

Yeah, it seems like the big differences are really at the consumer-tech layer, right? There’s something that looks more like a licensing regime, with a certain established body of tests and certain requirements before you actually go to market in a consumer-facing way. Aside from that, I would struggle to identify too many big differences in terms of the overall patterns of thought or the way that concerns are conceptualized.

I guess the other big thing that we highlighted in terms of difference is that there is a greater fascination with AGI and a sense of an end goal or some sort of event horizon in the West compared to China. But certainly, I would say there’s a lot more similarity than difference.

One question, though: Given all these things, I haven’t personally tested this systematically to validate it, but I’m sure you’re aware that Dario from Anthropic said of the DeepSeek R1 model that it had very little in the way of guardrails and was rather willing to help with queries about biological weapons and so on.

I think he said it was the least-secured model they had tested in terms of refusals on those kinds of dimensions. How should we make sense of that? I guess, first of all, would you agree that that is a true assessment of that model?

And if that is right, what happened there? How do we have all these criteria and processes, yet something gets released with seemingly less safety baked into it than we've seen from most of the models that come out in the US?

So there are different angles that I could offer. First of all, the approach to AI safety in China is mainly driven by binding regulations and standards led by the government, as we have discussed. That includes registration, pre-deployment testing, and labeling of AI-generated content.

But one effect of this is that Chinese companies may be less likely to release safety evaluations for something that isn't directly in the regulation. The compliance costs required to deal with existing regulations reduce the appetite for taking on additional safety measures.

In the industry section of the *State of AI Safety in China* report, we looked at the practices of all the Chinese AI developers, and they implement all the standard safety techniques, from training-data filtering to safety fine-tuning, RLHF, Constitutional AI, and real-time misuse monitoring. These are very similar to some of the techniques used by major Western companies.

But those practices appear to be primarily geared toward the 31 categories of risk that are in the existing regulation, and the vast majority of Chinese companies have not publicly shared evaluation results for CBRN or loss-of-control risk.

I also think the AI industry is making some progress. I don't know whether you have seen this, but just last week, DeepSeek-R1 was peer-reviewed and published in *Nature*. This is really the first time for any widely used large language model, and in its public-facing safety evaluation section, the authors are being quite candid.

They said that a public, open-weight model is vulnerable to further fine-tuning that could compromise safety protections. Their own evaluation also found that reasoning models like DeepSeek-R1 tend to expose more sensitive or risky knowledge.

But according to the peer-reviewed article, the safety level of DeepSeek-R1 is generally comparable with other state-of-the-art models, comparable with, for example, GPT-4o. In addition, our Concordia AI team performed independent evaluations of the DeepSeek models, and we have seen some improvements in safeguards related to biological-weapons queries just within the last few months.

Specifically, on the SAB-Bench Bio benchmark, the refusal rate of the DeepSeek model increased from around 11% in the V3 version to 54% in the V3.1 version. The refusal rate for harmful biological prompts is around the median compared with other models. I could link to some of the data and graphs from our platform for those who are interested.

So overall, it sounds like, again, more similar than different. I just had this conversation with Adam from FAR.AI, and they had just done some red-teaming of the defense-in-depth systems that companies have created. One of his big takeaways is that all of this stuff is happening on a just-in-time basis, where, if you were really trying to be as safe as you could be, you would probably reverse the order of some things.

You would build the systems that you're going to use to maintain control, and especially on these bio-risk things, which I do think are honestly pretty scary—maybe not quite yet, but soon, if not already—you would have those developed before the model was ready.

But it seems like, in contrast, what we're actually doing is training the model, seeing what it's capable of, and then being like, "Oh, it's actually getting pretty good here. We better tack something on to try to address that." It seems like basically the same kind of thing is happening, at least for now, on the Chinese side on some of these risks: we're first creating the model that is capable, and then we're figuring out what to do about it.

Do you think that will flip? It seems like, with these licensing regimes, there's at least some structure that could flip it. What would flip it? Would it be expanding from 31 to some additional categories?

And what about open source? Debate on this runs super hot, too. How committed do you think Chinese decision-makers are, I guess spanning government and companies? How committed do you think Chinese decision-makers are to continuing to open-source indefinitely?

Or do you think there could be a time, perhaps not too far in the future, where there's a different analysis that's like, "Hey, these things are just too unwieldy. We really need to keep them on infrastructure that we control," because that's obviously part of how you would run a broader defense-in-depth strategy?

Yeah, I think AI safety is a much broader landscape than just AI-enabled bioterrorism, right? For example, I remember looking at the AI Risk Repository from MIT. It lists 24 subdomains and 1,000 different types of risk.

So I do think every country and company might have a slightly different set of safety priorities and challenges. We shouldn't be judging one player on just a single narrow metric.

But I think you're right that model registration and pre-deployment testing is quite a flexible tool that could be adapted to address new concerns from CBRN, loss of control, or other emerging challenges from advanced AI. In fact, the Chinese government has repeatedly said that the system could evolve over time.

The AI safety governance framework by TC260 that I mentioned earlier could help the national standard-setting body issue specific guidelines in the coming years. We're already seeing some developments in this space, including agentic AI, open-source governance, and other critical issues involving open weights.

I think there are tremendous benefits, both in terms of safety and innovation more broadly. At the World AI Conference in Shanghai in July, China announced the Global AI Governance Action Plan.

Across all these documents, there is a sense that open source is very beneficial and that China should capitalize on this development. That is similar to the US action plan, which is also pro-open-source.

At the same time, the Chinese documents have also mentioned the possibility of misuse from open weights. I think that's why we've seen papers from, for example, the UK AI Security Institute. The paper *Deep Ignorance* recommends removing hazardous knowledge from the data at the pre-training phase, and that recommendation is also in the TC260 document that I mentioned earlier.

In general, it seems like this is a new research challenge that stakeholders in China and elsewhere are really trying to tackle.

So maybe the vision, if I'm filling in the gaps correctly, would be a pretty strong commitment to open weights going forward, making that safe by doing enough filtering of the training data so that the version that's put out as open weights to the public just doesn't know about key questions in virology or whatever, and literally can't help you there—as opposed to just having the sort of surface-level refusal training.

Then, presumably, on a more structured-access basis, there would still be, of course, models with the full scientific dataset or whatever. But those might be the ones that would be withheld from open-weight release, and you'd have to have certain access or approval to use those models, with a know-your-customer regime or whatever sort of other additional layers of control.

Yeah, I think we do need a defense-in-depth approach, and we're in a pretty early stage of developing the science and practice of open-weight AI risk management. There are other potential interventions, like tamper-resistant safeguards, responsible AI licenses, or monitoring at the open-source platform level.

Many of those might not work, and there's a lot of debate within the research community, but I think the general shape is that there is a lot of positive interest and commitment in open source. Let's try to make it safer and less prone to misuse.

Let's shift gears to the relationship between the 2 countries. I know this is something you're also working on. Obviously, it's not great, broadly speaking.

One big thing that you hear from a lot of US government officials—current and former, military, intelligence community, whatever—is this general sense that you can't trust the CCP.

I think what a lot of people would say in response to all these things that we've discussed is basically, "That all sounds nice, and it might all be true, and we do maybe even respect the degree to which the Chinese government is taking care of its own population." In terms of development, the track record is undeniable.

When it comes to making sure there's good standards for medical chatbots or whatever, we could say, "Sure, that's great," and it sounds like they might be doing a pretty good job. Maybe we could even take a lesson from it.

But then you still get people falling back to, "But when it comes to the real game of international power, we just can't trust the CCP." That's the prevailing sentiment for a lot of people, and so you'll get people saying things like, "Whatever they say, whatever they do domestically, whatever they do to protect their kids from AI girlfriends."

When it comes to militarizing things, when it comes to racing for some sort of breakthrough superintelligence that is going to create some sort of strategic advantage, we have to assume that they’re going to do it. Because we have to assume they’re going to do it, we have to do it.

Is there anything you could say to those people that would be reassuring? If I want to at least create some doubt in their mind that there might be more opportunity for collaboration than their current outlook suggests, what could I say to them?

Certainly, there is quite a lot of distrust on both sides, but I think in AI, the specifics do matter. For example, if there is concern around developing advanced AI in secret, that is just not grounded in the reality of AI development. The state-of-the-art AI models from China are largely released openly. From Kimi to MiniMax, Qwen, and DeepSeek, they are now among the world’s best open-source AI models.

I think that level of transparency, and the possibility of third-party auditing and research, should allow for greater trust between the companies and the governments. I also think the most advanced AI models in both the US and China are developed by the private sector, not inside a government. It is the same global community of AI researchers—a few hundred top elite researchers, most of whom know each other—and I think that common ground of scientific vision and commercial development is another factor where there should be more common ground between the 2 countries.

Would you say—and I definitely think this is important to keep in mind—that the distrust runs both ways from the Chinese perspective? Is there a similar worry? With OpenAI, for example, we hear these conflicting narratives. On the one hand, Sam, for example, has said, “You people outside of the frontier companies don’t know how good you have it. The gap between what we have internally and what you get to use as a retail customer is only a couple of months.”

During those couple of months, we’re working as fast as we can to get it all buttoned up so that we can release it. Basically, he’s saying, “You guys don’t appreciate how close to the absolute bleeding-edge frontier you are.” But then, at the same time, you have, “Oh, we have an IMO gold-medal model, and that’s going to be a while before we release that, by the way.”

Then you have AI 2027-type narratives from, again, another former OpenAI insider, saying that one of the things we should be watching for is a divergence between the models that get released to the public and the models that are developed and deployed internally, perhaps with this idea of trying to do some sort of recursive self-improvement, intelligence explosion, whatever.

From the Chinese perspective, US companies can’t make quite as much of a case about, “Hey, we’re releasing everything open source,” but there is at least some claim that, “We’re not hiding much, and everything’s coming out pretty quickly.” Is that credible from the Chinese perspective? Or is there a worry that, sure, there’s GPT-5, but what we really don’t know is what OpenAI has done in secret, or what they might be doing in partnership with the US government that we have no visibility into?

Is that a worry that people get stuck on? Is there a mirror image of what I described in the US? So many of these conversations go off the rails with, “Well, China’s going to do it.” Does that happen in China, too, where they say, “The US is going to do it, so we have to keep going because we can’t stop them”?

I don’t think there is as much fear and paranoia. As I said earlier, the AI Plus initiative by China shows that the main push is to mobilize AI resources and direct them toward applications. That is a sign that applications, the economic engine, and delivering for the people remain the top priorities for the government and for the party.

I do think that, when it comes to export controls, the Chinese AI industry views them as a clear attempt by the US to stifle China’s technological progress and maintain the US’s status as the leading global superpower. On that front, I think it has created a sense of resentment.

What is often missed is how this also affects everyday applications. Much of the computing power, much of the FLOPs, is used for civilian applications, from generating AI art to powering AI medical diagnosis. In effect, export controls could deny a better quality of life for everyday Chinese people. I think that also creates a bottleneck for addressing some of the shared concerns, like risks from advanced AI, between the 2 countries.

I do think that is perhaps one of the most significant issues preventing, at least from the Chinese perspective, a sense of common ground and possible cooperation with the US.

What do you make of the recent decision from the Chinese government? We’ve seen multiple reversals on this, right? The Trump decision to go ahead and sell the H20s—I think it’s worth reminding people that the H20 was designed in response to the government standard that said you could only sell up to a certain level of chip to China. My understanding is that they said, “Okay, great. We’ll make a chip of that kind and sell it to China.” Then it was, “Oh, no, you don’t.”

I’m not really sure how companies are supposed to operate given that kind of relationship with the regulator. Nevertheless, Trump eventually comes around and says, “Yeah, it’s fine. Just give me a little cut of the action, and you can do it.”

I was quite surprised when the Chinese government said, “Actually, hold on. We don’t want them.” How do you make sense of that? Of all the signals we’re getting from China about how seriously they’re taking AI, most of them are pretty similar, but this one feels to me like, “Whoa.” If you’re really serious, if you really think this is revolutionary technology, you would want to import those chips while you can.

Of course, you still want to develop your own domestic capacity to produce chips, but why not both? The seemingly obvious strategy is to import while you can and invest intensively at the same time. How do you make sense of that refusal of the H20s? Do you think that refusal of American chips will be persistent if there are other chips available to buy?

As of September 22, Chinese companies are not purchasing the RTX PRO 6000D from NVIDIA, which is seen as low quality and somewhat overpriced. I think there’s also a lack of trust in the H20 specifically, and Chinese regulators have raised concerns about the integration of backdoors—specifically, remote location tracking and remote shutdown control integrated into the chips.

First of all, this could be reflecting a shift in the overall balance of national power as well. When the Trump 1.0 administration started the trade war and the Biden administration rolled out a series of export controls, China was less prepared. Now, I think China is demonstrating that it is not dependent on the US for its technological future. In fact, maybe the US is also dependent on China for critical minerals and rare earths in some of the recent trade negotiations.

Second, this is backed by a growing level of confidence in the domestic AI chip industry in China. The export controls have acted as a catalyst to accelerate the development of homegrown alternatives. Companies like Huawei are making significant progress with their Ascend line of chips.

Just this month, Huawei announced the new Atlas 950 SuperPoD, which would support more than 8,000 chips. The Atlas 950 SuperCluster would also use more than 500,000 chips as a cluster. Perhaps from the Chinese industry’s perspective, it doesn’t want to be relying on the American stack. This is a critical window to develop the homegrown alternative.

I still think what the analysts I’ve read, who I think are generally taken to be the most credible among Western analysts of the chip industry, seem to boil down to something like this: Of course Chinese companies are going to make good progress here over time. Of course there’s no preventing Huawei from becoming a scaled chip manufacturer on a decade-long timeline. But still, over the next 5 years, the expectation is that production capacity is just going to be dramatically higher for the West, notably including Taiwan in a critical role.

The NVIDIA-TSMC-Western complex is going to produce—I would say the median estimate would be—comfortably more than an order of magnitude more chips than Chinese domestic producers will be able to produce. If that is true, it’s going to be a big difference in terms of how big the training runs can be, how big the deployments can be throughout the economy, and so on.

Do you think that analysis is wrong? Or if that analysis is right, I come back to the point that I would still buy those H20s while I could.

Maybe they do get bricked at some point, or something. I can't speak definitively to whether there are any backdoor technologies or attack possibilities built into those. I would guess not, but I don't really know.

What do you make of that? Because it seems like that is the conventional wisdom in the West: It's going to be a dramatic difference in production capacity, and that's going to give the U.S. a pretty durable edge, at least over a 5-year time scale. I think the perception from the West is that it's just a plain mistake for the Chinese government to refuse the H20s, that it's a face-saving measure that's ultimately self-defeating or something along those lines. But the analysis could be wrong. Maybe it is going to happen a lot faster, and maybe the gap wouldn't be so big. What's your expectation?

I think there are 2 other factors. One is that China could leverage relative energy abundance to overcome some of the limitations on a single-chip basis. For example, a few months ago, there was the release of CloudMatrix 384 by Huawei, which uses more than 5 times as many Ascend chips, but more than compensates for each chip being less powerful compared with an NVIDIA Blackwell GPU.

It matters less for China if the primary trade-off is energy efficiency, given that China has significantly expanded its power grid, adding an amount of capacity equal to the entire U.S. grid in the last decade alone. Most of that could be pretty renewable, with some of the largest installations in the world of solar, hydro, and wind, and now a potentially leading role in nuclear energy deployment as well. In the Chinese stack, China could prioritize scale over power density.

I think another factor is that beyond hardware, Chinese AI companies are also trying to become more efficient at training large models. As detailed in the peer-reviewed article in Nature, the DeepSeek-R1 cost maybe 500 NVIDIA H100 chips. That's pretty impressive given the performance that it is able to produce.

That energy point is interesting. I have a little bit of an intuition around that that I haven't really developed. Basically, the idea there is that we can potentially make all the trailing-edge chips we might need, and the primary disadvantage of those is efficiency. But we've got so much energy, and we've got the ability to provide national subsidies for critical industries, such that maybe we can just make up for it that way, and the deficit isn't actually so big when it comes to what we can practically do in terms of large-scale training.

That is definitely very interesting, and certainly the requirements on training, I think, have been relaxed. This is not something I would say I'm a real expert in by any means, but not too long ago there was this idea that you have to have everything in one data center—the bandwidth is such a limiting factor, and so on—and now it seems like we've taken, collectively, the global research community has taken pretty big bites out of that problem.

Now we are training across multiple data centers. Now we are seeing more distributed training come online. We're also seeing all sorts of efficient weight communication and update schemes. Not to mention the fact that inference itself is becoming a bigger and bigger part of training due to the rise of reinforcement learning. So the trillion-dollar cluster all in one place, all with super-high interconnect, being a hard requirement—that also seems to have fallen off quite a bit compared to what the discourse might have suggested 18 months ago or so.

Going back to this trust issue a little bit, I think the point about China open-sourcing the models is at least somewhat compelling. But it still leaves this question: What are they doing in secret? Sure, there are all these great things coming out, and sure, they're open source, but on both sides, we really don't know what's going on in some data center somewhere, between the government and a leading company or whatever. We just have a really hard time getting visibility.

Without visibility, and without a foundation of trust, it's really hard to be confident that the other side is not doing some dangerous experiment that we should be concerned about. And I do think that's a pretty symmetrical concern. Do you have any ideas about how we can create a stable equilibrium between the 2 countries?

I'm sure you read the “Superintelligence Strategy” document from Dan Hendrycks, Eric Schmidt, and Alexandr Wang not too long ago, where they described mutually assured AI malfunction, or MAIM. I wasn't super compelled that that would actually be a stable equilibrium, but I did feel like, hey, at least somebody's trying to sketch one out.

Do you have any sort of narrative or sense of how the 2 countries, aside from a revolution in relations, could get comfortable with some amount of trust that could create some stability, such that neither side feels compelled to do dangerous things out of a fear that the other side might be doing it in secret?

That's a fascinating and very important question. The concept in the “Superintelligence Strategy,” as I understand it, is to prevent any single nation from achieving AI dominance, and countries—especially the great powers—are prepared to sabotage and destroy the AI infrastructure of a rival.

On the positive side, it's good that it acknowledges that dominance by a single superpower is a problematic objective and a flawed paradigm, and is trying to chart a path of civility in a world of very powerful AI. But I do think the analogy to mutually assured destruction, or MAD, from the nuclear era is also problematic for 2 reasons.

First, it really lacks a clear, observable red line in the world of AI. The logic of nuclear deterrence works because using a nuclear weapon is a detectable event. You can't really hide a nuclear test or a launch of an ICBM, so that creates a clear red line.

But if it hinges on a trigger point of aggressive development of superintelligence, what does that really mean in practice? We have this METR graph of the length of tasks that AI could do that is doubling every 7 months. Is there a certain level of that graph that would count as getting into the intelligence explosion? I worry that it is quite vague if we try to apply that concept in the world of advanced AI.

Arguably, if you think about it from the Chinese perspective, the U.S., with some of the prominent AI leaders, is already talking about AGI in 2027 and also implementing export controls and so forth. One could imagine that the U.S. is already racing for a monopoly on superintelligence. Does that trigger mutually assured AI malfunction? I hope not.

That leads me to the second point: the risk of instability and escalation. If countries actively endorse this strategy, then it would basically be declaring a willingness to go to war with one another. It's not just about attacking a data center. It would be destroying one of the most valuable national assets, one of the most critical pieces of infrastructure of another country.

I think that is a very terrible and unpredictable event, and I don't see that as a plan for stability. Quite the opposite: It seems like a potential hair-trigger alert of terrible risk from AI.

Unfortunately, I agree, and I think you're very right to highlight just how intense the fog of possible war is going to be around this. For better or worse, and, I guess, for false positives and false negatives, I would say that if you're sitting in one country trying to assess what's going on in some data center in the other country, your visibility into this is low in every respect, starting with what you should care about, as you rightly pointed out, but then also whether you're getting an accurate signal on what's really going on.

It just seems like it's going to be extremely hard for anybody to have much confidence that they would be doing the right thing at the moment when they would be making a decision to sabotage some key infrastructure. So, yeah, I'm with you. Again, I applaud the effort to look for some sort of stable equilibrium. I did not feel like they found it in that piece.

Do you have any better ideas? I mean, not to put you on the spot, but we need all the help we can get here. Is there any stable state you could imagine getting to, again, without an overly utopian imagination of a revolution in relations? We can come to that next, in terms of working toward that. But if we assume that distrust levels remain relatively high, is there any way to make that a stable situation?

I wrote some of the recommendations in the Time magazine piece. I think one strategy would involve building on 3 core pillars. The first pillar would be for the international community, especially between the AI great powers, to define a set of concrete red lines for advanced AI.

That is a set of agreed-upon limits on the capabilities and behaviors of frontier models. For example, that would prohibit the development of systems that could pose a catastrophic or even existential risk to humanity.

The possibility of AI leading to the proliferation of weapons of mass destruction, the possibility of autonomous AI being able to self-replicate or self-improve, and also leading to an intelligence explosion that either humanity as a whole or other countries can no longer comprehend and control.

The second pillar would be continuous testing and evaluation for early-warning indicators. Ideally, we can create an architecture—a set of shared protocols—so that countries and leading AI companies can share the most relevant evaluation results with one another.

The final pillar involves creating a set of emergency-response protocols. If certain model behaviors are found to be early-warning indicators, what should you do? What should be triggered when those thresholds are being crossed? That could involve implementing more safety and security measures, or mandating human oversight. Especially in moments of crisis, it is really critical that countries and the international community have a prepared plan to manage the situation rather than let the risk escalate.

How optimistic are you that our governments can come together and do this in any sort of credible way? I've voiced the view from the U.S. government that you can't trust the CCP. I totally expect that the Chinese government feels similarly about the U.S., even if it's not, “You can't trust a particular individual,” or whatever. We obviously have such radical changes in attitudes at the top that it's very hard for the Chinese government to know what they're going to be dealing with in 2028—or I guess 2029, when the next president takes office.

We've had—I’m not really sure of the status of this—but there was at least the one “no AI in the nuclear chain of command” soft agreement. I'm not sure if that has been ratified or institutionalized in some way. Could we expect governments to come together and do more in the direction that you're advising right now? If not, what else can we do?

You've got interesting ideas around Singapore as a meeting place, and obviously there are these Track 1.5 and Track 2 dialogue-type things that you're putting together. Where can we actually make progress toward these goals right now?

You're right that, given all the tension between the U.S. and China, it's positive that the leaders of the 2 countries were able to identify AI risk as an area of dialogue. They have held meetings on the topic and have also built toward a joint agreement that there should be human control over nuclear command and control.

A few years ago, this would have been thought of as pretty surprising. At least there is a positive step. Obviously, I'm not an expert in the AI and nuclear space, and the agreement is pretty high-level at the moment. There is some work to be done in terms of operationalizing the agreement.

I do think the overarching concept of having a human in the loop on some of the most critical decision-making systems could be applied to broader discussions on AI safety. We already have leading scientists and academics from China and the U.S. agreeing that loss of control is a serious issue. There are probably steps that we can build not just between the researchers, but also between the industries, and then eventually at the Track 1 level, to ensure a human in the loop when we get to certain levels of autonomous, advanced AI systems.

Going away from the official government-to-government track, tell us about the dialogues, convenings, and so on that you're helping to put together. What's happening at those? What is the theory of change? Is it about keeping connections warm and keeping the research communities connected to each other? How do you conceive of the value that all that work is driving?

Track 2 dialogues and scientist-to-scientist conversations have long been a mechanism for exchange on issues of common concern between great powers. Primarily, they could foster trust and increase mutual understanding among the participants, and also help formulate and refine policy solutions for their respective governments.

One example was the Pugwash Conferences on Science and World Affairs, which was held regularly since 1957. That was really a pivotal time for reducing catastrophic risk from nuclear weapons.

We have put out a piece on Track 2 dialogues on AI between China and other countries. Overall, I think these dialogues have produced several useful outcomes. For example, revealing areas of scientific consensus on frontier risk, and solving particular issues that experts are better suited to solve at the moment—for example, looking into threat modeling and the potential risks from the convergence of AI and biological risks. I think this is just a critical area where we should continue.

How about Singapore as a venue for that sort of thing? Is that a unique opportunity in today's world? I've never even been to Singapore, so I've got a lot to catch up on.

I think Singapore is one of the top countries in AI R&D globally, and is also ranked highly for overall readiness for AI innovation. It's a very forward-leaning country.

Concordia AI recently published a report on the state of AI safety in Singapore, and it seems like Singapore is playing a pretty outsized role in global and regional AI governance. For example, it is able to convene international AI events like the Singapore Conference on AI, which produced a consensus document on global AI safety research priorities.

That consensus was developed by over 100 global experts who came together to build a shared technical agenda for general-purpose AI safety, including loss-of-control risk. Another distinct feature is that Singapore is a very vibrant hub for both Chinese and U.S. companies. Many of the leading companies from the U.S. and China have offices and a presence in Singapore.

Singapore has a very vibrant assurance hub doing testing and evaluation on these models, including its Singapore AI Safety Institute. Given this combination of convenience, neutrality between the U.S. and China, and a very strong AI ecosystem in itself, it is pretty well placed to be a meeting place between the East and the West. I think that's also the reason why we're excited to have an office in Singapore.

Is there any other sort of international cultural initiative that you might promote? One idea that I've had—and you can react to this idea, but I'm especially interested to hear any other ideas that you have—is to create something like an MMLU for morality, which I think is going to be a tricky project.

I'm struck by the fact that when companies put out new models, I particularly noticed this with Gemini 2.5 Pro, in their official blog post announcing the new model, I believe they reported 12 different benchmark results. None of them pertained to safety, alignment, morality, law-following, or anything along those lines. It was all just raw capability.

This got me thinking: if there were 1 number that we could look at to get a sense for how well-behaved a model is, that could be a really powerful carrot. It could potentially lead companies in that direction. If it became the norm that you were going to report that number when you put out a new model, and it became 1 of the dimensions of competition, it seems like it could be a pretty meaningful contribution to the overall trajectory of AI development.

There are a lot of caveats. You probably have to have a bunch of subscores under that 1 score. You might have to have localized scores as well, because I do think countries are going to want somewhat different behavior on certain aspects of those questions. But it seems like a good idea to me. What do you think about that idea? What other outside-the-box ideas do you think are relatively neglected right now? What do we need people to step up and do that is not yet happening?

I'm fully supportive of creating these positive race-to-the-top dynamics, incentivizing companies and researchers to compete on AI safety research to create safer, more trustworthy models. To some extent, this is already happening. People have to report their performance on MMLU, but also HarmBench and other AI safety-, ethics-, and risk-related benchmarks.

I don't think we can rely on any single benchmark. We just need a portfolio covering all those different dimensions. So, yes, I think it would be great if we had more leaderboards of AI safety performance.

Any other moonshot projects that you would love to see somebody act on that just aren't happening today?

I think it would be great if we could align on a global framework for frontier AI risk management. From China, we work with Shanghai AI Lab on this framework. This closely resembles some of the frameworks developed by other global players—for example, some of the leading AI labs in the U.S., and also, recently, the EU Code of Practice.

I really think this convergence of approaches is a very positive development. But we are still in the early days of creating international AI standards.

One analogy I think of is that when people fly between different cities—either from Europe to the U.S. or from Singapore to China—we all abide by the same set of international safety standards. It's quite incredible how much safer it has been to get on a plane.

I remember a statistic showing that the number of accidents from getting on an airplane has dropped by, I don't know, 2 orders of magnitude just within the last few decades.

How about heroes? Are there people on either the US or China side that you think should have higher status than they do, or just be more appreciated? Who's doing great work to either reduce tensions or build understanding and these shared frameworks? Who do you admire? That could be someone I should have on the show in the future, but who should people look to for inspiration?

Yeah, I think one of the most urgent priorities right now is to build a shared international understanding of the risks from advanced AI. One really positive and landmark moment I experienced was the global AI Safety Summit at Bletchley Park in November 2023. I remember attending the summit and seeing ministers from the US, China, and other countries—a total of 28 countries—come together to sign the Bletchley Declaration.

A specific outcome of that summit was an agreement to nominate international experts to work on an independent report led by Yoshua Bengio, featuring contributions from 100 experts around the world. It really aims to be sort of the IPCC for AI safety, and I think that is a very positive example.

There are a number of other international forums that are making progress in this space. For example, on loss of control, there are the International Dialogues on AI Safety, which recently agreed that there is a growing level of deception and scheming behavior from frontier AI models. There is also the Global AI Biosecurity Forum, which published a statement on the biosecurity risks at the convergence of AI and the life sciences, and some of the world's leading scientists from both the AI and synthetic biology communities also signed on to that statement.

How about individuals who aren't famous or powerful? What can we do? I want to go to China for multiple reasons. I've never been, and it sounds fun. The food sounds amazing. Can I do anything that would actually be meaningfully positive? How would you coach people who want to make their own grassroots, even if modest, contribution? What can we do to make the world marginally better?

Thank you for asking that. First of all, I think we all need more people-to-people exchange and understanding. Trying to go beyond the headlines and creating a more nuanced understanding is a first step.

This is what Concordia AI tried to do, right, by publishing these AI safety developments in China, Singapore, and Asia. And vice versa, we also try to absorb and learn from different parts of the world, including from the West, in our work within China.

For people outside of China, coming to AI conferences in China could be a good opportunity. For example, there is the World Artificial Intelligence Conference that happens every summer. I think that is a very rich platform for discussing the frontiers of AI innovation, but also AI governance and safety.

In terms of donations, I think philanthropy could be a powerful force for making a difference in AI safety: funding ambitious scientific research, training the next generation of leaders, and fostering informal dialogues. In some ways, I think the field of AI safety philanthropy is still in its early stages, much like climate science was a few decades ago. Getting involved now means that one could be an early shaper of this critical domain.

I just did an AI safety charity review project this summer, and it was very hard for philanthropists to figure out what is actually helping. We had a number of organizations, and I think, by and large, they did receive grants, but I noticed from my perspective as an evaluator that it's very hard to read a document that says, "We're trying to do this and this to foster improvement in US-China relations," and have any sense of whether it's working or having any impact. It's tough.

How do you know what's working? If you were to advise somebody to look for signals in that kind of thing—or even just for yourself—how do you know what's working and what's not working? What are the feedback mechanisms that you trust to know that you're making the world marginally better?

That's a great question. I think it depends on the specific interventions, but when it comes to creating these international conversations, one metric could be whether we are exploring topics and issues that are pretty important and urgent but that no one was talking about before.

To be concrete, we recently ran an international dialogue with a focus on crisis preparedness, exploring scenarios such as how AI could enable large-scale cyberattacks on critical infrastructure. We found that policymakers and elite decision-makers were all pretty surprised that this could happen in the next few years. I think we created a space for people to be better prepared for a world of powerful and unfamiliar AI.

Another metric could be whether we could create consensus and outcome documents. I mentioned a couple of examples before, and I think this really shifts the Overton window in the international conversation to show that this is something that could be agreed upon. Repeatedly, I think we have shown that international collaboration on catastrophic risk is something that is important and tractable to make progress on.

When you're doing this kind of international-relations improvement work in China, do you feel like you're in a minority position? Here, the folks who are focused on this certainly feel that they're in a minority position. My sense is that they feel things are getting worse, not better—that they're basically trying to resist a negative trend, but the negative trend is still happening.

They're kind of a minority voice, and it's hard for them to break through because the dominant narrative is so hawkish and so focused on rivalry and winning. Do you feel similarly in China, or do you feel like you're more mainstream there in some sense than people here feel?

I feel pretty mainstream. I think there are a lot of stakeholders, ranging from academics to industry to think tanks, that view international cooperation on AI governance as not only accepted but something to be prioritized. In our reports, we have documented a lot of institutions and individual experts that have done this consistently throughout the years.

Certainly, when it comes to prominent AI conferences in Beijing, Shanghai, and elsewhere, in our experience, the organizers really want to engage with international experts and guests and would put in a lot of time and resources to send those invitations. When people come to China, a lot of effort is made to ensure that they feel comfortable, that they can explore the cities, and that they have a good time.

I think there is a lot of goodwill in general. I don't feel like I am doing this as part of a very small minority or marginalized community. Not at all.

Yeah, good. That's an important data point. Maybe one last big-picture, sort of outside-the-box question: If you had some large amount of money—say, 1 billion or several billion dollars—are there any mega-project-type things that you could see people putting together that might really move the needle?

I'm thinking as far-fetched as a joint research center on some small Pacific island, where you could imagine scientists coming together to do very sensitive research in an environment that neither side could really defend. That is a sort of obvious, sequestered space for this kind of super-sensitive work.

I don't know if that's a good idea or a bad idea, but that strikes me as the kind of thing that, if we were really trying to come up with creative solutions, we'd hear a lot more about—outside-the-box ideas like that. Do you have any outside-the-box ideas that you think maybe somebody should pick up and run with?

Yeah, I think having an international joint research project or lab with a focus on frontier AI safety sounds like a great idea. I think we really lack a space where researchers can come together not just to discuss but to make concrete progress.

I don't have very many creative ideas. What the world probably needs more are sensible, pragmatic ideas to make progress. All too often, even the most commonsense things to do—especially when it comes to international collaboration and policy—are not being done. So I would probably respond to your question in a different way: We just need to make more progress on the low-hanging fruit.

Yeah, double down on the basics. Is there anything that we haven't touched on that you think we should, or that you would want to make sure people are aware of before I let you go?

I think I would just like to mention that around 2023, discussions on frontier risk also took off around the world, obviously because of ChatGPT, but in China as well. You saw prominent Chinese scientists sign on to the statement that extinction risk from AI should be a global priority alongside other concerns.

There was an important moment at a Beijing AI conference featuring people like Geoffrey Hinton, Andrew Yao, Sam Altman, and others, with a full day of speeches and panels on AI safety.

And so that was really one of the first times when risks from AI were discussed so prominently at a technical AI conference in China. Since then, the landscape has changed quite a lot as well. As of 2025, there are more than 30 AI research groups in China that have devoted a substantial amount of time to frontier safety.

In the beginning, it was mostly along the lines of RLHF or jailbreak defense, but now I think there are also a lot more papers on scalable oversight of superhuman AI systems, mechanistic interpretability, and other areas that are much closer to advanced AI safety concerns. I think this expansion of research highlights that academics in China think this is a problem worth exploring and directly investing their time and effort into, and are also pretty well aligned with some of the research happening in the West.

Any highlights from that body of research that you think I should take a deeper dive into? Particularly, are there any that are notably distinct in terms of their approach from the things that might be more familiar to me as somebody who primarily follows Western developments?

Again, I think 95% of those papers and research are pretty similar. But I do think that in the last 6 months, there have been more papers that focus on the risks from embodied AI, both in terms of how that could align with human values and also in terms of jailbreaking a foundation-model-controlled robot. In general, it seems like physical AI is a theme of Chinese AI in 2025.

Yeah, that’s maybe something we’ve under-discussed, actually, because there are so many commonalities. But it does seem like China has a huge manufacturing strength that, if and when these humanoid robots actually start working, is going to be hugely important for producing them at scale.

I haven’t seen anything that I can recall from Western researchers that focuses on this, apart from a little bit from Google, I suppose, because they are doing this stuff. But it does seem relatively neglected. It seems like a collective blind spot in the West: We’re actually going to have humanoid robots. They probably are going to work, there may be a lot of them walking among us, and how does that change the way we should be thinking about safety and control?

It does strike me as something that is underdeveloped here. So to hear you highlight that as something that might be unique in the safety literature there is interesting. That’s maybe something I can take a deeper look into, and I definitely appreciate any pointers to particular papers, people, or whatever.

Is there anything else that we should know, or should take a minute to contemplate, when it comes to the relative emphasis on embodied AI broadly, or humanoid robots specifically, that’s going on in China right now?

I’m sure there’s also a lot of AI safety literature going on in other parts of the world, but perhaps it’s worth highlighting the concerns for safety in industry documents. There is an AI industry association in China that put out these voluntary safety commitments. They have been signed by more than 20 major Chinese companies, and the latest version highlights risks from agentic AI but also embodied AI.

I think this really reflects how the industry is looking ahead and thinks that its models could be integrated into the physical world at some point, and that this is a time to start thinking about safety and policy measures as well.

Yeah, I definitely see some of these short videos online of these robots getting kicked over, bouncing back, doing flips, and stuff, and it’s like, “Whoa.” Not only might these things be as smart or smarter than us, but they’re potentially going to be a lot more agile and robust to physical assaults than we might be before long, too. So, yeah, it’s going to be a strange world in the not-too-distant future.

During Chinese New Year, there was a national TV program showing the robots from a company, Unitree, having group dances. I know hundreds of millions of people in China probably watched it, and now they are running marathon competitions between AI robots and humans. Very soon, they will be doing full-scale Olympics between the humanoids and humans in Beijing and other cities. So it is a very interesting world.

What is the cultural reaction to that like? One thing that people often say when we debate what future life is going to look like, where we’re going to get meaning from, and what’s going to matter is that chess has never been more popular.

Isn’t it interesting that we had the first computer that beat a grandmaster in chess like 20—I think in the 1990s, maybe 30, almost 30 years ago now? Then there was the period when the human and the AI together were best, and now the AIs are just the best. But people are still very interested in watching humans play chess. They’ll say that nobody’s that interested in watching AIs play chess against each other. Maybe the real chess experts are.

How would you describe the Chinese reaction to this sort of thing? Do you think the Chinese public will be interested in a sort of robot Olympics? Will that be something that captures the public imagination? Because it seems like people here don’t expect that to capture public interest, but maybe it’s different there.

No, I think many people do find it somewhat entertaining and amusing. There is also a genuine function to show that some companies perform better in these sorts of real-world benchmarks, as it were, and to get beyond the hype.

For example, the marathon competition between the humanoids and the humans shows that there is still a lot of room for improvement for the humanoids. It also creates societal discussions about what that means for the potential impact on the labor market.

There was a local incident involving people protesting that self-driving cars were making too much progress and could take away jobs from drivers in a central province of China. So obviously, there are controversial and negative ramifications as well.

Yeah. How much does that get—I mean, that’s another thing we haven’t covered. There are so many aspects of this that we can’t possibly cover them all, but since you raised that, is there anxiety in the Chinese public that this could be bad for me personally?

In the US, there’s definitely a broad, quite widespread sense that even if AI gets really good, even if it can do everything better than I can do, and even if we have abundance in theory, I’m not really sure that’s going to benefit me. I’m speaking not as Nathan, but as a sort of general plurality of the public. I’m not really sure that’s going to benefit me.

I still worry that the rich might get richer and I might be left behind. If I’m driving a car for a living and all of a sudden the AIs can do it better, I’m not sure what’s going to become of me. Does that same anxiety exist across the Chinese public, or is there perhaps a higher level of confidence that the benefits from these sorts of things will in fact be broadly shared?

It’s hard to summarize the views of more than a billion people, obviously, but how would you summarize the vibe there?

Obviously, I can only speak to my partial observations and anecdotes. But what do we have as humans? We have our intelligence in our minds, and then we have our bodies doing physical work. I think it’s very natural for people to ask what happens if AI could do both in the future.

In the last few years, the unemployment rate, especially among young people, has increased quite a bit in China. Everyone has a bachelor’s degree, education levels have gone up, and so it’s much harder to get a job at a bank or in the government doing white-collar work.

People also have to consider food delivery or being a driver on DiDi, the Chinese version of Uber. But there are also self-driving cars and all these humanoids that can do what anyone can do physically. So I do think there is a sense of anxiety among significant portions of the population as well. It often surprises me how little governments are addressing this concern, because it seems so salient for most people.

Yeah. That’s interesting. I would have maybe expected the Chinese government to be more forward-looking, or a little bit more prepared than what I perceive the US government to be. But it sounds like maybe not. Again, it is all coming at us quite fast, so maybe I shouldn’t be surprised.

I think there could be a feature and a potential strength that China could leverage. It is a socialist country, and if the profits of AI are dependent on chips, which are also dependent on energy, data centers, and land as critical infrastructure, most of that infrastructure is owned by the state, not by private companies or state-owned enterprises.

You could imagine a model of distributing the benefits of AI using these state-owned assets.

Yeah, I think we’re going to need something like that here. I’ve often—not that I have the answer by any means—suggested that the time is probably now to start working on a new social contract, and it’s been slow to develop. But I do give Sam Altman, in particular, credit for investing personally in universal basic income studies.

And it does seem to me like we're going to need to do something to decouple one's fundamental right to live a decent life from your ability to contribute meaningfully to the economy. No matter how far the AI continues to progress from here, it seems like just with implementation of what we have, there's going to be very significant displacement, and we really can't expect our current structures to handle that in a graceful way, to put it mildly. So, somebody needs to take some leadership there, and we currently have kind of a vacuum. But yeah, sounds like there's opportunity for people to step up in the Chinese context as well.

I'll just ask it again. Anything else we haven't talked about that you think people should be aware of before we break?

I don't think so. I think maybe, as an expression of gratitude, thank you for all the thoughtful and open-minded discussions on your podcast. I think this is really amazing, with all the work that you're doing and also for bridging international understanding. I'm really pleased and happy to have this conversation with you, Nathan.

Well, that's very kind, and I'll do my best to live up to it, but right back at you. I appreciate all the hard work you're doing to maintain and hopefully even build positive momentum in shared understanding of the AI phenomenon, and particularly how we can make sure we're on the right side of history with this thing, because it doesn't seem like that's going to happen by default, and it's going to take some heroic efforts from folks like you. So, I really appreciate what you're doing.