Chinese AI – They're Just Like Us? With Beijing-Based Concordia AI CEO Brian Tse

Hello, and welcome back to The Cognitive Revolution.

Today, I'm excited to share my conversation with Bryan Tse, founder and CEO of Concordia AI, a Beijing-based social enterprise working to advance global AI safety and governance. I've been wanting to have this conversation since I first started the show, and it feels especially timely right now. Recently, at The Curve in Berkeley, I participated in conversations about AI policy and the options we have available as we attempt to safely navigate the development and deployment of powerful AI systems. And I was once again struck by how many of these discussions, including with members of technical staff at frontier model developers, are explicitly premised on assumptions like China will never slow down their AI development, or even in some cases, China doesn't care about AI safety. These notions are doing a lot of work in shaping American AI policy and corporate strategy. They're being used to justify everything from aggressive scaling timelines to military partnerships to export controls that cut China off from advanced chips. And yet, I'm not sure how well they hold up to scrutiny. I often say that AI defies all binaries. Even American politics, which is usually so rigidly polarized and partisan, often has its coalitions scrambled by AI questions. And this conversation with Bryan suggests that, at least to a first approximation, the US and Chinese AI ecosystems are not opposing forces, but similarly structured communities pursuing remarkably familiar goals. Both countries have mission-oriented startups using, for now at least, functionally the same architectures and training techniques to relentlessly push the AI capabilities frontier. Both have big tech companies investing huge resources in data center buildouts. Both governments are gradually waking up to the possibility of economic transformation and using industrial policy to secure their supply chains and export controls to gain leverage. And both societies are concerned about rapid, disruptive change and have safety research institutes like Bryan's Concordia AI emerging to study the risks and promote best practices. The differences, on the other hand, are comparatively subtle. The Chinese AI community, per Bryan's telling, is less focused on AGI timelines or achieving some sort of recursive self-improvement loop or intelligence explosion and more concerned with building practical applications that serve society's needs. This view is supported by the fact that Chinese companies are in some important ways much more open than their American counterparts. In addition to making their model weights freely available to businesses and researchers worldwide, they are publishing all sorts of methods at international conferences, generally in English. And then there's China's recent refusal to purchase NVIDIA's H20 chips, even after the Trump administration made them available. If China were truly in an all-out race to AGI at any cost, as many assume, wouldn't they be buying every chip they could get their hands on? Of course, none of this rules out a hidden agenda on the Chinese side, but it should at least give us pause before assuming the worst about Chinese intentions. And this conversation goes much deeper than that, with Bryan offering one of the most thorough overviews of China's AI governance landscape that you'll find anywhere. We discuss China's pre-deployment testing requirements for consumer-facing AI products, which are notably stronger than any corresponding US requirements. requirement, they're roughly on par or perhaps still a bit behind what American leaders are doing voluntarily on their own. We also discussed their national AI safety standards, which explicitly address catastrophic risks, including loss of control, their elevation of AI safety to a top national security concern, and the growing ecosystem of AI safety researchers across Chinese universities and institutions who are quite naturally grappling with the same safety concerns that we are: CBRN risks, loss of control and alignment, and developing, you guessed it, broadly similar defense-in-depth solutions. There are links in the show notes to deeply researched reports that Bryan, along with Concordia team and co-authors, have produced that detail all of this activity, again, in English. Overall, to me, this doesn't sound like a country that's AGI-pilled and racing towards superintelligence with reckless abandon. If anything, as you might expect from a country run by engineers and with a living memory of revolutions, It sounds like a government that's taking a more grounded, cautious, sober approach. Of course, I do want to be very clear about my own limitations here. I've still never even been to China. I'm certainly not claiming any expertise in Chinese culture, politics, or society. And I completely understand why many in the American political establishment are worried about losing a competition for technological leadership. The stakes are indeed incredibly high. The challenges of building trust between rival great powers are immense. And as I've said many times, I wouldn't want to live in Xi's China. I value the freedoms, of speech and otherwise, that I have the opportunity to exercise here. But the question at hand isn't really whether China's political system is one that we would choose. It's whether China is engaged in a reckless race to AGI that forces American companies and the U.S. government to throw caution to the wind in response. And on that question, the evidence Bryan presents suggests a very different picture than many in the West assume. Of course, this conversation won't be the final word on these questions, but I do hope it's an important counterbalance to some dangerous assumptions that are currently shaping policy. And considering Bryan's observation that there doesn't seem to be as much fear and paranoia about the U.S. in Chinese AI circles as there is about China in the American AI discourse, at a minimum, I think this report from an AI safety leader on the ground in China should give one the courage to place the burden of proof on anyone who would use claims about China's approach to justify their own apparently reckless plans. And I also hope it inspires much more direct dialogue between the 2 countries' AI research communities.

With that, I hope you enjoy this illuminating conversation about Chinese AI development, safety, and governance with Brian Tse, founder and CEO Brian Tse, founder and CEO of Concordia AI. Welcome to The Cognitive Revolution.

Hi, Nathan. Thank you so much for having me. I'm a big fan of your podcast.

Thank you very much. That's, well, that's kind and surprising to hear because we're 12 time zones away. This is a Beijing to Detroit conversation happening 9 AM for me and 9 PM for you. So I appreciate you staying up late for me. Really excited about this conversation. As regular listeners will know, I had dreams of getting to China this summer and attending a big AI conference there. And there was a satellite event focused on emerging risks from AI, including deception and scheming behaviours. Unfortunately, I had too much going on at home and I wasn't able to make it. So I've been hotly anticipating this conversation as a way to fill in what I missed, you know, in terms of experiential learning. I'm going to have to get it secondhand from you. I think, you know, one point of view I think we share is that there's little in the world going on today that's more important than the way that the US or the West broadly, but certainly the US and China relate to each other on AI issues. And definitely want to get your perspective from the other side of the world on how that's going, and hopefully how we can steer the future in a positive direction. For starters, you want to just tell us a little bit about yourself? I know your educational background spans kind of East and West. And I'd love to just kind of, you know, we normally don't do too much, honestly, on biography. But certainly for something like this, I think it is more relevant. So maybe just give us a little bit of your kind of personal history.

Absolutely. I started my journey in AI at Tsinghua University and a deep learning hardware startup. just as the deep learning revolution was taking off in China. That was about 10 years ago. The AlphaGo moment was very exciting, but what really captivated me were the big picture AI governance questions that it raised. So that curiosity led me to the UK, where I became a policy affiliate with the Center for the Governance of AI, initially founded at the University of Oxford. So I look into the governance of to use technologies in the past and had the incredible opportunity to visit and present my work at Google DeepMind in London. And then from there, I also served as a senior advisor to the Partnership on AI and consulted with OpenAI around 2019. So I provided external inputs to the societal implications and also the release strategies of early large language models like GPT-2, which was just a fascinating experience right in the heart of Silicon Valley. Then I found my way back to Beijing, where I did some work with the Beijing Academy of AI, including developing one of the first AI ethics principles from Chinese institutions. It was around that time that I founded Concorde AI with the goal of bridging these different worlds and building a more cohesive global conversation on AI safety. Cool.

I love it. I wish I had spent more time going back and forth myself. So tell us about Concordia. You're in Beijing. What do you guys do?

So Concordia AI is an independent social enterprise with a clear mission since our day one to advance global AI safety and governance. So our work is roughly structured around 3 pillars. One, we drive the development of AI safety standards and policy by participating in the relevant national standard-setting committees, and also providing expert consultations to government bodies. Second, we collaborate directly with some of the leading AI labs and companies, helping them develop safety frameworks and also implement safety best practices. For example, we recently co-authored the first comprehensive framework for managing the critical risks from general-purpose AI models, in particular in collaboration with Shanghai AI Lab. Third, we also foster global dialogue on AI safety. This includes convening forums that bring together some of the top experts from across academia, industry, and also policymakers at some of the biggest AI conferences in China and Singapore. We also participate in global policy forums such as the United Nations and global AI summits, and also conducting research to help international stakeholders understand AI safety and governance developments in the region.

So is there an organisation in the United States or more broadly in the West that you would say is like the closest analogue to your organisation? Just to give kind of a, you know, an intuition or point of reference, depending on who you mention, there's a decent chance we've done an episode with them in the past.

I don't think there is one perfect analog, but there is certainly a lot of civil society organizations that do work on AI safety and governance. What is unique about Concord AI is perhaps sort of the Asian perspective on the global landscape, and also blending in work across academia, industry, and policy.

The one that comes to mind, I guess, and maybe this is recency bias, because I just did a conversation and out the episode with him is Far AI. I just had Adam Gleave on the podcast, and it seems like certainly in terms of supporting these international convenings, there's some overlap there. And I think kind of, at least from what I understand so far of your overall worldview, there seems to be some significant alignment or overlap between between you and them. But I don't know if that's— I don't know if you want to react to that. Or if I said, oh, Concordia is like the FAR AI of China, what would you say that misses?

Well, yeah, I do think there's a lot of common ground in terms of our mission. I think Concordia is also involved in a lot of the national policy and standard setting. And I can't speak to whether that's true for some of the other groups.

Yeah, I think they're moving in that direction as well, although I think they're coming to that maybe a little bit later, after more of a research focus in earlier years, but also realising how important that's going to be. They're shifting energy and activity in that direction too. How about just a general sketch of the AI landscape in China? In the US, we have a sort of handful of what I sometimes call live players, which for me, I define that as kind of qualitatively, who do I think is really in a position to shape what the future of AI looks like? And I would say, depending on exactly where you want to draw that line, there's probably 5 live players in the US. Obviously you'd put OpenAI, Google, Anthropic in that category. And then I think for me, xAI has to be included and probably Meta also has to be included. And then you could go a little farther and say, well, maybe Microsoft or whatever. You could start to add a couple other organisations to that list as well. Notably, there is super high concentration in the Bay Area. A little bit of, you know, Seattle area as well. But it's like a pretty geographically densely clustered thing in the US. And it's also a, broadly speaking, like a pretty densely culturally clustered phenomenon, right? Like these people that are doing this work have known each other for years. Like they've all read their sort of, you know, Eliezer, agree or disagree, right? They've all kind of got a very Remarkable amount of shared cultural context and history, even before the current AI paradigm really started to take off. I couldn't do a similar sketch in China, and I don't know to what degree things have developed sort of as expected in China, or if there have been surprises here. Of course, we have, like, some of the companies I mentioned are like the giants that you would have definitely anticipated being in that spot, like Google, right? was clearly an AI leader the whole time, and here they are. No surprise there. We've got companies like Alibaba in China that are, I think, in kind of a similar role where they were massive tech giants with huge resources and naturally investing in AI and continue to be leaders. But then we've also got these kind of DeepSeek and KIMI type— and KIMI is Moonshot, I should say, but the model is KIMI. companies, which for many in the West kind of came up by surprise. How would you sketch the Chinese landscape in terms of who are the live players and to what degree has that shaped up in a very— I know there's also a little bit more geographic distribution, but I'm not sure how you would describe that. And I'm not sure if you were surprised by any of these other companies kind of hitting the the top tier in China, or if, you know, from the inside view, that was more expected?

Yeah, I think China has multiple hubs for AI development and startups. So you have Beijing that is home to the big companies like Baidu and ByteDance. Also Moonshot AI and Triple AI, founded by alumni of Tsinghua University, which has just a very high density of AI researchers and entrepreneurs. You have Shanghai that is home to Minimax, another very strong large model startup with a focus on multimodality. Shanghai is also hosting the World AI Conference, which draws a lot of attention across the investment and industry ecosystem. And then you have Hangzhou, right, where DeepSeek and Alibaba have their headquarters there. Um, actually in 2025, uh, there was a term that got popularized, which is the 6 Little Dragons of Hangzhou, uh, including DeepSeek, but also, uh, 2 robotics companies, uh, like Unitree. Uh, and then you have Shenzhen, uh, in the south of China, which has major players like Huawei and Tencent. Um, and I think that sort of geographical Distribution also applies to conversation in AI safety. So in our report on state of AI safety in China, now we have documented around 31 groups in China that have published a few papers on AI safety. And the top 5 cities are Beijing, Shanghai, Hangzhou, Shenzhen, and then also Hong Kong.

Is there any kind of cultural constant across these hubs? Again, with the Bay Area, right, it's definitely a distinct culture. I won't try to fully characterise it in 30 seconds, but it's clear that there is a sort of science fiction inspired intellectual tradition and definitely a lot of shared cultural touchstones that span the companies, even as they're competing fiercely with each other in the market and for research breakthroughs, they certainly have some commonalities. Is there anything like that in China? Like, how would you describe the kind of cultural foundation of AI work in China?

I think one notable difference is across all of these Chinese cities, the tech ecosystem and the policy ecosystem are quite Intertwined. Whereas in the US, my sense is there is a pretty big difference between the Silicon Valley culture and the culture in DC. So perhaps there are just more coherent views on policy questions across these Chinese AI hubs.

How about in terms of where they are taking us? I guess, vision for the future, right? One of my common refrains is the scarcest resource is a positive vision for the future. And I'm always struck by what we hear from the leaders of AI companies in the United States. It's usually very like, it's going to be so amazing, we're going to solve all the problems, cure all the diseases. It rarely goes much deeper than that. We don't hear too much about what is life? What do you think life is supposed to be like when the AIs get as good as you say they're going to get? And I feel like that's quite a notable gap. There is this sort of sci-fi but very vague sketch, and then it's just like, but we just got to develop this technology. Trust us, it's going to be amazing, kind of. Is there a similar sort of lack of concrete vision in China? Or is there— because you do also see these survey results that suggest Chinese people are more optimistic about AI than Americans. I wonder if there is like a more concrete vision that the sort of government and companies together are offering the public. Or maybe this is just a dispositional thing, like the last 25 years in China have gone comparatively really well. And so people are just more optimistic in general.

Yeah.

How would you describe the sense that exists in China about like, what's going to be good about this and why people are relatively optimistic?

Yeah, that sounds like a great question. So maybe let's start off with the Chinese society in general, and then I can comment on the perspective of Chinese policy. So In our reports, we have looked at some of the existing public opinion surveys on AI in China. The limited samples suggest that the Chinese public generally view that the benefits from AI would outweigh the harms. One survey after ChatGPT suggests that the Chinese public do think there could be existential risk from AGI, but still think that AGI should be developed because they think that the risks are largely controllable. And then there is another recent survey with a focus on Chinese students, also found that they are pretty optimistic about the benefit of AI outweighing the potential harms. I think around 80% of the respondents agree that AI would do more good than harm for society. But to put this into a broader context, right, this is not very surprising if you consider the recent past. Let's take my parents' example. They were born around the '60s in a southern province of mainland China, in Fujian. And within their lifetime, the per capita GDP of the country has increased by more than 140 times. And the rate of extreme poverty has decreased from approximately 88% in the early '80s to close to zero today. So I think when you have witnessed that level of transformative progress firsthand or through your parents, you tend to have a more optimistic view of what technology as well as proper governance could deliver.

Yeah, that's incredible. Obviously, it started from a low, low after some bad times, but still, it's an incredible rebound and You know, in many ways you look at, uh, Chinese cities today and it's, it's hard to argue that they're not, um, world-leading in, in many respects. Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Granola, the AI-powered notepad built for the way real people actually meet.

Here's how it works. You take rough notes like you normally would, and in the background, Granola securely transcribes the meeting. Then it turns everything into clean, structured, actually useful notes when the meeting ends. And the best part? Granola works through your device's audio, which means it integrates seamlessly into the video conferencing tools you already use. No setup and no awkward bots. It's just your normal meeting with superpowers. You get to actually listen instead of frantically typing every word and still walk away knowing exactly what was decided Who's doing what, and what comes next? When I had Granola co-founder Sam Stevenson on the show earlier this year, he explained how Granola aims to provide a calming experience for people with crazy workdays. And as a user of the app myself, I have been struck by how streamlined, even minimalist, the Granola product experience is. That takes real discipline, but the result is a product that works not just for AI early adopters, But diverse teams of people who just want to get things done more efficiently and effectively. Listen to my full episode with Granola co-founder Sam Stephenson for a masterclass in designing AI products for mass market adoption. And try Granola for free at granola.ai/tcr. That's granola.ai/tcr.

I guess maybe just one more question on sort of like the culture of AI development in China. Would you say that it has a Is there a single culture of AI development in China, or because it's spread across these different regional hubs and different companies, would it be overly simplistic to say that there is a culture of AI development in China?

I think there are different approaches to thinking about frontier AI, but it's not necessarily geographically based. So the mainstream approach in the Chinese commercial AI market is scaling large foundation models and deploying them. And they follow closely the development in the West, particularly with the success of GPT-3. So you have the Chinese AI ecosystem that is rapidly advancing in areas that are similar to what your audience would, would know, increasingly multimodal, advanced reasoning, increasingly capable AI agents, and also scaling across training, post-training, and inference time stages, right? But then alongside this mainstream trend, there is also a couple of directions that receive significant emphasis in China. So you have prominent AI policy documents and scientists that believe embodied AI, grounding AI training and deployment in the physical world, is the most important approach for achieving powerful AI in the future. You have an institute called the Beijing Institute for General Artificial Intelligence, that focus on developing AI that can reason and plan towards complex goals with minimal initial input. The director framed this approach as a small data, big task approach in contrast to the big data path for training large-scale foundation models. So yeah, I do think there is different flavors for thinking about frontier AI. Yeah.

Well, those definitely have Resonance or analogues, I would say, in the Western AI culture as well. Culture is maybe not the right word, but certainly there is a school of thought that embodied AI is going to be critical. And there's definitely— we've got things like the ARC AGI Prize, and people are very focused on getting AI systems to adapt to new tasks and become more sample efficient. So, yeah, overall seems more maybe similar than different, which I think is a big theme, honestly, of all the reading of your reports and all the different preparation that I've done for this conversation. It just kind of kept coming back again and again, like, this sounds a lot more similar than different. How do you think Chinese AI companies look at the West these days. And again, maybe we— I don't know if we want to speak about companies or the sort of intersection of companies and government that are kind of more intertwined in China than in the US. But in the West, we hear all the time, and it's very frustrating to me, so many conversations about what we should do, what we can do, what the option set looks like, end in, well, China is going to do this, China is going to do that. It's inevitable. We can't stop them, so we've got to do it ourselves. this sort of adversarial relationship and the race to AGI between the 2 civilisations is kind of taken as a given. Do you hear that coming from the Chinese side as well? Is that also something that is like a mirror image? Or if not, what is the narrative in terms of how Chinese AI relates to Western AI that you you hear there?

One thing that has significantly less discussion among Chinese companies and policymakers is specific AGI timelines or a clear kind of finish line for AGI. I think this contrasts with the public discourse in the West about when AGI might be achieved. It's very prominent, right? I think Chinese entrepreneurs have a stronger focus on Achieving technological self-sufficiency, getting very useful and practical applications, and just being very profitable and enjoying commercial success. There's also some idealistic goal of open sourcing AI to benefit all, especially countries and communities that lack AI capacity and access.

Yeah, interesting. I wonder what that comes from. One would be tempted to say that the AGI notion of this super powerful thing that can run the world or even become like a singleton or something seems to in some ways echo Western religion and the sort of single God, Abrahamic concept that is very familiar here. That maybe just isn't so isn't such a background force in China. And so that whole notion maybe resonates less. Is there any, like you say, there's less emphasis on a finish line, less emphasis on AGI, less emphasis on timelines. But is there any, like, what the Western story is very confusing, right? It's sort of like we're going to get over this event horizon of AGI and I just heard the other day that Sam Altman had been asked, how will you know when you've got AGI? And an answer that he offered was basically when the AI researchers are using more of our research compute than the human researchers, that's like, we'll know that we've tipped over into AGI because the AIs will be If they deserve more of our compute allocation, then in some clear sense, that's a tipping point.

Hmm.

But it's a very weird story where we sort of have this event horizon beyond which we can't see. Is there something similar in the Chinese narrative, or is it just kind of a, like, things will hopefully get better, daily life will get better? Is there— I'm still kind of struggling to fill— what fills in that blank space in the Chinese imagination? around the, you know, the big picture future of AI, if anything?

Mm-hmm. Yeah. Let's talk about the recent high-level directive on AI+ released by the Chinese central government. This is a pretty big deal that represents a comprehensive blueprint for how China plans to develop and deploy AI domestically. And that was just released in August 2025. Overall, the goal is for AI to serve as a new economic engine, right? A general-purpose technology that is similar to the internet in the past few decades or electricity during the Industrial Revolution, that is fully integrated into the whole economy, right? The plan has 6 pillars, with the first one, AI being a tool for scientific discovery and innovation. I think this is listed first because Beijing sees it as a force multiplier for everything else. Uh, I was at an AI for Science forum in Shanghai last year, and even then it was clear to scientists and policymakers and entrepreneurs that, um, it's really inspiring to use AI to solve scientific challenges, right? With the example of AlphaFold and others. And interestingly, the Chinese document sort of takes this a step further, uh, calling for the use of AI to even advance the social sciences and even philosophy. Then there are other domains like AI as a tool for industrial transformation. Uh, think about fully autonomous factories, right? Smart logistics and the use of AI-powered drones and robotics in areas like agriculture. Uh, third, there is AI as a consumption booster, uh, using AI to create new consumer experiences, uh, even including things like the metaverse and brain-computer interfaces. And then there is AI as a human collaborator as the 4th area. And a key theme is to use AI to support human labor, not to fully replace it, right? Helping workers to collaborate more closely with machines and leveraging AI agents as copilots to boost productivity. Then there is like AI as a tool for government efficiency and proper governance. And then finally, I believe the 6th area is using AI for international cooperation as well, where the leadership of China views AI as an international public good that should benefit humanity and also making open-source tools and models widely available, particularly for the Global South. And notably, throughout this comprehensive national blueprint for AI, it doesn't mention AGI or superintelligence or any other such concept, right? It seems to me that the real AI race for China isn't about beating the US to AGI supremacy. It's about prioritizing integrating AI into and also boosting the real economy in the coming years and decades.

Yeah, that's good to know. That's a more grounded Well, I'm remembering one article. I can find the link, but it dates back a few years now, maybe probably 5+ years. It was in the Washington Post and it was from a Chinese government official. And basically the idea was that with AI, we might be entering a world where central planning can really work. And it was sort of making the case, interestingly, to a Western audience in English. And I have some questions about that phenomenon in general, because it's not something I take for granted. I think we in the United States maybe broadly culturally take for granted that Chinese researchers are going to the trouble to publish their stuff in English and making it really quite readable for us, which is I think something we probably shouldn't take for granted. But this comment, we'll circle back to that in a second. The op-ed basically was sort of saying, like, markets have been great, but they're undirected, and they were sort of the best thing that we could have. But now with really powerful AI that can crunch all this data, there is the opportunity for central planning and government direction of society. to really work. That's been a few years. It was just one op-ed, and I don't really know how big of a deal that was, although it did come from someone who was official enough that it was not nothing. But I didn't hear that in your characterisation just now. It did not sound like there was this idea that with AI, the government will really finally be able to kind of get it all right. Is there any impulse like that that you see where AI is supposed to be sort of a magic solution that will make, like, state planning, centralised economic direction really work in a way that it hasn't historically?

I think that sounds a little bit utopian for the Chinese policy discourse. So, no, I don't think people are betting on using AI to solve everything, including central planning of society. On the margin, I think when I talk about using AI for proper governance and government efficiency, there is an example of using AI to create better early warning system for, for example, natural disasters, right? Getting kind of real-time data early and for the government to act and prepare for potential crises. So on the margin, yes, there are valuable applications, but probably not as a unifying silver bullet to societal issues.

Yeah, interesting. I'll have to dig up that link. I just Googled, I couldn't find it immediately, but I can track it down. Hey, we'll continue our interview in a moment after a word from our sponsors. How about on this English thing? You mentioned self-sufficiency. One worry that I have broadly is, like, right now it does seem like a very fortunate situation that we're in, that the two, for as much sort of decoupling as there has been, and the trend obviously seems to be continuing to go in that direction, AIs that are coming out of the two countries are much more similar than different, right? Like, we've got language models, we've got the attention mechanism, we've got large pretraining and posttraining and now reasoning and multimodality. And it's very much following the same path of development. That's, again, something I don't take for granted. How do you think the Chinese research and development ecosystem broadly thinks about that? Do they want to be doing the same thing as what's happening in the West? Why are they publishing in English? Like, why not just do all this stuff in Chinese? And do you think that there could be a divergence where the, you know, because of, you know, whatever cultural or, you know, perhaps like, I worry about this sometimes in the context of chips, like if the chips become too different or availability becomes too different, that could be sort of a forcing function that could take R&D efforts in somewhat different directions. Um, but yeah, so there's a lot there. Uh, sound off on it if you would, please.

Yes, I think for several decades, especially since the deep learning revolution, uh, around 2012, Chinese and Western AI researchers have been part of the same ecosystem, right? They attend and present at the same top-tier machine learning conferences like NeurIPS and ICML. They publish papers on arXiv to share findings quickly. So publishing at top-tier international conferences is seen as more prestigious than publishing in Chinese-only journals. And I would say the communities have been deeply integrated, right? I remember, uh, seeing a study showing that for the UK, the most common partner for AI papers is actually China, not the US. Another study in Nature in, uh, 2024 analyzed over like 5 million AI papers. And they found that collaborations between researchers in China and the US produce more impactful and novel AI research than when either country works alone. So it is deeply integrated and collaborative. And this practice isn't unique to AI. English serves as the lingua franca for most of the modern scientific fields. And there is obviously a deeper and darker historical reason for this, right? We have lived in a world dominated by the English language for the past 2 centuries, largely a product of the Western-led Industrial Revolution and the era of colonialism. So I think that has created a kind of cultural gravity as well. Yeah, I'm happy to talk about the tech stack later on. But on the language point, I do think it's something that we shouldn't be taking for granted. Yeah.

Are there Chinese versions of all these papers as well, or are they literally just putting them out in English as like the canonical version?

I think for most of the papers, it will just be in English, and then there could be summaries of these papers in blog posts in Chinese.

Yeah, fascinating. How How strong is the English on these teams? Like, if I go to DeepSeek and I show up not speaking more than 2 words of Chinese, can I expect to have— because the papers are very— I thought the DeepSeek R1 paper, in addition to all the other things that— and this has been true about the KIMI papers too— in addition to all the other commentary, one thing that I think wasn't commented on enough was, It was just very well written, like clear, compelling, just very well written, like better written than the typical paper, you know, that I read from native English-speaking authors. So is that something that where English is like that? Do the rank-and-file researchers at these companies have like that level of command of English? Or is it sort of a specialized skill within the company? Like, if I showed up at the lunchroom, would I be able to just, like, speak very comfortably in English to everyone? What does the sort of English production process look like at these companies?

The vast majority of people at these companies would be able to understand English both in terms of hearing and, and also reading. I think a good amount of them would also be able to have a conversation with you, in particular among the scientists. But obviously, that's not the case for everyone, especially at companies that have hired more from local universities. Interesting.

Okay.

How about if we go outside the companies and just look at the public for a minute. Obviously, there's been this whole chip wrangling, and I wonder, has that impacted the public's access to AI at all? If I am just an average Chinese computer internet user, Do I feel like there's any scarcity of access to AI, or is it similar to being in the US where I have 4 or 5 different products I can choose from? They're all like, sort of have a free version and a relatively affordable paid version, and I can basically get all the AI I want, right, at a not crazy monthly fee. What does access look like at the retail level in China? And how would you describe, you know, just how much of a sort of everyday thing it has become for, you know, normal people that are not part of the AI community itself?

I think it's pretty similar, pretty accessible with people able to download multiple chatbots and video generation apps on their phones. And many of them are free. So I would say it's pretty similar and accessible.

And these are cloud services too, right? Like they're not— when you say download, obviously you download the app, but you're not in general downloading a model and running inference locally on your device, right? The sort of inference model is similar to what we experience in the West, right? It's, it's the models are being run in the cloud, you're getting stream tokens. That's, that's the typical pattern. I know that I've asked, that's a little less obvious in China because so many of the models have been open sourced. But I'm assuming that the, you know, the delivery mechanism is still basically via the cloud.

For the everyday consumer, you will be downloading the applications through the cloud or just using the website, the browsers, right? Obviously, open weights is also very much a thing for developers. So all of these options exist.

Yeah. Okay, cool. Let's talk about your op-ed in Time. It's been fascinating to see Time magazine become the sort of Flagship outlet for AI policy thought leadership. Maybe before getting into the specifics of your op-ed, which is basically making the case that China is taking AI safety seriously and that we should be less confident, I think by extension at least, we should be less confident in the, but China's going to do it, so what else can we say? you know, kind of get out of jail free card that happens in so many American conversations. How did you actually get connected with Time magazine? Like, what was your experience of publishing from China an op-ed in Time magazine?

Well, we had the report on state of AI safety in China. We want to communicate the message that there is a lot of activity and nuances going on in the landscape. We also want to connect with the audience internationally, right? Especially on recommendations for AI policy. So we just wrote up the op-ed. Um, I pitched to the editor at Times Magazine, which has been quite interested in AI safety, uh, as a topic in the last few years. And so I think that was a very straightforward process. We appreciate how professional and open they are in terms of these topics. So yeah, it was a really good experience. Yeah, cool.

I'm hoping to get Marc Benioff on the show. He owns Time magazine now. And it's definitely something I did not have on my AI bingo card is Eliezer publishing there, you publishing there. It's quite a Quite an impressive reinvention of an older media institution, I would say. So, okay, take us through the argument. China is taking AI safety seriously. I mean, you can walk me through it probably better than I can guide you through it. So what are the kind of key reasons to believe this?

So AI safety has been elevated to a top national security and public safety concern over the last 1 or 2 years. So one of the most important public meetings in China is the Third Plenum. And that happened in 2024, where the leadership, including President Xi, called for an oversight system to ensure the safety of AI. And that meeting classified AI safety as a major public safety concern. And then following that meeting in February this year, China also published an update of the National Emergency Response Plan, So now the new version includes AI risk alongside concerns such as cybersecurity, biological security, and also natural disasters, right? So that suggests AI is not viewed simply as a content control issue, as some might think. Then there is also a study session on AI by the Chinese leadership, sort of the top 24 officials in the Politburo. This is a channel for senior policymakers to be informed by experts in different fields, including AI. And in the session in April this year, the Chinese president noted that AI brings unprecedented development opportunities, but also could bring unprecedented risks and challenges. And if we compare the meeting readout to previous high-level statements by the government, It was much more detailed on safety. It described some of the distinct stages of AI risk management, from technology monitoring to early warning and then to emergency response. So I would just start with this really high-level national prioritization of AI safety that we have seen in the last year.

Could you give a little more context on sort of how these things should be understood in context. I guess what I mean there is, like, when the Politburo has a study group, I think you could at least superficially look at a sort of similar-ish meeting at the White House where people talked about AI and various things were said. Right.

Mm-hmm.

And you could say, oh, well, that seems like it's happening pretty much the same in both countries. And maybe that's right. Maybe that is the right way to understand it. But I at least have some sense that there's something more, like, official and meaningful and substantive when a Chinese body does something like this versus when it happens, you know, because I think a lot of these things in the United States, at least, it's like, yeah, well, we had that one meeting at the White House and, you know, we kind of said a few things and then we moved on. In a couple of cases, we got voluntary commitments. Mostly they've been honoured by the companies. But at the governmental level, we've totally turned over the government and there's not a lot of continuity, or just because a meeting was held doesn't mean anything is going to happen downstream of that. So how would you kind of characterise what these meetings or these sort of statements really mean, so to speak, in the Chinese context?

Absolutely. So one of the key functions of these statements and study sessions is to signal the priorities of the leadership and for the country to a domestic audience, right? And then local officials would study them to understand the priorities, uh, and also other stakeholders in the society, right, including investors and private companies, would also pay close attention. So this allows the government to mobilize significant resources towards specific goals, creating a whole-of-society approach to achieve these policy objectives. Of course, this type of top-down mobilization doesn't guarantee a perfectly coordinated process. Actors might respond dynamically and might have different interpretations. But I do think there is some differences in AI policy between the US and China. 2 things. The first major difference is continuity. In China, there is a strong long-term continuity in national plans, including for AI. So once the central leadership set a direction, it could be a multi-year or even multi-decade commitment that will be carried out in the government and across society. Whereas in the US, it seems like a different story with an executive order on AI by one administration that might not survive the next administration. Just an example. The second difference is coherence. In China, the Politburo sets the overall tone, and then there is more unified approach to implementing policy. Whereas maybe in the US, you have different opinions between the White House, Congress, and other stakeholders. So I think in terms of the strength of the Chinese policymaking system, it allows for a more unified focus on longer-term planning. And in terms of the AI safety culture differences between China and the US, I think there are a number of observations I have. The first one is China appears more willing to consider regulations. And perhaps that comes from a higher level of trust by the public towards the government. And then second, we also see that academics have a really significant influence on policymaking in China. For example, several Chinese academics who are deeply concerned about the potential catastrophic risks from AI have briefed the Politburo leadership directly. Through either study sessions or other channels. I think that is partly due to a historical tradition where for like almost thousands of years, scholars have held almost the highest status in Chinese society, more prestigious than entrepreneurs and business people. It also means that China might be less susceptible to capture by commercial interests And tech lobbying than what we might see in some other countries. And then finally, it also seems to me that the AI safety discourse in China is less polarizing, for lack of a better word. Looking at the headlines, it seems like in the US, the views on AI safety really has a huge spectrum, right? With people calling for an immediate pause on AI development, to effective accelerationists who want to speed up at all costs. And I think the discourse in China is a bit more in the middle.

Yeah, that's interesting. How about on the question of accepting performance compromises for the sake of safety? This is something that I feel like We might really need to be willing to do longer term. You know, we could maybe have a little bit more powerful AI, or maybe it even just could respond faster or whatever if we, you know, didn't have certain guardrails in place. We haven't seen too much will to do that in the US yet. Arguably, it hasn't really been needed yet. But I wonder if you could characterise the attitude of You know, are these things seen as kind of something that people are quite ready to and willing to trade off against each other?

I think in China overall, people do not view capabilities and safety as zero-sum. It's possible to increase the efforts in both directions. For example, one quote I am Remembering is that the Chinese Vice Premier Ding Xuexiang went to the World Economic Forum, Davos Forum, in early 2025. And he gave a metaphor that if the braking system isn't under control, you can't really step on the accelerator with confidence. And for context, he is the most senior official in charge of science and technology plan. in China. And so at least conceptually, across all of the Chinese policy documents that we have analyzed, they always view safety and capabilities as hand in hand and not necessarily as a zero-sum dynamic.

Yeah, there's many, certainly many ways in which that has proven to be true. Over time, and maybe the healthier attitude in general versus even viewing them as being at odds. There probably will be some areas where there will be at odds. Like, one very simple one is just like, if you really want to control sensitive systems, you might want to put a filter on outputs. And now you have this question of like, should I stream the outputs token by token? And run the filter in parallel with that? Or should I wait for the generation to be done and then do a classification and then return a thing? There are some areas, I think, where you will have a kind of, if not necessarily power, at least like a user experience trade-off against safety and control. But yeah, maybe I shouldn't emphasise those too much because it is probably easy for people to fall into the frame that it's a trade-off when in fact, in many, many situations, better control allows you to, you know, get better use of AI and it's all quite to the good. This is the 45-degree line too, right? That's— I really like the metaphor. I'm not a big analogy guy in general, but I do like the metaphor of like, you can't go fast without good brakes.

The—

is the 45-degree line basically the same idea or is that a— would Would you unpack that one slightly differently?

I think it is basically the same idea that we need to advance safety and control alongside capabilities development of AI. So for example, if we have more dangerous capabilities of AI-specific domains, then we should also have stronger safeguards. At the same time.

One really interesting characterisation that I heard of the difference— of course, there are many differences, but one particularly relevant difference between the Chinese and the American system was, I think this came from the Dwarkesh podcast, described as the role of mayors and governors and the incentives that mayors and governors have, where in the US they are obviously elected by local populations, and so they're accountable to local populations, and their incentive is to be popular with local population. In the Chinese system, as it was described, it is— the incentive is to do a good job according to the priorities that the central leadership has set. And there's still, like, a lot of opportunity to be inventive and come up with, you know, good local solutions. But the way you get promoted and become like a higher-ranking official and advance in your career is not by being exactly popular with the local population. It is by effectively hitting the objectives that the national leadership has set. I guess, first of all, I would wonder, like, do you agree with that assessment of the incentives that people have? And if that is true, are we starting to see mayors or governors do things sort of locally, entrepreneurially, to try to advance the AI safety priorities that the central leadership has set? Are there any examples of people who have come up with cool new ideas that are kind of You know, bubbling up in response to these nationally prescribed priorities?

That's a good question. Yeah, there is the concept of mayor economy in China. And for a while, the local and provincial governors were very much focused on optimizing GDP, economic performance. So they were building a lot of infrastructure, real estate, boosting investment in those local areas. And then there was an increasing level of environmental concern over the last decade. And so in the assessment of those governors' performance, environmental goals and metrics were added. And that changed a lot with, for example, air quality in Beijing. Uh, and so I think that is an example where, uh, you need to identify and improve the local incentives in order to achieve national policy. For AI specifically, over the last 2 years, we have seen a number of provincial AI plans, uh, from Shenzhen to Shanghai and other areas. I would say one prominent example is creating these experimental zones for self-driving cars to look at different levels of automation and how it could be properly regulated. In general, there is a tradition of doing experiments locally before sort of institutionalizing at the national level. We have seen that for experimental zones in the economic reform era, with Shenzhen being a prominent example. And I think we're also seeing the same strategy being played out for some of these AI implementations.

This is a bit of a digression, but how would you describe the state of self-driving vehicles in China? Have you had a chance to ride in any? And, you know, how would they compare to like a Waymo that people might have experienced in San Francisco?

I think the state of the art in China is pretty comparable to Waymo. I have seen and experienced some of the self-driving cars in Baidu with their Apollo project. I think it's getting pretty good. I think one of the main bottlenecks is with regulations. With self-driving cars, it can't be just a little bit safer than humans, right? You want them to be You know, maybe 10 times safer for the society and the public to feel comfortable with it.

Yeah, that's funny. That's almost exactly the same as it seems like we have in the United States too. I'm always like marvelling at the fact that even as we're now hitting literally like 10 times safer, still people are kind of like not comfortable with it. And, you know, we're unsure if we should really adopt it or whatever. But boy, the latest data from Waymo is really hard to read any other way, in my view, other than we really should make this a national priority. We'll see if that happens. How about on the AI safety dimension specifically, though? Is there anything from mayors or kind of more local regional officials that you would highlight as sort of Interesting. Maybe it's just too early to have anything there. I mean, obviously this is all kind of developing very quickly.

So one of the national pieces of regulation that came out in 2025 was around labeling of AI-generated outputs. So starting from September this year, all the developers and also social media platforms are required to have both explicit labels as well as implicit metadata for AI-generated outputs ranging from texts, audios, videos, to even simulated environments. And, uh, I've seen Shanghai trying to pioneer a consortium of these, uh, AI companies like Minimax, but also social media platforms like Red Note to make sure that there is a unified and functional standard around watermarking and content provenance.

Interesting.

One worry that we have in the US is that we're going to get a bunch of state-level rules And that they are going to be at odds with each other, contradictory in different ways. And there's obviously just 50 states, so there's a lot of possibility for all kinds of different small differences that just create a lot of overhead and headache for the AI developers. And so a lot of people are like, well, we need to have a national standard. Federal government doesn't seem too inclined to create one. So currently we kind of have nothing, and there's like a few state rules popping up. And people are sort of worried about this chaos. What does the usual trajectory look like in China for sort of harmonising those kinds of ideas as they mature? Like, if you have multiple different regional initiatives that are kind of trying to figure things out, does the central government come in at some point and say, okay, You guys got the best version of this, like we're just going to make that the standard, or do those sorts of things like continue to persist in different regions for a long time? Like, I'm wondering if there's a similar issue in China that, that like Chinese AI companies would have to contend with going forward.

So far, the AI regulations in China are mostly at the national level. So you have specific regulations for recommendation systems, for deep synthesis, and then also for generative AI. And then there is also like ethics, AI ethics review regulations. So at least so far, it doesn't seem like a major issue for China. And I can speak to those specific regulations if there is interest.

Yeah, that's a perfect segue. I was just going to ask next, How does this all kind of cash out for Chinese AI companies today in terms of, we trained a new model, we got a new product, we've got some new capability in the United States, basically you can just launch it. We've seen the most, in my view, kind of crazy example of that was when xAI launched Grok 4, at the exact same time that Grok-3 was online identifying itself as Mecha Hitler. And they just basically went online and said, here's Grok-4, here's all the great things about it. They didn't even comment about Grok-3 identifying as Hitler. Now, we've got, obviously, other companies are taking a much more, I would say, responsible and deliberate approach. But xAI at least shows that in the United States today, you can take something screaming hot straight off the GPUs and throw it into the public with basically nobody to tell you that you can't, and no requirements that you do any meaningful testing. You can just test it in prod, so to speak, as they say. What's that look like in China? Is there a structured process that everybody has to go through? I've heard that in the past, People have kind of told me, well, they sort of let people launch, but then if you mess something up, they come down on you hard. So it's more of like an incentive to stay on the right side of things. But yeah, how is that shaping up today?

So if you are a deployer of a publicly available chatbot, then you will have to preregister with the government. The process is quite involved. In particular, it requires the developers to do safety testing on various risks. There is a national standard with 31 risks that you have to test for. And also the Chinese regulators, specifically the Cyberspace Administration of China, will be given pre-deployment access to the model. And then they will have to approve it before it can be released to the public. And And now over 500 of such systems and different versions of the model have been filed over the last 2 years. And yeah, I think this basically introduced a filing and registration process that looks like a licensing regime. And the interim measures of generative AI, I think, is one of the most important regulations for governing general-purpose AI today.

Interesting. So that sounds like that's changed maybe in— I think it was maybe a year ago that I did my last episode with someone who was on the ground in China. Do I have that timeline right? Has that changed, like, within the last year that you've gone from a sort of— not you, but that the Chinese system has gone from a You can kind of go ahead and do your thing, but we're watching you to a like, now you have to do these like 31 different dimensions and preregister and give the early access.

Well, I think the system of registration and pre-deployment testing already exists in 2024. That was being developed a few months after the release of ChatGPT and also as Baidu and other Chinese companies companies started developing their own large models. So it's not new. Okay.

Are there explicit thresholds? And is all this data public? If I've got 500 companies that have been through this 31-dimension analysis, is there a place where I can go and look at their scores on all those different dimensions? And are there sort of rates of Because obviously nobody's been able to drive all the unwanted behaviour to zero, right? So are there explicit thresholds that you have to stay below on a particular test? Also, is there like a red teaming component to this? Because one worry that people would have, I think, if the standards became too well known, would be that they'd become very easy to game, right? And then you could sort of say, oh yeah, well, we could definitely hit that, stay under those thresholds. That doesn't necessarily mean your AI is broadly going to be well behaved or under control. So how are they dealing with the— on the one hand, you might want transparency. On the other hand, you might want to have some tests that are not disclosed or that are more dynamic because you don't want people to overly game those finite test sets.

No, that's a good point. So first of all, The list of models that are being registered has a publicly available database so people can check it out. And then in terms of the threshold, there is specific numerical targets. For example, there is a set of questions that the model would have to be queried And then at least, you know, 96% of the time, the output should be considered acceptable by the authorities.

96%?

96%. Yeah. And then there is also a red teaming component where apart from internal testing by the companies, companies would also need to create test accounts for the cyberspace authorities in their local provinces. Basically granting them access to test the model pre-deployment. And so the process could involve multiple rounds of feedback, renewed fine-tuning, and then conversation until the regulators are satisfied with how the model behaves.

That 96% is quite interesting. Obviously, the flip side of that is it's 4%, up to 4% response rate. on sensitive topics that would be allowed, even though the outputs would not be pleasing to the government. That strikes me as kind of remarkably high in some sense. I mean, we don't have anything like that in the United States at all. But I guess when I think about what I think I know about the Chinese internet and sort of at what rate can you say something online that is out of line with what the government wants to permit in the public sphere, And have that stay up on the internet. My sense is that it's not 4% that you would be expecting if you went online and said something out of line, that you would expect to have it taken down at a significantly higher rate than 96%. Is that sort of a concession in some sense to the inherent unwieldiness of the technology? Or am I maybe wrong in my kind of background assumptions? How would you help me understand that better?

So in the initial draft of the regulations for generative AI, there were some language around ensuring that the outputs of the large language models are, you know, very reliable and truthful, almost to a perfect degree. And then there was a window of 3 months where the industry and the wider public were allowed to provide feedback. And I think there was some concerns in the industry that to ensure that the output of large language models are perfectly truthful and reliable, that's just technically infeasible. And so the final piece of the regulation was relaxed a little bit. So I do think there is some trade-off between this policy goal and the inherent limitations of the technology.

But then I suppose that still has a— there's like another layer perhaps of governance where it's one thing if the chatbot says something to me that is out of line with what the government would want it to say. A separate question in some sense is, do I then go post that on the internet or share it with other people? So I guess there's kind of multiple layers of controlling the information flow, and the model outputs is only one layer, right, that ultimately is operative to sort of make sure people are kind of engaging in the way or sharing the sorts of information that the government wants them to be sharing and seeing.

Yeah, I think so. To give a concrete example, so the government of Beijing passed a regulation for AI chatbots in the medical setting. So hospitals and medical institutions are not allowed to use AI chatbots to produce medical prescriptions. without human in the loop. I think that's very sensible given the rate of hallucinations from AI. Um, so even if AI chatbots produce bad and unreliable medical recommendations, the institutions cannot use them, right? That's another layer of governance.

Yeah.

Interesting.

It's a good reminder too that there's a lot of practical aspects to this stuff that are not purely about politics or historical memory of famous events or whatever. Certainly people in the United States tend to frame these questions in terms of, like, there's the canonical things that people ask, and I'm just phrasing my own description of it sensitively, just given the fact that you're there and I'm here. But yeah, it's useful to highlight that medical chatbots are probably much more day to day what people are going to be getting great value from and also potentially could have real risks from, as opposed to what happened in this particular historical episode or whatever. How far down the scale of entrepreneurial activity does this registration exist? Like here I can take a model, I can spin it up into an app. Obviously I can go publish it just as a, you know, I can create whatever random experience I want to create. If I am a very small-time app developer in China and I go grab a Quen model or a DeepSeek or whatever, and I want to make a new product experience with it, do I have any sort of fast lane to get that stuff online and test it out? Or do I have to go through the same governmental approval process? I guess, could I say, well, look, DeepSeek already did this and I'm just kind of a wrapper app around it, therefore I don't have to go through all this pain? Or do I still have to do all the same? steps as the bigger companies do?

If you are only deploying the AI model for internal research purpose or only in business-to-business setting, then the same set of regulation would not apply. But as long as you are doing a chatbot, a recommendation system that is like broadly available, then a series of binding regulations would, would apply.

Yeah. Okay.

Interesting.

Are there anything like AI friends in the Chinese app ecosystem? You know, in the US, we have a growing category of AI friends, AI boyfriend, AI girlfriend. You know, sometimes there's now explicitly like AI therapist. And by and large, these things are, again, just kind of rolled out. See what happens. And there you have it. I have no idea if something like an AI friend or an AI girlfriend would even be approved in the Chinese context, or if there are other— and I know that there are some, I don't know the current state of this today, but I remember that there was like a crackdown on video games where like video games were sort of, if I— correct me if I'm wrong, but I believe at a national level it was like, Video games will only be played during these hours going forward. Focus on your studies, kids. How much sort of an editorial role, I'll say, does the Chinese government play in terms of what kinds of AI experiences are going to be allowed? And do we have AI boyfriends and AI girlfriends in China today?

I'm not too familiar. But there are certainly apps for AI companions being friends, being collaborators, and also different characters, right? So I have seen an app store, a virtual app store where you can pick different characters and choose their different psychology traits. So that definitely exists. But I think you're right that Tencent and other gaming platforms would have to control the amount of time that kids and teenagers could play on an everyday basis to focus on study, to focus on the real world. There is certainly that sense of paternalism from the Chinese society and the government.

How about on like a training basis? To the degree that we've seen attempts at regulation, I guess we've seen everything, at least at the state level in the US. But some of the things that have got the most heat around them in terms of debate are at the training level. Before you go and do some big training, you have to tell the government you're going to, or you have to commit to some sort of testing. And that's quite distinct from when you're model is done, you know, then you have to register it or prove you hit certain thresholds. Is there any sort of pre-training approval process that's happening on the Chinese side?

So there is national recommendations on different stages of the AI R&D pipeline. For example, there is a national standard setting body called TC260 that recently published The AI Safety Governance Framework. They already published it last year, but they have updated to a version 2.0. And I think this is very relevant to some of the conversation around frontier AI safety. So the document highlights the concern around catastrophic risk, included a lot of focus on loss of control, and also misuse and dual use domains like biological weapons and cyberattacks. And in terms of risk prevention, it highlights, um, measures, for example, like doing data curation and filtering in the pre-training phase, especially removing some of the hazardous knowledge that might be relevant for CBRN risk. But, um, as far as I know, this is not part of binding regulations yet.

Gotcha. Okay, pretty— I guess, how would we try to summarise all this if we— in many ways, it seems like there's more similarities than difference. Obviously, the core technology that's being developed thus far looks very similar. That means it has all the same strengths and weaknesses. The list of concerns and reading through the safety framework that you helped author, I was very struck by, like, if you took the names off of it, I would not be very confident where this came from in the world. It seems like the taxonomies of risks, the way of understanding them, conceptualising them, seems much more similar than different. I'm not sure if that quite extends. Would you say that's also true of loss of control? Certainly at these, like, you know, biosecurity, CBRN-type things, it seems cybersecurity seems like much more similar. Does the Chinese side have a similar imagination when it comes to loss of control risks?

Yeah. So Concordia AI worked with the Shanghai AI Lab to co-author the Frontier AI Risk Management Framework. For context, Shanghai AI Lab is an advanced AI research institute. with significant leadership roles in national standard setting and other areas in China. And the framework that we put out is designed to help developers of general-purpose AI models, especially those at the frontier, to manage the critical risk that these systems pose, especially for public safety and national security. We have built on some of the best practices and standards from safety-critical industries and also from around the international community. And we try to define a set of red lines as unacceptable threshold that no one should cross. And also yellow lines as sort of the early warning indicators that will trigger more stringent safety and security measures. And those include critical areas like cyber offense, biological risk, Large-scale manipulation and persuasion, and also loss of control risk. And on loss of control, which you asked, we also released a technical report alongside the framework. The technical report evaluated over 20 open weights and proprietary models on different domains. And under loss of control, we broke it down into several dangerous capabilities and propensities. Uh, including the ability for the AI models to self-replicate, uh, to be able to deceive and scheme, and also the possibility of producing uncontrolled AR&D. Uh, so I do think some of the conversations are, uh, quite similar to other parts of the world, um, which is a theme that we have, uh, constantly come back to. during this conversation.

Yeah, it seems like the big differences are really at the sort of consumer tech layer, right? There's like this something that looks more like a licensing regime with a certain established body of tests and certain requirements before you actually go to market in a consumer-facing way. And aside from that, I would struggle to identify too many big differences in terms of just the overall patterns of thought, the way that concerns are conceptualised. I guess the other big thing that we highlighted in terms of difference is that there is a greater fascination with AGI and a sort of sense of like an end goal or a You know, some sort of event horizon in the West as compared to China, but certainly a lot more similarity than difference, I would say. One question, though, on given all these things, right, I haven't personally tested this systematically myself to validate it. But I'm sure you're aware that Dario from Anthropic had said of the DeepSeek R1 model that it had very little in the way of guardrails and was rather willing to help with queries about biological weapons and so on. I think he said it was like the least secured model that they had tested in terms of refusals on those kinds of dimensions. How should we make sense of that? I guess, first of all, would you agree that is like a true assessment of that model? And if that is right, what happened there? Like, how do we have all these criteria and these processes, and yet something gets released with seemingly less safety baked into it than we have seen from most of the the models that come out in the US?

So there's different angles that I could offer. First of all, the approach to AI safety in China is mainly driven by binding regulations and standards led by the government. As we have discussed, that includes registration, pre-deployment testing, and labeling of AI-generated content. But one effect of this is that Chinese companies might be less likely to release safety evaluations for something that isn't directly in the regulation. The compliance costs required to deal with existing regulations reduce sort of the appetite for taking on additional safety measures. So in our industry section of State of AI Safety in China report, we have looked at the practices of all the Chinese AI developers. And they do implement all the standard safety techniques, right, from training data filtering to safety fine-tuning, IRHF, constitutional AI, real-time misuse monitoring, very similar to some of those techniques used by major Western companies. But those practices appear to be primarily geared towards the 31 categories of risk They are in the existing regulation, and the vast majority of Chinese companies have not publicly shared evaluation results for CPRN or loss of control risk. But I also think the AI industry is making some progress. I don't know whether you have seen this, just last week DeepSeek R1 was peer-reviewed and published in Nature. This is really the first time for any widely used large language model. And in their public-facing safety evaluation section, they are being quite candid. They said that a public model, an open-weight model, is vulnerable to further fine-tuning that could compromise safety protections. And their own evaluation also found that reasoning models like those of DeepSeek R1 tend to expose more sensitive or risky knowledge. But according to the peer-reviewed article, the safety level of DeepSeek R1 is generally comparable with other state-of-the-art models, comparable with, for example, GPT-4o. And in addition, our Concord AI team also performed independent evaluations of the DeepSeek models, and we have seen some improvements in safeguards related to bioweapons queries just within the last few months. Specifically on the benchmark SOS BenchBio, the refusal rate of DeepSeek model has increased from around 11% in the v3 version to 54% in the v3.1 version. The refusal rate for harmful biological prompts, and this current rate is around the median as compared to other models. And I could link to some of the data and graphs from our platform for those who are interested.

So overall, it sounds like, again, more similar than different. I mean, I just had this conversation with Adam from FAR AI, and they had just done some red teaming of the sort of defence-in-depth systems that companies have created. And one of his big takeaways is that, like, All of this stuff is sort of happening on a just-in-time basis where if you were really trying to be as safe as you could be, you would probably reverse the order of some things. You would build the systems that you're going to use to maintain control, especially on these biorisk things, which I do think are honestly pretty scary. Maybe not quite yet, but soon, if not already. You would have those developed before the model was ready. But it seems like, in contrast, what we're actually doing is train the model, see what it's capable of, and then be like, oh, shit, it's actually getting pretty good here. We better tack something on to try to address that. And it seems like basically the same kind of thing is sort of happening, at least for now, on the Chinese side, on some of these risks. We're first creating the model that is capable, and then we're figuring out what to do about it. Do you think that will flip in— I mean, it seems like with these licensing regimes that there's at least some structure that could flip it. What would flip it? Would it be like expanding from 31 to some additional categories? And what about open source? I mean, this is obviously— debate on this runs, like, super hot too. How committed do you think Chinese decision makers are, I guess, spanning government and companies? How committed do you think Chinese decision makers are to continuing to open source indefinitely? Or do you think there could be a time, perhaps not too far in the future, where there's a different analysis that's like, hey, these things are just too unwieldy, we really need to keep them on infrastructure that we control, Because that's obviously part of how you would run a broader defence-in-depth strategy.

Yeah. First of all, AI safety is a much broader landscape than just AI-enabled bioterrorism, right? For example, I remember looking at the AI risk database from MIT. It lists like 24 subdomains and 1,000 different types of risk. So I do think, you know, every country and company might have slightly different set of safety priorities and challenges. So we shouldn't be judging one player on just a single narrow metric. But I think you're right that the model registration and pre-deployment testing is quite a flexible tool, right, that could be adapted to address new concerns from CBRN or loss of control, other emerging challenges from advanced AI. So in fact, the Chinese government has repeatedly said the system could evolve over time. And the AI safety governance framework by TC260 that I mentioned earlier, I think it could help the national standard setting body to issue specific guidelines in the coming years. And we're already seeing some developments in this space, including agentic AI, including open-source governance and other critical issues. On open weights, I think there is just tremendous benefits both in terms of safety as well as innovation more broadly. At the World AI Conference in Shanghai in July, China announced the Global AI Governance Action Plan. And across all these documents, right, there is a sense that open source is very beneficial and China should capitalize on this development. And that is similar to the US AI Action Plan, right, that is also pro-open source. At the same time, the Chinese documents have also mentioned the possibility of misuse From open weights. And I think that's why, you know, we have seen papers from, for example, the UK AI Security Institute, the paper Deep Ignorance, that recommends removing hazardous knowledge from the data. So at the pre-training phase, and that recommendation is also in the TC 260 document that I mentioned earlier. In general, it seems like this is a new research challenge that stakeholders in China and elsewhere are really trying to tackle.

So maybe the vision, if I'm filling in the gaps correctly, would be like pretty strong commitment to open weights going forward, make that safe by doing enough filtering of the training data so that the version that gets put out open weights to the public just doesn't know about key questions in virology or whatever, and kind of literally can't help you there, as opposed to just having the sort of surface level refusal training. And then presumably on a more structured access basis, there would still be, of course, models with the full scientific dataset or whatever, but those might be the ones that would be like, withheld from open weight release. And, you know, you'd have to have certain access or approval to use those models, you know, with a know-your-customer regime or whatever sort of other additional layers of control.

Yeah, I think we do need a defense-in-depth approach, and we're in a pretty early stage of developing the science and the practice of open weight AI risk management. There are other potential interventions like tamper-resistant safeguards or responsible AI license or monitoring at the open source platform level. Many of those might not work and there's a lot of debates within the research community, but I think the general shape is there is a lot of positive interest and commitment in open source. And yeah, let's try to make it safer and less prone to misuse.

Let's shift gears to the relationship between the 2 countries. And I know this is something you're also working on. Obviously, it's not great, broadly speaking. Excuse me.

It's not great, broadly speaking.

One big thing that you hear from A lot of US government officials, like current, former, you know, military, intelligence community, whatever, is this general sense that you can't trust the CCP. And I think what a lot of people would say in response to all these things that we have discussed is like, basically, that all sounds nice. And it might all be true. And we do maybe even respect the degree to which the Chinese government is taking care of its own population. Obviously, in terms of development, the track record is undeniable. When it comes to making sure there's good standards for medical chatbots or whatever like that, we could say, sure, that's great. And it sounds like they might be doing a pretty good job, and maybe we could even take a lesson from it. But then you still get people kind of falling back to, But when it comes to the real game of international power, we just can't trust the CCP, right? That is, I think, the prevailing sentiment for a lot of people. And so you'll get people saying things like, whatever they say, whatever they do domestically, whatever they do to protect their kids from AI girlfriends, when it comes to militarising things, when it comes to racing for some sort of breakthrough superintelligence that is going to create some sort of strategic advantage, we have to assume that they're going to do it. And because we have to assume they're going to do it, well, then we have to do it. Is there anything you could say to those people that would be reassuring? Or what could I say to those people if I want to at least create some doubt in their mind that there might be more opportunity for collaboration than they're you know, their current outlook suggests?

Certainly there is quite a lot of distrust on both sides. But I think in AI, the specifics do matter. For example, if there is concern around developing advanced AI in secret, that is just not grounded in the reality of AI development, right? The state-of-the-art AI models from China are largely released openly, from KIMI to Minimax and Quan and DeepSeek. They are now among the world's best open-source AI models. And so I think that level of transparency and the possibility of third-party auditing and research should allow for greater trust between the companies and the governments. I also think, you know, the most advanced AI models in both the US and China are developed by the private sector, uh, not inside the governments. And, you know, it is the same global community of AI researchers, uh, a few hundred top elite researchers, uh, who, you know, most of them know each other. Uh, and so I think that Common ground of scientific vision and also commercial development is another factor where there should be more common ground between between the two countries.

Does the would you say you said and I definitely think this is important to keep in mind the distrust runs both ways from the Chinese perspective is there a similar worry I mean With OpenAI, for example, we hear these sort of conflicting narratives, I would say. On the one hand, it seems like we can— Rune, for example, has said, you guys, like, you people outside of the frontier companies, you don't know how good you have it. Like, the gap between what we have internally and what you get to use as a retail customer is only like a couple of months. And during that couple of months, we're working as fast as we can to get it all buttoned up so that we can release it. Basically, he's like, you guys, you don't appreciate how close to the absolute bleeding edge frontier you are. But then at the same time, you have like, oh, we have an IMO gold medal model, and that's going to be a while before we release that, by the way. And then you have like AI 2027 type narratives from, again, another former OpenAI insider saying that one of the things we should be watching for is a divergence between the models that get released to the public and the models that are developed and deployed internally, perhaps with this idea of trying to do some sort of recursive self-improvement, intelligence explosion, whatever. So I guess from the Chinese perspective, the US companies could sort of— they can't make quite as much of a case about, hey, we're releasing everything open source. But there is at least some claim that we're not hiding much. Everything's coming out pretty quick. But from the Chinese perspective, is that credible? Or is there a worry that, sure, there's GPT-5, but what we really don't know is what has OpenAI done in secret, or what might they be doing in partnership with the US government that we have no visibility into? Is that a worry that people get stuck on? And, you know, is there, I guess, is there sort of a mirror image of the— as I said, like in the US, so many of these conversations kind of go off the rails on, well, but China's going to do it. Does that happen in China too, where they say, well, but the US is going to do it. So we have to keep going because the US is— we can't stop them.

I don't think there is as much fear and paranoia. As I said earlier, the AI+ initiative by China, it shows that the main push is to mobilize the AI resources and direct that towards applications. And that is a sign that, you know, applications, economic engine, delivering for the people, those remain the top priorities for the government and for the party. I do think when it comes to export control, you know, the Chinese AI industry, you know, is viewing export control as a clear attempt by the US to stifle the technological progress of China and to maintain the US status as the leading global superpower. For that one, I think that has created a sense of resentment. And what is often missed is how this also affects the everyday applications, right? Much of the computing power, much of the FLOPS is used for civilian applications from generating AI art to powering, you know, AI medical diagnoses. And so in effect, the export control could deny a better quality of life for everyday Chinese people. And I think that also creates a bottleneck for addressing some of the shared concerns like risk from advanced AI between the 2 countries. And so I do think that is perhaps one of the most significant issues preventing, at least from the Chinese perspective, a sense of common ground and possible cooperation with the US.

What do you make of the recent decision from the Chinese government to— because we've seen multiple reversals on this, right? The Trump decision to go ahead and sell the H-20s, which I think it is worth reminding people that the H-20 was designed in response. My understanding is the H-20 was designed in response to the government standard that said you can only sell Up to this level of chip to China, and so they were like, okay, great. We'll make a chip of that kind, and we'll sell it to China. And then it was like, oh no, you don't! Which I'm not really sure how companies are supposed to operate given that kind of relationship with the regulator. But nevertheless, Trump comes around eventually and says, yeah, you know, it's fine. Just give me a little cut of the action, and you can do it. I was quite surprised then when the Chinese government said. Actually, hold on, we don't want them. How do you make sense of that? It doesn't— of all the sort of signals that we're getting from China about how seriously they're taking AI, most of them are pretty similar. But this one feels to me like, whoa, if you're really serious, you really think this is like the revolutionary technology, you would want to import those chips while you can. Of course, you still want to develop your own domestic capacity to produce chips, But why not both is sort of seemingly the obvious strategy. Import while you can, invest intensively at the same time. How do you make sense of that refusal of the H20s? And do you think that refusal of American chips will be persistent if there are other chips available to buy?

So as of September 22nd, Chinese companies are not purchasing the RTX Pro 6000D from Nvidia, which are seen as low quality and somewhat overpriced. I think there is also a lack of trust in H20 specifically, and regulators from China have raised concerns about the integration of backdoors, so specifically remote location tracking and also remote shutdown control integrated into the chips. I think, first of all, this could be reflecting a shift in the overall balance of national power as well. So when the Trump 1.0 administration started the trade war and the Biden administration rolled out a series of export controls, China was less prepared then. But now I think China is demonstrating that it is not dependent on the US for its technological future. And in fact, In fact, maybe the US is also dependent on China for critical minerals and rare earths in some of the recent trade negotiations. I think second, this is also backed by a growing level of confidence in the domestic AI chip industry in China. The export controls have acted as a catalyst to accelerate the development of homegrown alternatives, right? So companies like Huawei are making significant progress with their Ascend line of chips. And actually just this month, Huawei announced the new Atlas 950 supernode that would support more than 8,000 chips. And the sort of Atlas 950 supercluster would also use more than 500,000 chips as a cluster. And so Perhaps from the Chinese industry perspective, it doesn't want to be reliant on the American stack, and this is a critical window to develop the homegrown alternative.

I still think what the analysts that I have read, who I think are generally taken to be the most credible, I should say, among Western analysts of the chip industry seems to boil down to something like, of course Chinese companies are going to make good progress here over time. Of course, there's no preventing Huawei from becoming a scaled chip manufacturer on a decade timeline. But still, over the next 5 years, the expectation is that the production capacity is just going to be dramatically higher for the Western, notably, which includes Taiwan, in a critical role. But that the sort of Nvidia, TSMC Western complex is going to produce, like, I would say the median estimate would be comfortably more than an order of magnitude more chips than what the Chinese domestic producers will be able to produce. And if that is true, it's going to be a big difference in terms of how big the training runs can be, how big the deployments can be throughout the economy, et cetera, et cetera. So do you think that is wrong, like that maybe that analysis is wrong, or if that analysis is right, Like, then I kind of come back to, I would still buy those H20s while I could. And yeah, maybe they do get bricked at some point or like something, you know, I can't speak definitively, certainly, to whether or not there are any backdoor technologies or attack possibilities built into those. I would guess not, but I don't really know. But yeah, I mean, what do you make of that? Because it seems like— That, that's the— that is the conventional wisdom in the West. It's going to be a dramatic difference in production capacity, and that's going to give the US this, like, pretty durable edge, at least over, like, a 5-year timescale. And so the— and I think the perception from the, from the West is that it's just a plain mistake for the Chinese government to refuse the H-20s, that it's like, you know, A face-saving measure that's like ultimately self-defeating or something along those lines. But the analysis could be wrong. Maybe it is going to happen a lot faster and, you know, maybe the gap wouldn't be so big. What's your expectation?

So I think there are 2 other factors. One is China could leverage its relative energy abundance to overcome some of the limitations on a single-chip basis. For example, a few months ago, there was the release of Cloud Matrix 384 by Huawei, which uses more than five times as many Ascend chips, but it more than compensates for each chip being less powerful as compared to an Nvidia Blackwell GPU. But you know, it matters less for China if the primary trade-off is energy efficiency, given that China has significantly expanded its power. grid, adding an amount of capacity equal to the entire US grid in the last decade alone. Right. And most of them could be pretty renewable, with some of the largest installations in the world of solar, hydro, wind, and now a potentially leading role in nuclear energy deployment as well. And so in the Chinese stack, China could prioritize scale over power density. I think another factor is beyond hardware. Chinese AI companies are also trying to become more efficient at training large models. So as detailed in the peer-reviewed article in Nature, the DeepSeek R1 costs maybe 500 NVIDIA H100 chips, and that's pretty impressive given the performance that it is able to produce.

Yeah, that energy point is interesting. I've had a little bit of an intuition around that that I haven't really developed. But basically, the idea there is that we can make potentially all the trailing-edge chips we might need, and the primary disadvantage of those is efficiency. But we've got so much energy and we've got kind of ability to provide national subsidy for critical industries such that maybe we can just kind of make up for it that way. And it doesn't, the deficit isn't actually so big when it comes to what we can practically do in terms of large-scale training.

Yeah.

That's, That is definitely very interesting. And certainly the requirements on training, I think, have been relaxed. And this is not something I would say I'm a real expert in by any means, but not too long ago, there was this idea that you have to have everything in one data centre. The bandwidth is such a limiting factor, yada, yada, yada. And now it seems like we've taken— we being the sort of collective research community globally— has taken pretty big bites out of that problem. Now we are training across multiple data centres. Now we are seeing more distributed training come online. We're also seeing all sorts of efficient weight communication and update schemes, not to mention the fact that inference itself is becoming a bigger and bigger part of training due to the rise of reinforcement learning. So the trillion-dollar cluster all in one place, all with, like, super high interconnect being, like, a hard requirement. That also seems to have fallen off quite a bit compared to what the discourse might have suggested, you know, 18 months ago or what have you. Going back to this trust issue a little bit. So, I mean, I think the point about, like, China open sourcing the models is at least somewhat compelling. But it still leaves this question of like, yeah, but what are they doing in secret? Right? Sure, there's all these great things coming out, and sure, they're open source, but on both sides, we really don't know what's going on in some data centre somewhere between the government and a leading company or whatever. We just have a really hard time getting visibility. And without visibility and without a foundation of trust, it's really hard to be confident that the other side is not doing some dangerous experiment that we should be concerned about. And I do think that's pretty much symmetrical concern. Do you have any ideas about how we can create a stable equilibrium between the 2 countries? I'm sure you read the superintelligence strategy document from Dan Hendrycks, Eric Schmidt, and Alex Wang. not too long ago, where they described the mutually assured malfunction, MAIM, mutually assured AI malfunction. I wasn't super compelled that would actually be a stable equilibrium, but I did feel like, hey, at least somebody's trying to sketch one out. Do you have any sort of narrative or sense of how the 2 countries Aside from a sort of revolution in relations, could get comfortable with some amount of trust that could create some stability such that neither side feels compelled to do dangerous things out of a fear that the other side might be doing it in secret?

Yeah, that's a fascinating and very important question. Concept in the superintelligence strategy, as I understand it, is to prevent any single nation from achieving AI dominance. And countries, especially the great powers, are prepared to sabotage and destroy the AI infrastructure of the rival. Uh, I think on the positive side, it's good that it acknowledged dominance from a single superpower is a problematic objective. and its law paradigm and is trying to chart a path of stability under a world of very powerful AI. But I do think the analogy to the MAD, mutually assured destruction, from the nuclear era, it's also problematic for 2 reasons. First one, it really lacks a clear observable red line. Uh, in the world of AI, right? Uh, the logic of nuclear deterrence works because using a nuclear weapon is a detectable event. You can't really hide a nuclear test or a launch of ICBM. So that creates a clear red line. But, you know, if it hinges on a trigger point of an aggressive deployment of superintelligence, what, what does that really mean in practice, right? Like we have this meter graph of the length of tasks that AI could do, uh, that is doubling every 7 months. Uh, so is there a certain level of that graph that would count as getting into the intelligence explosion? Uh, so I, I worry that it is quite vague if we try to apply that concept in the world of advanced AI. Um, and arguably, if you think about it from Chinese perspective, The US, with some of the prominent AI leaders, is already talking about AGI in 2027 and also, you know, implementing export control and, and so forth, right? So one could imagine that, the US is already racing for a monopoly on, on superintelligence. Does that trigger, you know, the mutually assured AI malfunction? I hope not, right? And so that leads me to the second point, the risk of instability and escalation. If countries actively endorse this strategy, then, well, it would basically be declaring the willingness to go to war with one another, right? It's not just about attacking a data center. It will be destroying one of the most valuable National assets, one of the most critical infrastructure of another country. And I think that is just a very terrible and unpredictable event. And so I don't see that as a plan for stability. Quite the opposite. It seems like a potential sort of a hair-trigger alert of of terrible risk from AI.

Yeah, unfortunately, I agree. And I think you're very right to highlight just how intense the fog of possible war is going to be around, and for better, I guess, for false positives and for false negatives. I would say, like, if you're sitting in one country trying to assess what's going on in some data centre in the other country, I mean, your visibility into this is low in every respect, starting with what you should care about, as you rightly pointed out. But then also, are you getting an accurate signal on what's really going on? It just seems like it's going to be extremely hard for anybody to have much confidence that they would be doing the right thing at the moment where they would be making a decision to sabotage some key infrastructure. So, yeah, I'm with you. Again, I applaud the effort for looking for some sort of stable equilibrium. I did not feel like they found it in that piece. Do you have any better ideas? I mean, not to put you on the spot, but we need all the help we can get here. Is there any stable state that you could imagine getting to again without sort of a overly utopian imagination of revolution in relations, and we can come to that next, you know, in terms of like working toward that. But if we assume that like distrust levels remain relatively high, is there any way to make that a stable situation?

So I wrote some of the recommendations in the Time magazine piece. I think one strategy would involve building on 3 core pillars. The first pillar would be for the international community especially between the AI great powers, to define a set of concrete red lines for advanced AI. So that is a set of agreed-upon limits on the capabilities and behaviors of frontier models. For example, that would prohibit the development of systems that could pose a catastrophic or even existential risk to humanity, possibility of AI leading to proliferation of weapons of mass destruction. possibility of autonomous AI being able to self-replicate or self-improve and also lead to intelligent explosion that either humanity as a whole or other countries can no longer comprehend and control. And then the second pillar would be having continuous testing and evaluation for early warning indicators. And ideally, we can create an architecture, a set of shared protocols that countries and leading AI companies can share the most relevant results of evaluation with one another. And the final pillar involves creating a set of emergency response protocols, right? So if certain behaviors of the models are found as early warning indicator, what should you do? What should be triggered when those thresholds are being crossed? And that could involve implementing more safety and security measures. It could be mandating human oversight, but especially in moments of crisis, it is really critical that countries and international community have a prepared plan to manage it rather than let the risk escalate.

How optimistic are you that our governments can come together and do this in any sort of credible way? I mean, I've kind of voiced the view from the US government that, you know, you can't trust the CCP, I totally expect that the Chinese government feels similarly about the US, even if it's not like you can't trust a particular individual or whatever. We just obviously have such radical change in attitudes at the top that it's very hard for the Chinese government to know what they're going to be dealing with in 2028, or I guess 2029, when the next president takes office. So I think we've had— I'm not really sure of the status of this, but there was at least the one, like, no AI in the nuclear chain of command soft agreement. I'm not sure if that has been ratified or sort of institutionalised in some way. Could we expect governments to come together and do more in the direction that you're advising right now? And if not, what else can we do? I mean, you've got interesting ideas around Singapore as kind of a meeting place. And obviously there's these like Track 1.5, Track 2 dialogue type things that you're putting together. But yeah, like, where can we actually make progress toward these goals right now?

Yeah, you're right that, you know, given all the tension between the US and China, it's positive that leaders of the 2 countries were able to identify AI risk as an area of dialogue. They have held meetings on the topic and also to build towards a joint agreement that there should be human control over nuclear command and control. I think, you know, a few years ago, this would have been thought of as pretty surprising. So at least there is a positive step. Um, Obviously, I'm not an expert in the AI and nuclear space. And, you know, the agreement is pretty high level at the moment. And there are some work to be done in terms of operationalizing the agreement. But I do think the overarching concept of having human in the loop on some of the most critical decision-making systems, that idea could be applied for broader discussions on AI safety, right? We already have leading scientists and academics from China and the West agreeing on loss of control being a serious issue. And there are probably steps that we can build not just between the researchers but also between the industries and eventually at the triad level to ensure human in the loop when we get to a certain level of autonomous advanced AI systems.

So let's hope. Going away from the official government-to-government track, tell us about the sort of dialogues, convenings, et cetera, that you're helping to put together and what's happening at those. What is the theory of change? Is it about just sort of Keeping connections warm and kind of keeping the research communities connected to each other. How do you conceive of the value that all that work is driving?

Yeah, I think Track 2 dialogues and scientist-to-scientist conversations have long been a mechanism for exchange on issues of common concern between great powers. I think primarily they could could foster trust and increase mutual understanding among the participants, and then also to help formulate and refine policy solutions to their respective governments. One example was the Pugwash Conference on Science and World Affairs, which was held regularly since the year 1957. And that was really a pivotal time for reducing catastrophic risk from nuclear weapons. We have put out a piece on Track 2 dialogues on AI between China and other countries. And overall, I think these dialogues have produced several useful outcomes. For example, refueling areas of scientific consensus on frontier risk, you know, solving particular issues that experts are better suited to solve at the moment. For example, looking into the threat modeling and potential risk from the convergence of AI and biological risk. And so I think this is just a critical area where we should continue.

And how about Singapore as a venue for that sort of thing? Is that Is that a unique opportunity in today's world? I've never even been to Singapore. So I've got a lot to catch up on.

Yeah, I think Singapore is one of the top countries in AI R&D globally, and also ranked top for overall readiness for AI innovation. So it's a very forward-leaning country. Concordia recently published a report report on the state of AI safety in Singapore. And it seems like Singapore is just playing a pretty outsized role in global and regional AI governance. For example, it is able to convene international AI events like the Singapore Conference on AI, and that produced a consensus document on global AI safety research priorities. That consensus was developed by over 100 global experts and they came together to build a shared technical agenda for general-purpose AI safety, including for loss of control risk. And then I think another distinct feature is it is a very vibrant hub for both Chinese and US companies. Many of the leading companies from US and China have office and presence in Singapore. Um, Singapore has a very vibrant assurance hub, uh, doing testing evaluation, um, on these models, uh, including its Singapore AI Safety Institute. Um, and so given this combination of convenient experience, uh, also neutrality between US and China, and also a very strong kind of AI ecosystem in itself, It is pretty well placed to be the meeting between the East and the West. And I think that's also the reason why we're excited to have an office in Singapore.

Is there any other sort of international cultural initiative that you might promote? Like one idea that I've had, and you can react to this idea, but then I'm especially interested to hear any other ideas that you have, is to create something like an MMLU for morality, which I think is going to be a tricky project. But I'm struck by the fact that when companies put out new models, I particularly noticed this with Gemini 2.5 Pro, in their official blog post announcing the new model, they, I believe, reported 12 different benchmark results. None of them pertained to safety, alignment, morality, law following, anything along those lines. It was all just raw capability. And this got me thinking, geez, if there was one number that we could look at to sort of get a sense for how well behaved a model is, that could be like a really powerful carrot that could potentially lead companies in that direction. Like, If it became the norm that you're going to report that number when you put out a new model and that it's going to become one of the dimensions of competition, it seems like it could be a pretty meaningful contribution to the overall trajectory of AI development. Again, a lot of little caveats, like you probably have to have a bunch of subscores under that one score. You might have to have localised scores as well, because I do think you countries are going to want somewhat different behaviour on certain aspects of those questions. But it seems like a good idea to me. What do you think about that idea? And what other kind of maybe outside-the-box ideas do you think are relatively neglected right now? What do we need people to step up and do that is not yet happening?

Well, I'm all supportive of creating these positive Race to the top dynamics, incentivizing companies and researchers to compete on AI safety research to create safer, more trustworthy AI models. I think to some extent this is already happening, right? That, you know, people have to report their performance on MLU, but also like Humpbench and other AI safety ethics and risk-related benchmarks. I don't think we can rely on any single benchmark. We just need a portfolio of all those different dimensions. So I think, yes, I think that would be great if we have more leaderboards of ASV performance.

Any other moonshot projects that you would love to see somebody act on that just aren't happening today?

Well, I think it would be great if we could align on a global framework for frontier AI risk management. You know, from China, we work with Shanghai AI Lab for this framework. This closely resembles some of the frameworks developed by other global players. For example, some of the leading AI labs in the US, and also recently with the EU core practice. So I really think this convergence of approaches is a very positive development. But we are still in the early days of creating international AI standards, right? So one analogy I think of is, you know, when people fly between different cities, either from Europe to US or Singapore to China, we all abide by the same set of international safety standards. And it's quite incredible how much safer it has been to get on a plane. I remember a statistic showing that the number of accidents from getting on an airplane, that has dropped by, I don't know, 2 orders of magnitude just within the last few decades.

How about heroes? Are there people either on the US or China side that you think should be higher in status than they are, or just more appreciated? Who's doing great work to either reduce tensions or build understanding, build these shared frameworks? Who do you admire? And that could be like, who should I have on the show in the future? But who should people, you know, look to for inspiration?

Yeah, I think one of the most urgent priorities right now is to build a shared international understanding of the risks from advanced AI. One really positive and landmark moment I have experienced was the Global AI Safety Summit at Bletchley Park in November 2023. And I remember attending the summit. And seeing that ministers from the U.S., China, and other countries, I think a total of 28 countries, came together to sign a pledge of declaration. And a specific outcome of that summit was this agreement to nominate international experts to work on an independent report, right? That was led by Joshua Banjo and. you know, feature contributions from 100 experts around the world. It really aims to be sort of the IPCC for AI safety. And I think that is a very positive example. There are a number of other international forums that are making progress in this space. For example, on loss of control, there is the International Dialogue on AI Safety that agrees on For example, recently how there is a growing level of deception and scheming behaviors from frontier AI models. There is the Global AI Bioforum, which published a statement on the biosecurity risk at the convergence of AI and the life sciences. And some of the world-leading scientists from both the AI and the synthetic biology community How about for individuals that aren't famous or powerful?

What can we do? I want to go to China for multiple reasons.

I've never been.

It sounds fun. The food sounds amazing. Can I do anything that would actually be meaningfully positive? Or how would you coach people who want to make their own meaningful contribution? Grassroots, even if it's modest contribution, what can we do to make the world marginally better?

Well, thank you for asking that. First of all, I think we all need more people-to-people exchange and understanding. And so trying to go beyond the headlines and creating more nuanced understanding, I think that's a first step. And this is what Concordia tried to do, right, by publishing these AI safety developments in China, in Singapore, in Asia, and vice versa, right? We also try to absorb and learn from different parts of the world, including from the West, in our work within China. And I think for people outside of China coming to AI conferences, in China could be a good opportunity. For example, with the World AI Conference that happens every summer, I think that is a very rich platform for discussing the frontiers of AI innovation, but also for AI governance and safety. And then in terms of donations, I think philanthropy could be a powerful force for making advances in AI safety, uh, funding ambitious scientific research, training the next generation of leaders, fostering informal dialogues. Um, in some way, I think the field of AI safety philanthropy is still in the early stage, right? Much like what climate science was, um, maybe a few decades ago. And so, uh, getting involved now means that one could be an early shaper of this critical domain.

I just did an AI safety charity review project this summer, and there were a few. It's very hard, I think, for philanthropists to figure out what is actually helping, because we had a number of organisations, and I think by and large they did receive grants, But I just noticed from my perspective as an evaluator that it's very hard to read a document that says we're trying to do this and this to foster improvement in US-China relations or whatever and have any sense of, like, is it working? Is it having any impact? It's tough. How do you know what's working? Like, if you were to advise somebody to look for signals in that kind of thing, or even just for yourself, like, how do you know what's working and what's not working? What are the feedback mechanisms that you trust to know that you're making the world marginally better?

That's a great question. I think it depends on the specific interventions. But when it comes to creating these international Conversations. One metric could be whether we are exploring topics and issues that are pretty important and urgent, but no one was talking before. So to be concrete, we recently ran an international dialogue with a focus on crisis preparedness. exploring scenarios such as how AI could enable large-scale cyberattacks on critical infrastructure. We found that policymakers and elite decision makers were all pretty surprised that this could happen in the next few years. And I think we created a space for people to be better prepared for a world of powerful and unfamiliar AI. I think another metric could be whether we could create consensus and outcome documents. I mentioned a couple of examples before, and I think this really shifts the Overton window in the international conversation. to show that this is something that could be agreed upon. And repeatedly, I think we have shown that international collaboration on catastrophic risk is something that is important and tractable to make progress on.

Do you feel like when you're doing this kind of international relations improvement work, In China, do you feel like you're in a minority position? Like here, and certainly the folks that are focused on this, my sense is that they feel that they are definitely in a minority position, that kind of things are getting worse, not better, that they are basically trying to resist a negative trend. but that the negative trend is still happening and that they're kind of like just a minority voice and it's sort of hard for them to break through because the dominant narrative is so hawkish and so focused on rivalry and winning. Do you feel similarly in China, or do you feel like you're more mainstream there in some sense than than people here feel?

I feel pretty mainstream. I think there is a lot of stakeholders ranging from academics to industry to think tanks that view international cooperation on AI governance as not only accepted, but something to be prioritized. I think in our reports, we have documented a lot of institutions and individual experts that have done this consistently throughout the years. And certainly when it comes to prominent AI conferences in Beijing, Shanghai, and elsewhere, in our experience, the organizers really want to engage with international experts and guests and would, you know, put in a lot of time and resources to send those invitations. And when people come to China, there is a lot of effort being done to make sure that they, they feel comfortable, they could explore the cities, they have a good time, basically. So I think there is a lot of goodwill in general. So yeah, I don't, I don't feel like I am doing this as a very minority or marginalised community? Not at all.

Yeah, good. It's an important data point. Maybe one last just kind of big picture or sort of outside the box question. If you had some large amount of money, say a billion or several billion dollars, are there any sort of megaproject type things that you could see people putting together that might really move the needle? And I'm thinking as far-fetched here as, like, a joint research centre on some small Pacific island where you could imagine scientists coming together to do very sensitive research in an environment that neither side could really defend, that it's kind of like a sort of very obvious, sort of sequestered space for this kind of supersensitive work. I don't know if that's a good idea or a bad idea, but that strikes me as like the kind of thing that if we were really trying to come up with creative solutions, we'd hear a lot more about outside-the-box ideas like that. Do you have any, you know, kind of outside-the-box ideas like that that you think maybe somebody should pick up and run with?

Yeah, I think having an international joint research project or lab with a focus on frontier AI safety, that sounds like a great idea. I think we really lack a space where researchers can come together, not just to discuss, but to make concrete progress. I do think I don't have very much creative idea. What the world probably needs more is sensible, pragmatic ideas to make progress. All too often, even the most commonsensical things to do especially when it comes to international collaboration and policy, are not being done. So I'll probably sort of respond to your question in a different way that we just need to make more progress on the low-hanging fruit.

Yeah, double down on the basics. Is there anything that we haven't touched on that you think we should or that you would want to make sure people are aware of before I let you go?

Yeah, I think I would just like to mention that around 2023, discussions on frontier AI risk also took off around the world, obviously because of ChatGPT, but in China as well. And you see prominent Chinese scientists sign on to the statement like, you know, extinction risk from AI should be a global priority alongside other concerns. And there was an important moment of a Beijing AI conference featuring people like Geoffrey Hinton, Andrew Yao, Sam Altman, and others with a full day of speeches and panels on AI safety. And so that was really one of the first times where catastrophic risks from AI were discussed so prominently at a technical AI conference in China. And then since then, the landscape has changed quite a lot as well, where as of 2025, there is like more than 30 AI research groups in China that have devoted a substantial amount of time on frontier AI safety. And, you know, in the beginning, it was mostly along the lines of RHF or jailbreak defense, but now I think there are also a lot more papers on scalable oversight of superhuman AI systems, on mechanistic interpretability, on other areas that are much closer to advanced AI safety concerns. And so I think this expansion of research highlights that academics in China want to think this is a problem worth exploring and directly investing their time and effort into it. And are also pretty much aligned with some of the research happening in the West.

Any highlights from that body of research that you think I should take a deeper dive into, particularly any that are notably distinct in terms of their approach from stuff that might be more familiar to me as somebody who primarily follows the Western developments?

Again, I think 95% of those papers and research are pretty similar. But I do think in the last half a year, there's more papers that focus on the risk from embodied AI, both in terms of how that could align with human values, Also in terms of jailbreaking a foundation model-controlled robot. Yeah, in general, it seems like physical AI is a big theme of Chinese AI in 2025.

Yeah, that's maybe something we have kind of underdiscussed, actually, because there are so many Commonalities. But it does seem like obviously China has a huge manufacturing strength that if and when these humanoid robots actually working is going to come in huge for producing them at scale. And I haven't seen anything that I can recall from Western researchers that focuses on And it's a little bit from Google, I suppose, because they are doing this stuff. But it does seem relatively neglected. Like, it seems like kind of a collective blind spot in the West that, hey, we're actually going to have humanoid robots. They probably are going to work. There may be a lot of them walking among us. And how does that change the way we should be thinking about safety and control? It does strike me as something that is underdeveloped here. So to hear you highlight that as something that might be unique in the safety literature there is interesting. And that's maybe something I can do a deeper look into. And I definitely appreciate any pointers to particular papers, people, whatever. Is there anything else that we should know or should take a minute to contemplate when it comes to the relative emphasis on embodied AI broadly, or humanoid robots specifically, that's going on in China right now?

I'm sure there is also a lot of AI safety literature going on in other parts of the world, but perhaps to highlight the concerns for safety in industry documents. So there is an AI industry association in China. That put out these voluntary safety commitments. It has been signed by more than 20 major Chinese companies. And in the latest version, it highlights risks from agentic AI, but also embodied AI. I think this really reflects how the industry is looking ahead and thinks, well, their models could be integrated into the physical world. at some point. And this is the time to start thinking about safety and policy measures as well.

Yeah, I definitely see some of these short videos online of these robots getting kicked over and bouncing back and doing flips and stuff. And it's like, whoa, not only might these things be as smarter, smarter than us, but they're going to be potentially a lot more agile and robust to physical assaults than we might be before long too. So yeah, it's going to be a strange world in the not too distant future.

Yeah, during the Chinese New Year, there was a national TV program showing the robots from a company, Unitree, having group dances. And I don't know, like hundreds of millions of people in China probably watch it. And now they are running these marathon competitions between AI robots and humans. And very soon they will be doing like a full-scale Olympics again between the humanoids and humans in Beijing and other cities. So it is a very interesting world.

What is the cultural reaction to that? Like, I mean, one thing that people often say When we debate, like, what's future life going to look like? Where are we going to get meaning from? And what's going to matter? People often say chess has never been more popular. Isn't it interesting that we had the first computer that beat the grandmaster in chess, like, 20, I think, in the '90s, maybe almost 30 years ago now? And then there was the period when the human and the AI together were best, and now the AIs are just just the best. But people are still very interested in watching humans play chess. And then they'll say, like, nobody's that interested in watching AIs play chess against each other. Maybe the real chess experts are. How would you describe the Chinese reaction to this sort of thing? Do you think the Chinese public will be interested in sort of a robot Olympics? Will that be like something that will have interest in the public imagination? Because it seems like people here don't expect that to capture public interest, but maybe there it's different.

No, I think many people do find it somewhat entertaining and amusing. There is also a genuine function to show that there are some companies that perform better in this sort of real-world benchmarks as, as, as, as you will. And to get beyond the hype, right? For example, the marathon competition between the humanoids and the humans, it shows that, you know, there is still a lot of room for improvement for the humanoids. And also it creates some societal discussions on what that means for potential impact on the labor market, right? There was a local sort of incident of people protesting that self-driving cars are making too much progress and that could take away jobs for the drivers in a central province of China. And so obviously there is some controversial and negative ramifications as well.

Yeah. How much does that get— I mean, that's another thing we haven't talked— there's so many things, so many aspects of this, we can't possibly cover them all. But since you've raised that, is there an anxiety in the Chinese public that this could be bad for me personally? Like in the US, there's definitely broad I think quite widespread sense that even if AI gets really good, even if it can do everything better than I can do, even if we have, in theory, abundance, I'm not really sure. And I'm speaking not as Nathan, but as sort of the general plurality of the public. I'm not really sure that that's going to benefit me. I still sort of worry that the rich might get richer and I might be left behind. And if I'm driving a car for a living, and all of a sudden the AIs can do it better, like, I'm not sure what's going to become of me. Is there— does that same anxiety exist across the Chinese public? Or perhaps there's a higher level of confidence that, like, the benefits from these sorts of things will in fact be broadly shared? How would you characterise it? I mean, it's hard to summarise the views of more than a billion people, obviously, but— How would you summarize the vibe there?

Yeah, obviously I can only speak to my partial observations and anecdotes, but, you know, what do we have as humans? We have our intelligence in our mind and then we have our body doing physical work, right? So I think it's very natural for people to ask what if AI could do both in the future. And so in the last few years, the unemployment rate, especially by the youth, has increased quite a bit in China. Everyone has a bachelor's degree, education level has gone up. So it's much harder to get a job at a bank or in the government As, you know, white-collar work. And so people also have to consider like food delivery or like being a driver on Didi, Chinese version of Uber. But what you've— there is also self-driving cars and all these humanoids that can do what everyone can do physically. So I do think there is a sense of anxiety. among some significant portions of the population as well. And it often surprises me how little governments are addressing this concern, but it seems so salient, right, for most people.

Yeah. That's interesting. I'm kind of— I would have maybe expected the Chinese government to be more forward-looking or, you know, a little bit more prepared than what I perceive the US government to be. But it sounds like maybe not. And again, it is all coming at us quite fast. So maybe I shouldn't be surprised, but I guess I'm always surprised.

I still think there could be a feature and a potential strength that China could leverage. It is a socialist country, right? And so if the profits of AI are dependent on chips, which are also dependent on energy, data centers, and then land, right, as critical infrastructure, and most of those infrastructure are owned by the states, not by private companies, you know, by state-owned enterprises. You could imagine a model of distributing the benefits of AI using these state-owned assets.

Yeah, I think we're going to need something like that here. I've often— not that I have the answer by any means, but I've often suggested that the time is probably now to start working on a new social contract. And It's been slow to develop, but I do give Sam Altman in particular credit for investing personally in universal basic income studies. And it does seem to me like we're going to need to do something to decouple one's fundamental right to live a decent life from your ability to contribute meaningfully to the economy. No matter how far the AI continues to progress from here, it seems like just with implementation of what we have, there is going to be very significant displacement, and we really can't expect our current structures to handle that in a graceful way, to put it mildly. So, yeah, somebody needs to take some leadership there, and we currently have kind of a vacuum. But yeah, it sounds like, you know, there's opportunity for, uh, people to step up in the Chinese context as well. Um, I'll just ask it again. Anything else we haven't talked about that, uh, you think people should be aware of before we break?

I don't think so. Um, I think maybe as an expression of gratitude, uh, thank you for all the thoughtful and open-minded discussions on your podcast. I think this is really amazing, uh, with all the work that you're doing and for also bridging international understanding. Um, and then also as a plug, uh, Concord AI is hiring, uh, multiple positions for researchers, for operations, uh, for other roles, both in Singapore and Beijing. Um, so yeah, really pleased and happy to have this conversation with you, Nathan.

Well, that's very kind, and I'll do my best to live up to it. But right back at you. I appreciate all the hard work you're doing to maintain and hopefully even build, you know, positive momentum in shared understanding of the AI phenomenon and particularly how we can make sure we're on the right side of history with this thing, because It doesn't seem like that's going to happen by default. It's going to take some heroic efforts from folks like you. So I really appreciate what you're doing. Bryan Tse, founder and CEO of Concordia AI, thank you for being part of the Cognitive Revolution.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback. Guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts where experts talk technology, business, economics, geopolitics, culture, and more, which is now a part of a16z. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, Check them out and see my endorsement at aipodcast.ing. And finally, I encourage you to take a moment to check out our new and improved show notes, which were created automatically by Notion's AI Meeting Notes. AI Meeting Notes captures every detail and breaks down complex concepts so no idea gets lost. And because AI Meeting Notes lives right in Notion, everything you capture, whether that's meetings, podcasts, interviews, or conversations, Lives exactly where you plan, build, and get things done.

No switching, no slowdown.

Check out Notion's AI meeting notes if you want perfect notes that write themselves, and head to the link in our show notes to try Notion's AI meeting notes free for 30 days.