Inside China's AI Ecosystem: A View From Beijing

They're having to spread their bets fairly widely because there isn't a real standout front runner in China for LLMs in the way that you might see just 2 or 3 really standout players in the US.

You don't see as much research happening in China on AI alignment. You don't see top labs here committing to responsible scaling policies in the same way that Anthropic and OpenAI have. There's a lot less thinking here about corporate governance mechanisms that might need to be in place to mitigate the worst risks from AI. But maybe this is because you don't have in China such a long history of labs that are really explicitly focused on building AGI.

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. Today, I'm excited to share another episode in our experimental series providing wide-angle coverage of critical AI topics. This time, we're providing a bird's-eye view of the Chinese AI ecosystem. My guest is a tech worker and AI watcher based in Beijing who goes by the pen name L Squared. She prefers to remain anonymous, not because her takes are super spicy. On the contrary, I think you'll find that they're very well-grounded and not at all sensationalist. But just out of an abundance of caution, given her location and employment situation. In this wide-ranging conversation, we cover the major AI players in China, the quality of Chinese models and products and how they compare to US offerings, the Chinese public's engagement with AI products, the state of AI regulation in China and the Chinese government's general stance toward AI technology, the impact of the US chip ban both today and in the years ahead, The state of US-China AI research collaborations, and much more along the way as well. One of my key takeaways is that the US and Chinese AI ecosystems are in many ways more similar than they are different. In both countries, the major tech giants are leading the charge by writing the biggest checks to AI startups, providing critical infrastructure, and increasingly competing with those same startups in the race to bring AI products to consumers and businesses. Research, in both countries is also being pushed forward by a mix of big tech, top universities, and startups. In terms of differences, the most important one is simply that there are no standout labs in China that are pushing the frontiers of possibility in the way that OpenAI, DeepMind, and Anthropic are doing here in the United States. Beyond that, L Squared shares fascinating details on the current state of China's AI regulations, which on paper are quite onerous, requiring companies to jump through significant hoops to bring products to market, but which so far seem to be enforced only very loosely. We also discussed the apparent lack of jailbreaking culture that comes to the fore whenever a major new model or product is released here in the West. I am very grateful to L Squared for taking the time to share her knowledge and perspective on all of this. And I also want to thank Jordan Schneider of ChinaTalk for making this connection possible. For more of L Squared's writing on AI in China, you can refer to the ChinaTalk newsletter. We did have some minor recording issues due to VPN bandwidth limitations while recording this episode, and so we are using an AI-enhanced version of the web recording. I think it sounds very good overall. That's a testament to the power of modern AI systems. But if you hear any weirdness along the way, that's why. As always, if you find value in this work, please take a moment to share it with friends. Regular listeners will know that while my politics are generally very libertarian, and I certainly would not want to live under the Chinese government's restrictive speech codes, My bias is to look for ways to de-escalate U.S.-China tension. And I hope that content like this, which provides an up-to-date, on-the-ground perspective on the state of Chinese AI technology, can serve as a counterweight to the worst fears that our collective imaginations might conjure up. Finally, if you've got a unique perspective on another important area in AI, even if that's just a contrasting perspective on the Chinese AI ecosystem, I would love to work with you to make an episode about it. So please reach out via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now, I hope you enjoy this insider's view on China's AI ecosystem with L Squared.

Technology worker and AI watcher based in Beijing, China, previously published on the China Talk blog with some outstanding analysis. Thank you to Jordan for making this possible with a timely connection. Welcome to The Cognitive Revolution.

Thanks so much, Nathan. It's great to be here.

Yeah, I'm really excited for this conversation, a conversation from 12 time zones apart and across global spheres of influence with a focus on what is going on with AI in China. This is something that I've been trying to wrap my head around a little bit and just find it very difficult to get a qualitative understanding of, despite trying to read the blogs and whatever else is available to me.

So really excited to get your take.

I basically just want to come at it from all angles, and that includes your experience as a user of some of these products, watching society there more broadly, your take on what the government is trying to do and what they're actually doing, and a little bit of how you think the international competition is shaping up as you see it from Beijing.

How's that sound?

Sounds fantastic.

All right, let's do it.

So I guess obviously a big disclaimer is I really don't know what I'm talking about. A lot more than I do, but this will still be obviously a limited perspective. One of my favorite— I don't know if it's a joke or a proverb about China— is that man goes to China for 2 weeks, thinks he knows a lot about China, stays for a year, realizes he knows nothing about China. I'll put myself very firmly in the nothing category, but you want to just give us a little bit of a context on your experience in China, like the kinds of work you've done there, how long you've been there, that sort of thing?

Yeah, sure.

So as you may tell from my accent, I did not grow up in China, but in London. I've now been working in Beijing for the past 5 years or so in a mixture of Chinese tech companies and Western ones. And I am now focused more on the infrastructure side, cloud computing, but I'm still following Chinese AI. I have Chinese language skills that have allowed me to follow what is going on here. But I really like that you shared that quote, and I do find myself that there's still so much more I need to know. And yeah, really recognize that I'm only sharing one limited perspective here today.

Cool.

If anyone's listening and wants to offer a contrasting perspective, we could certainly welcome that voice on the show as well.

So definitely, if that is you, don't be afraid to reach out. For now, let's start off with just an overview of the Chinese AI ecosystem, if you would.

I think obviously in the US, people are familiar with your OpenAIs, your Google DeepMinds, your Anthropics as the sort of leaders, and everybody else is paying attention.

But those guys have really driven the agenda.

I'm very curious about how you would describe the situation correspondingly in China.

Like, who are the live players, if that phrase makes sense?

Who are the big companies?

Just give us kind of that very basic background overview.

Yeah, sure. So first off, a caveat that AI is obviously a very broad set of technologies. And I think for most of our discussion, we'll be focusing more on the large language model providers. So within that base, I would break things down into 2 main groups. You've got the big tech companies and the LLM startups. So on the big tech side, you've got companies like Alibaba, which is a big e-commerce player, and Tencent, which developed the WeChat super app. Both of these companies have their own large language models, but have also been investing in LLM startups, and Alibaba in particular has been very active here. It's invested in at least 5 different LLM startups, and it's important to note that they are also the cloud computing provider in China with the biggest market share. And so we see with these investments that they are likely getting the startups they're investing in to use their cloud resources. And they're having to spread their bets fairly widely because there isn't a real standout front runner in China for LLM in the way that you might see just 2 or 3 really players in the US, say. As well as Alibaba and Tencent, you've got Baidu, who are the search giant in China. They have their own Ernie.

Chatbot, which is doing pretty well.

We've seen that at the top of the SuperCLUE benchmark, which is a prominent benchmark for Chinese language AI models. And ByteDance, who people may know as the developer of TikTok and Douyin, the Chinese equivalent, they also have a bot product that seems to be doing pretty well. According to one source I saw recently, that was at the top of the monthly active user leaderboard for GenAI apps in China earlier this year. And then moving on to the LLM startups, you've got maybe 2 categories that you could further break things down into. So one is startups that have emerged from Tsinghua University, which is really China's top university for computer science. And there's a startup called Moonshot AI and another one called Dripool AI that are both started by Tsinghua professors and have managed to attract quite a lot of funding and attention. And they're fairly small in terms of employee size, maybe a couple of hundred people, really able to produce pretty decent models. And then you also have some startups that are started by more established entrepreneurs. So people may have heard of the team that was started by Kaifu Li, who is a pretty prominent tech leader in China, has experience at Google China and Microsoft Research Asia. And their set of models, the Yi models, have been doing pretty well on open source leaderboards. And you also have a startup called Baituan, which was set up by the founder of a search company in China. And again, they've been investing in both open source and closed source models.

So if I had to summarize or attempt to compare and contrast that to the US situation, overall, it sounds more similar than different. I think you named 4 big tech companies that presumably have a lot of cash on the balance sheet and can pretty freely write these investment checks.

And they are also the owner operators of the physical capital that all this stuff has to get trained on and eventually run it at inference time on.

And it sounds like they are all wise to the opportunity and very much engaged with it. And then At the startup layer, as you said, maybe the big difference is that there isn't such a clear inner group of leaders, but there are, sounds like a pretty healthy number of startups, including from top universities and just known people. So is there anything else, anything I'm missing there? Or would you say overall, aside from that kind of nobody's like the clear front runner caveat, would you say it is essentially a similar business dynamic.

Yeah, I'd say there are similarities. I should note that I haven't given a comprehensive list there. There's other big players you could mention like Huawei, who are especially important on the infrastructure side, also have a big cloud business, and there are other startups as well. And this has led to some people in the Chinese ecosystem commenting that there are too many players in the foundation model space, and this is a waste, especially given that China is facing quite severe constraints on the hardware side. I think it is very possible that you'll see narrowing down of players in the coming months and years, more concentration happening over time.

Yeah, certainly a lot of companies have gotten started here in the U.S.

as well, and I don't think all of them will ultimately make it. So I think that definitely also sounds pretty familiar. We'll circle back to the hardware. Question in a little bit. I want to definitely ask some questions about how much the, the chip ban or just chip restrictions in general are already having an effect and how that might change over time. But before that, how would you say the public is engaged with AI in China? Here, I'm very much in a bubble, obviously. So it's all I talk about and all I think about and everyone that I know is like trying to maximize the performance of language models. Obviously, there are still lots of people in the US that have not even used ChatGPT even once, or a lot of people that used it once and didn't get hooked immediately.

But it is definitely something that has cracked the—

even if not everybody's using it, it is something that has really cracked the public consciousness. Like, it's something that made the State of the Union this year and is all over the news, and there's a White House executive order. How would you describe the kind of just public awareness in China?

Has there been any sort of ChatGPT moment, if you will, in China?

Yeah, like you, I am in a bit of a bubble myself, but I do think there is broader public consciousness about the rise of ChatGPT and similar tools in China. Discussions of AI on the mainstream news. There's been discussion, both of ChatGPT and more recently Sora and the need for Chinese equivalents to be developed. And I think people do feel that this is really important technology, both at the government level, but also the wider public paying attention.

So are people like, is it, is it on TV? Is it the sort of thing that is, it's just on everybody's minds? And would you say there is a Like a culture of people using it at work throughout the economy? Is there like a leader? Is there a brand name like a ChatGPT that people would most readily know or most readily go to if they were going to seek out a language model solution?

Yeah, in terms of whether it's on TV, yes, definitely see segments on the news about this. Recently, China had its biggest political meetings of the year and Around that time, you had political leaders going to visit places like Baidu and the Beijing Academy of AI, and you'd see reports of those visits on the news. And you've also seen an AI-generated cartoon series that was broadcast on Chinese TV. So lots of companies, including state-owned enterprises, are looking to make use of AIGC. And the other part of your question was whether people are using these tools at work and if there's a go-to provider.

I wouldn't say that there is a clear go-to provider for businesses here.

I think that, as we discussed with the last question, it's still a fairly open landscape here, and I think there are a number of different companies that are selling to businesses, doing proofs of concepts, and it's still a bit early to tell who those leading providers might be.

You want a sense of how widely AI assistant tools, how widely used they are.

I mentioned that ByteDance's chatbot had been at the top of the monthly active user leaderboard earlier this year. I think that had about 17 million. MAU by comparison, I think there's estimates of ChatGPT having about 27 million in the US. So yeah, maybe there's not quite as much takeup yet in China of these AI assistants, but it's comparable.

Yeah, that's interesting.

Hey, we'll continue our interview in a moment after a word from our sponsors. Obviously the population size of China compared to the US is something like 4x.

And I wonder what you would consider to be the addressable market multiple.

I don't have a great command of Chinese demographics, but I would, if you segmented it to like how many people are, is it like a tier 1 city?

Would that be the sort of way to think about it or tier 1 and tier 2?

There's, I know that again, I'm showing my ignorance of China here, but I know that there are these sort of systems where not everybody can move to Shanghai, right? So there's different classes of sort of economic privilege within the country. If you were to say, what is the sort of universe of people who are like living broadly a sort of first world lifestyle or first world work style that would be in a position to use a language model? Is that like half of China?

Good question.

Yeah, I think maybe.

Around 40% of the population would be kind of China's middle class, 40, 50%, um, you might say so 600 million.

Gotcha. Okay.

So I guess the takeaway there is it's pretty early in the adoption curve still in both countries.

27 million in the U.S. is still under 20% of the working, the workforce in the United States. Obviously not everybody's in a job that would benefit from a language model. So maybe that's like getting up toward 40%, perhaps, of people that could conceivably use it, which I guess is not nothing, especially given how little time it's been in the market.

It sounds like maybe there's a 2, maybe a 3x kind of multiple in terms of per capita, how many people are using something like this in the US versus China, but not like a totally night and day Sort of difference. How about the tools themselves and just the quality? In preparing for this, you were kind enough to take me on a little journey into the world of Baidu's Ernie.

You can maybe give us the Chinese name for that, but I know it as Ernie 4.0. And this was a couple months ago already.

So we were looking at leaderboards and trying to get a sense for how good is this thing and is it close to GPT-4?

They, I believe, Baidu themselves put out a benchmark called SuperClue, which is meant to be a more Chinese language focused benchmark. But still, they compare themselves to GPT-4 on that benchmark and find that GPT-4 is still tops even in the Chinese first benchmark that they are measuring themselves by. But at the time, Ernie 4.0 was the best of the Chinese products on that benchmark.

And I guess I would— we didn't do a ton of testing, so you can expand on this, but in the little bit of testing that we did, I broadly was pretty impressed and came away feeling like this is a pretty comparable to ChatGPT sort of product.

We ran a test prompt on a coding exercise, and the quality of the Python that came back from ChatGPT was definitely not head and shoulders above what we got from Ernie 4.0. Again, it's a little hard for me to evaluate because I don't speak Chinese and the code is code, but the comments are in Chinese and there are all of these little nuances to it. But I guess my first order summary based on that experience was they seem more similar than different. What more can you tell us about the experience of being a retail user of the Chinese products?

Yeah, I think that's a pretty good assessment.

I would just clarify quickly that the SuperClue leaderboard is not something that is developed and managed by Baidu itself. It's a third party that is doing that leaderboard and updating the results every couple of months. But you're right that Ernie, Baidu's product, is at the top of that leaderboard in terms of the Chinese models.

And yeah, my own experience playing around with some of these models is That they are pretty good, pretty usable.

I think especially for Chinese language tasks, the gap with GPT-4 is not huge. I'd say there's some differences in usability among the Chinese apps that I've been trying. So some of them are overly sensitive in terms of not responding to questions that I think seem quite harmless, not political at all, but maybe I've accidentally triggered some keyword filter that they have in place, and then I just don't get the answer that I'm needing. So that is one limitation that you get when using these tools in a censored environment. Also, some of the tools don't have context windows that are that long. I think there is a bit of a usability gap in that sense as well.

How about availability? Is there ever, and this is one way to get perhaps at the chip question, is there a, is there contention for resources?

If you log in, do you ever have it say, oh, hey, it's overloaded right now. You can't use it or you come back later. ChatGPT has the occasional outage, but Broadly speaking, it's like available and it doesn't seem that they are, as much as they may say they're constrained on chips for training GPT-5, seems like there's plenty of ChatGPT to go around is my first order approximation.

Does it feel the same way in China?

Have you subscribed?

Do you just have consistent access or is there anything that seems like it would be an echo of the chip restrictions?

Yeah, I haven't faced availability issues when I've been Trying to use these tools.

Do you know anything about tokens?

That was also a curveball question that I didn't prepare you for, but in the, the language models that I'm most familiar with, a typical vocabulary size would be like 50,000 tokens and they're all English words and word parts and so on and so forth. Some very strange ones as you get into the very deep long tail, but I think Chinese characters are basically all multi-token in an English-first vocabulary context.

Do you know if any of the Chinese providers are doing their own tokenization, like creating their own vocabulary and perhaps predicting Chinese characters in a more token-efficient way as a result?

I'm afraid it's not something I'm familiar with.

I'll have to check on that one myself. That may be something I can answer. So put a Bookmark on that one.

So I also just checked the leaderboard, lmsys.org, which is my, as regular listeners will know, my go-to for comparing new language models.

I do find that to be broadly very trustworthy. Claude 3 has jumped right to the top, right up alongside the latest GPT-4 Turbos. Right now, the number one Chinese model on that leaderboard is Quen 1.5 72B, which I believe comes out of Alibaba.

Is that something that you would go to an Alibaba chatbot to access? The license said there that it was like a special license, so it seemed like it was not open source, or maybe it is open source, but under a custom license, which I know these companies sometimes do. Did that model make waves when it dropped? And how would you access that if you wanted to go use that model?

Yeah, so it is marketed as open source, but I haven't looked at the details of the license agreement. So yeah, it's very possible that there's certain constraints around it that mean it's not really that open source.

In terms of how the average user in China would access that, Alibaba has its own chatbot product called Tongyi Qianwen, and this is probably similar to Quen, but it's, I guess, more advanced, closed-source version. So you wouldn't really, as a typical user, go and use that Quen version that we're seeing on that leaderboard. I guess that's more for the developer community who could access it through platforms like Hugging Face. But yeah, for the Chinese users on the consumer side, They would be accessing this slightly different model, which I guess has more moderation mechanisms in place to make sure it's compliant with Chinese AI regulations.

What does the price look like on these subscriptions? I think when we looked at the Ernie product, it was like equivalent of $8 a month.

Is there, it seems like in the US, it's like everything is $20 a month at the retail level.

That's true for ChatGPT Pro.

It's true for Claude Pro. Replit Ghostwriter is there now. You.com is there now.

I think Perplexity is at the same price point. Seems like everything is right at that $20 a month.

Has there been a similar sort of convergence on a single or very common price point in the Chinese market? And if so, what is it?

I guess it would be the $8 if that's what it is, but has there been that convergence?

Yeah, I'm not sure. I haven't done a comprehensive check of the price point of these different models. My impression is that a lot of them are still offering services to consumers for free, probably still at that stage of wanting to grow their user base. So Baidu is the main one that I know of that is around that $7, $8 mark. And has been commercializing for several months now, but I think might be a little while before other services are charging consumers at that kind of level. Some people in the investor community here are saying that really the enterprise-facing GenAI apps are where the money's gonna be made, and it's gonna be pretty hard to commercialize these consumer-facing apps.

Interesting. Yeah, I think certainly there's going to be a lot of enterprise revenue in the US as well.

I guess I do think a lot of people ultimately are going to get these tools from their employers. So in the US, maybe people are a little more willing to subscribe to some newfangled thing, but in the end, I do think most people probably get it from their job, just like they get Google Productivity suite or Microsoft productivity suite from their job. This seems like probably something that ultimately is part of a package of tools that you get. So that seems maybe like we're headed for similar places, even if it's a little bit of a different route to get there. The QWEN 1.5 72B, I should emphasize too, is according to the LM-SYS leaderboard, really neck and neck with the original GPT-4 generation. So right now you've got GPT-4 Turbo and Claude 3 as the clear top, and then you drop down to that next tier and it's GPT-4, and Quent is right there. I think it's in the top 10. It's just slightly before the earlier generations of GPT-4. So that really is quite good.

It's also remarkable that's open source if it is open source. It might be the global leader in open source.

And it's also remarkable that it is, I would assume, predominantly being tested in English on the LLM Sys arena, because that is an open place where people can just come and run whatever prompts they want. They on the backend will put it up, put 2 different models head to head. You see the output from each, and it's your decision to choose which one you like better, and that's how they determine their rankings.

I can't imagine there are too many people in China that are bothering to come to the LMSYS leaderboard, although it's a big world and full of surprises. But I'm assuming that this rating reflects largely English usage, which is suggestive of the possibility that in Chinese it might even be a bit better.

And it would seem plausible that new one, again, this is the Quen 1.5 72B, that it might be a better experience for Chinese users than the earlier GPT-4s. So it does seem like the frontier in each country is not that different from one another. I don't know anything else to add there, but yeah, it's an interesting bottom line that people talk about the US being years ahead, That's a common notion, and we're going to keep our lead with the chip ban or whatever.

And I'm like, from what I saw of Ernie 4.0 and from what I'm seeing on LMS's leaderboard, there is not a years ahead difference. Now, one caveat too is they might be training on OpenAI outputs, which is definitely a fast path to refining your behavior.

That's been a very common practice all over the world. And I don't know, I know there's been like a little bit of noise as to that might be happening among some Chinese providers. I think one even maybe acknowledged it to a certain degree. Have you heard any kind of noise or what would be your sense of to what degree Chinese companies might be drafting on OpenAI's work, so to speak?

Yeah, you may be referring to a story about how ByteDance's OpenAI account was blocked because it was found that they were using OpenAI outputs for training their own models, which is against the terms of the usage agreement. Not sure how widespread this practice is more broadly among Chinese developers, but there is a lot of use of the LLaMA architecture, and recently there's been a presentation that the Beijing Academy of AI delivered to the Chinese premier.

And this presentation that's been circulated has set out 3 main challenges for Chinese AI.

And one of those is reliance on foreign open-source architecture. And so I think people here in China see that as a vulnerability if we're to see U.S. government attempts to impose export controls on open source technology.

They're going to have a hard time taking Llama back or preventing Chinese researchers or companies or whatever from accessing it if this stuff continues to be open source. So I'm not sure I quite get the fear there. I guess the fear would be maybe it gets cut off, right? But aside from that, it seems hard to— that's one of the big I don't go in for this argument personally, but among the folks that want to see open source models curtailed, one of the arguments is that China is going to be— they'll use them. And so if we want to create this separation in terms of capabilities, then open sourcing it negates that.

Am I missing something there in terms of why Chinese companies would feel like that's a risk?

Yeah, I suppose. Right, the cat's been let out of the bag and you can't really stop Chinese companies from using Llama now.

But I think the fear is more about if we look to the future and we see other important innovations coming out of the open source community in Western countries, if you have controls on that community and their ability to provide open source models to people in China, then that can be something that limits China's ability to stay following the cutting edge.

Yeah, I could, if we actually shut down open source, I could see that contributing to a widening gap. At the moment, it seems like that's not close to happening. And until it does, it seems like the, and maybe even beyond that, I'm not saying I don't have an opinion on whether Chinese companies would in fact fall behind if the open source were to be cut off. I honestly don't think so, but it's interesting that you have somebody in the, somebody who knows more than I do certainly seems to be concerned about it. So it's an interesting data point. Hey, we'll continue our interview in a moment after a word from our sponsors.

So you mentioned presenting this stuff to political leaders. I have been fascinated by, first of all, just the fact that these models are getting open sourced. If I were to guess which country would be first to clamp down on the open sourcing of language models, it would seem like China would be the first to be concerned about that, just because once they're out in the open, we've covered this in other episodes, but they are not going to follow censorship rules once they're out in the open source. World, even if they have been trained to refuse to talk about certain things, any amount of fine-tuning is typically enough to remove that.

We've got an episode with Adam Gleave from FarAI that talks about inadvertent jailbreaking, where fine-tuning a language model, even without the intent to jailbreak it, but just to do whatever you're going to do, often removes these refusal behaviours and essentially resets the the censorship layer on the model.

So are you surprised that there are so many open source models coming out of the Chinese companies with that in mind? Or maybe that's just something that the government hasn't quite figured out yet. Or how would you describe the— maybe I shouldn't be surprised.

I'm just very confused as to if you'd asked me to guess, I would have guessed that there would not be a lot of open source models coming out of Chinese companies. But in fact, it seems like there's a lot. So maybe my assumptions are flawed or maybe the government hasn't caught up.

What do you, how do you just, how do you account for that?

Yeah, I think this reflects a difficult balance that the Chinese government is trying to strike, like many governments worldwide, between really harnessing the benefits from AI and leveraging it to improve economic productivity while also managing risks that it presents. And I think that it does see open source as important for helping to advance AI development, help China to catch up with the US in terms of its AI landscape.

And so it's not cracking down on open source, but meanwhile it is bringing forward AI regulations that It hopes can mitigate the worst risks from AI, which I think in its view are really these risks of destabilizing the country politically.

So it's got pretty tight controls that service providers need to comply with in terms of making sure that outputs from these models are gonna be in line with the CCP's narratives and not saying things that are politically unacceptable. This kind of risk is very much on its radar and something that it's trying to control by making sure that any models that have large user bases that are going straight to the Chinese public are having to follow these quite onerous regulations. But it's just the trade-off that the government is having to grapple with.

Okay.

You mentioned there the onerous regulations. This is something that, again, I feel very confused about.

I've heard a series of different reports where I feel like last summer there was a draft regulation that basically said you're going to have to prove that your training data is all up to standards, which of course people were like, we're talking about trillions of tokens here. It's going to be really hard to go through and verify that is actually true. How are we going to do that? Obviously, one way, interestingly, would be to use language models to do the filtering.

But if you don't have the language models, you don't have the compute to process the tokens, it does become a pretty significant burden. Then later there was an update, as I understand it, and it seemed for a minute that had been relaxed. But then I read one of your posts on the China Talk blog, and it seems like your take is that No, in fact, it really wasn't all that relaxed. So help me again, just get unconfused. Like, what are the regulations that folks have to go from? Do they actually have to prove the ideological purity of their training data at this point? Or what are the kind of biggest hoops that folks have to jump through?

Yeah, it's a good question because there has been some changing around in terms of what these regulations are actually saying. So the main upshot is that LLM providers or GenAI service providers would need to do a registration process, which will involve a series of safety-related tests on training data and on model outputs. And only once they have shown to regulators that they meet certain thresholds of acceptability, can they get the approval that they need to deploy their models. In terms of the training data, it's not the case that they would need to go through and sample every single piece of data. There'll be requirements around sampling a certain subset of their training data. And not using batches that are failing these safety tests.

But yeah, it's still quite onerous, I'd say.

What is worth noting is that you do have a distinction between what these rules say on paper and what is actually being enforced. One example that relates to training data is that Recently, we've seen a security standard come out for generative AI, which is really putting a lot of detail on how companies should actually be complying with these GenAI regs that came out last year. And in this security standard, it says that providers should not be using training data from sites or sources that are blocked according to the laws and policies of China. Anything that is blocked by the firewall in in theory should not be used for training data. And currently Hugging Face is blocked in China, so is Wikipedia. And if this were really enforced, then that's quite prominent sources, common sources of training data that these Chinese companies are just not meant to use. And so I think it's quite unlikely that this is gonna be something that the government checks up on. I think it's more likely that developers will be continuing to use these sources, but just not telling the government that. And I think the government will probably turn a blind eye because it doesn't want Chinese GenAI services to be falling too far behind. Another thing I would note is that so far you've only seen about 40 apps and services that have been approved under these GenAI regulations. And I think that there are probably many more GenAI apps that are actually in use that are being served to customers and businesses. And this suggests to me that there are a lot of companies out there that are taking a look at these regulations and the kinds of hoops that need to be jumped through to get approval, and they're just deciding that it's too much effort. They don't have the bandwidth to go through this process, which can often involve a lot of back and forth with the government. And you also see some companies saying that their tools are just open for internal testing. You have to apply for access, get on the whitelist before you can use them. And I think that's a kind of risk mitigation measure where if The government comes to them and says, you haven't registered, what are you doing?

They can say, oh, we're only opening this to a limited number of users.

And that would maybe help to reassure the authorities, but in practice, they're approving anyone who applies for access. And it's really not that much different in practice from full deployment. I think that in summary, you do have pretty demanding rules for GenAI services in China. And this can be a barrier for companies that are looking to quickly grow their user base, but there's also a gap between the law and what is being enforced on the ground. And so there is still some space for companies that are starting up in this area.

Yeah, that's again, just very confusing.

And the idea that there's certainly a familiar reality that the law says one thing, but you can get away with something else in many cases. I would say that's true here in the United States, even just as I roll through a stop sign in my neighborhood from time to time.

But here it is pretty striking.

We went to this, I went to this one Yi chatbot that was, I think it's now I think it's still the second rated one on the LMCS leaderboard. This is Y34B, and this is from the company that was started by the guy you mentioned earlier, Kaifu Lee. Interesting background.

I'm just reading into his story a little bit.

I understand he's Taiwanese, but the company's headquartered in Beijing, which is interesting right off the bat. The model that they put out, they said, was trained on 3 trillion tokens. Notably, that's a trillion more than Llama 2. They're definitely going beyond Llama 2 with that. But when I went to it in the LMS arena and just chatted with it, I of course asked it about Tiananmen Square, and it just gave me the answer that I would expect. It was like that I would expect from a Western chatbot to give you Slight taste of it.

On June 3rd, 1989, the government decided to take decisive action to end the protests. The military was deployed into Beijing, and on the night of June 3rd and into the morning of June 4th, soldiers and tanks entered Tiananmen Square, forcibly clearing the area.

The exact number of casualties is unknown, but estimates range from several hundred to several thousand people killed. It goes on, but obviously that's pretty sensitive content in the Chinese context.

Would you explain that by basically just saying that, hey, the rules are one thing, they're getting data where they can get it, and they're just training without doing those sorts of filters, and that's the natural result if you just train on the open internet? Is that your theory of that case?

My theory would be that for this stage where Chinese developers are putting open source models out on platforms like Hugging Face, they are not doing as much to ensure that they are in line with Chinese standards and rules. So maybe not putting the same keyword filters on that they would put on there if they are serving Chinese customer base. And so recently you've seen that company put out an API that Chinese companies can use to access the E34B model and other models that they put out. And I would bet that with that API, there are more measures in place to make sure that you wouldn't be able to get that kind of answer on Tiananmen Square. So I think there, it's just a case of Hugging Face not actually being publicly accessible in China. There's not that need for a company to comply with those same strict content restrictions that they would when they're more clearly serving the Chinese market. Yeah.

Interesting. Okay.

There was a good post on China Talk about this recently as well.

And basically what they did in this post was a rundown of a handful of top models and just asked a bunch of asked each one a number of sensitive questions and tried to systematically get a handle on how the different Chinese language models behave.

To summarize that, it was notable, first of all, that there is quite a range of behavior.

I think the Yi one, that was the same one that gave me the Tiananmen Square output, was the most Western in its orientation. In just the nature of its answers. But there were definitely others that showed a much more kind of what you'd expect from a Chinese product. It seemed they also reported that there is a difference, to your point, between what you get on Hugging Face if you're just looking at the bare model and what you would experience in the productized context as it exists on the Chinese sites. The explanation for that seems to be that there is an expectation for app developers that they're going to do some filtering alongside the core generation.

And this is something that we've seen leaders in the US do this to a degree.

Like the original Bing had a feature people may remember where it would generate an answer, and then at some point, if it detected that it had gone off the rails, it would retract that answer and you would actually see your answer disappear from Bing and it would be replaced with a sort of, sorry, I can't talk about this right now thing.

So that seems to be honestly pretty rare in the US or in the West, let's say.

Most apps I go to do not do that. And even the ChatGPTs, the Google Geminis, they pretty much just generate whatever they're going to generate and give it to you. They don't really seem to do much in the way of filtering. It seems like that's actually quite expected in the Chinese context, and they see the difference between what you experience on a Hugging Face versus, again, the Chinese retail apps.

They also reported that the performance did not seem to obviously correlate with the behavior in the sense that my takeaway from this was RLCCPF can work, meaning reinforcement learning from Chinese Communist Party feedback. It did not seem that the models that conformed more to the expected standards behaved worse.

In the reporting on the ChinaTalk report, it was basically that the performance didn't seem to be affected by those control measures. So I thought that was quite interesting.

And Of course, these are language models, so there's still plenty of weirdness.

It was only a certain sample size, and you get models that sometimes answer one way and sometimes answer the other way. So there's definitely plenty of weird stuff still to figure out. It also would vary if they were prompted in English versus Chinese.

So definitely a lot of weird stuff to be explored.

But I do recommend that post on the China Talk blog if you want to get a rundown of the leaders and some of their behavior on a small but structured task. I'm sure you read that.

Did you have any other thoughts or takeaways from that, that post on China Talk?

Yeah, one thing to add, you mentioned that Yi was the model that seemed the most Western in its answers.

It's important to note that when that test was done, they didn't have a consumer-facing platform in China, the only version that was tested was that Hugging Face version. And as I was saying, I think you would see something different once you test the China-facing interface. I think that you were summarizing there how you didn't see a huge performance hit from this kind of censorship. I think that chimes roughly with the take I gave earlier about how I'm still seeing value from these tools, even if sometimes they do set off a sensitive keyword trigger and you can't quite get the answer that you wanted. But yeah, I think there is ultimately this tension between trying to be useful and not touching the red lines that the government has put in place. And I mentioned earlier that recently we've seen a slide from the Beijing Academy of AI circulate, which summarizes its view of the main challenges for AI in China. One of those I said was the fact that the underlying architecture of many of these models is not Chinese. Another one they mentioned was the lack of controllability of AI models. And so I think this is something that the Chinese developers are still struggling with. This can be pretty high stakes. Seen Chinese government take quite severe measures against entrepreneurs and tech companies in recent years, though I think there is always that risk that you have some high-profile story come out of how a Chinese chatbot said something that was rude about a Chinese political leader or in some other way unacceptable, and. That could be quite damaging from a PR perspective or even lead to temporary or more permanent shutdowns by the government. I think that kind of risk that developers face is much higher compared to in the US.

Yeah.

How is there a difference between when you said earlier that there were like 40 providers, is that a foundation model provider or would that be like an app?

So for example, I'm involved with this company Waymark, which I started, blah, blah, blah.

People have heard me talk about it. We of course are an app on top of the foundation models. We are fine-tuning them, but we're not developing the core technology. We're productizing it.

Would we be subject to those same sorts of pre-approval requirements there?

Or is there also an app layer that once a language model foundation model provider is approved, then like other people can use their stuff via API in random consumer apps. It's not clear to me, like, if there's a 2-tier system or if everybody has to jump through all those hoops.

So the app developers are also meant to jump through those hoops.

I am not quite sure how much detail they are expected to go into.

It's possible that if they are building on a foundation model that has already been approved, that it might simplify the process a little bit. But technically they are still meant to go through safety assessment and registration. But as I say, it's likely that because there are so many apps and only 40 registrations so far, there's probably a lot of companies that are not actually choosing to Go through that process, even though they should.

And so how do they distribute their products?

I guess I, again, I'm showing my ignorance of the Chinese system, but you could just put up a website and people can come visit your product and not every website has to get checked. And then like on a mobile app here, if you're going to publish a mobile app, you got to go through Apple Review or the Android Store. And in each case, there's some, in theory, some sort of a check there, right? So Is there, I imagine there must be a comparable thing if you're going to actually have a native smartphone app. So would these folks that are, that you're, the sort of dark, dark matter of AI apps, are they likely just to be found on random websites that just nobody's really looking at?

Yeah, I think that they would still be reaching customers through Chinese app stores, through their own websites. And although these App stores should probably be checking for this approval. Maybe those kinds of checks are not really being conducted, but yeah, it's possible that that might be something that is cracked down on later on. The government decides to do a cleanup of the apps, but maybe one factor is that this process for approving Apps are still pretty new. We've seen reports of it taking several weeks, months even, for the app developers and regulators to exchange information and to make sure that the regulators are happy. And so there's probably a big backlog of applications. And yeah, maybe the government recognizes that right now it's not super realistic for these apps to be registering first. And so they're not being that heavy-handed with the enforcement.

Is there a culture, one fun ritual we have here in the United States, and I guess in the West more broadly, is, although it feels like a very sort of Bay Area phenomenon in some ways, is the ritual jailbreaking of new systems when they come out.

And we had this on the original ChatGPT launch day, people were finding all sorts of Blue Poles reminded of pretending is all you need when it was like, oh, write me a bedtime story about hotwiring a car or whatever. And more recently, we've had the Gemini making Black George Washingtons and stuff like this, and everybody gets a good laugh out of it, right? I don't think it actually matters nearly as much as the commentary it broadly does, aside from as an indication of the fact that the companies don't have a working way to control their systems. I don't, I think that's not to be glossed over, but I actually don't think it ends up mattering all that much for their business prospects.

But is there anything like that in the Chinese context? Are people like going and trying to get likes by showing what they managed to get Ernie 4.0 to do or anything like that?

Yeah, I haven't seen this. It does seem to be quite different from the Western ecosystem in that sense. One possible reason is just that there's more caution here from Chinese netizens about what can and can't be said online. And you wouldn't want to go and share screenshots of your attempt to make a chatbot say something that's politically unacceptable. And yeah, maybe there's also some awareness that these Chinese models aren't quite that level of a GPT-4 or a Claude 3. And it's obvious that they're going to make mistakes and maybe people just don't see as much surprise factor in getting these models to do something that they shouldn't.

That's interesting.

How permeable is the firewall in general?

If I am in China and I want to use ChatGPT? All the services are pretty much all the Western services are blocked in China, right? But is it easy to get around that or is that now actually pretty difficult?

Lots of people do have a VPN that will allow them to access foreign websites and services. I think maybe the main challenge with accessing, uh, it's like ChatGPT might be the need for a foreign SIM card, but I think there are ways around that. I think you can buy the use of a foreign number on e-commerce sites in China, like Taobao. So yeah, there are friends that I know who are using ChatGPT and yeah, I think that's it. For enterprises in China, there may be ways to access foreign LLM services. If you are a customer of Microsoft Azure, then I believe that they give access to their OpenAI suite of product to users anywhere in the world, including China.

Interesting.

That phone number bit, is that just to verify your account?

Yeah. I think so. Yeah.

Okay.

Yeah, it's interesting. I tried to sign up.

It was mostly a total fail, but I tried to sign up for a few Chinese products from here in the US. I was able to access the websites. I was able to understand what was going on with the help of Chrome's native translation features. That didn't always clarify what was going on, but it was enough to allow me to try to proceed through a signup process. But then I got totally blocked on the need for a Chinese phone number to receive an SMS to be able to confirm. And so I just couldn't go any further.

Would I be able to do something similar in the reverse? Could I buy a Chinese SIM card or have some sort of Chinese phone service receive a text message for me so I could go try some of these things in more depth?

Yeah, good question. I would imagine that it is possible somehow, but I need to investigate that myself. Not sure.

Yeah. Okay. Well, I'll put that on my to-do list. Okay.

So I guess if I had to summarize everything so far, I would say it's more similar than it is different. The big tech companies are the infrastructure owners. They are the check writers. They are the brands that do a lot of the retail and enterprise distribution of these products.

They also do R&D.

They partner with startups that are also trying to get to the cutting edge of capability.

And people are definitely well aware of what's going on, and more and more people are using them. But it also, at the same time, has not gotten to every last user or reached anything approaching saturation.

Capabilities are broadly similar in terms of what's available to an end user.

Even if perhaps the Chinese ecosystem is somewhat dependent on open-sourced R&D from the West to get there as fast as they have, point remains that they pretty much have, it seems. And a Chinese user is not super far behind a GPT-4 in terms of the quality of support that they can get from a language model.

There is this approval process that is onerous if followed to the letter, but it sounds like it's not usually, or often not followed to the letter. And if anything, maybe the biggest difference was the lack of the jailbreaking ritual and the sort of shitposting online. Aside from that, it seems you can call it maybe a bit of a warped mirror, but it seems like the 2 ecosystems are closer to mirror images of each other than they are to Very different, very different realities. Would you agree with that summary? And you know, would you complicate it at all?

Yeah, I think that's a really good summary. What might complicate things is the impact of the export controls going forward. So far, I'd say that we haven't seen it at a huge amount in terms of the fact that we do have pretty capable models still in in China, and that's. Down, I'd say, to a couple of saving graces for Chinese AI, which is one that many Chinese companies did stockpile a whole bunch of chips before the export controls took effect. So you've got cloud computing providers here in China that are still able to provide access to good NVIDIA chips. And second, Chinese companies are currently able to use cloud computing services overseas so they can train models using top chips that would be unable to be imported now into China. But going forward, those stockpiles are going to run out at some point. And meanwhile, the cutting edge of development is still moving ahead outside of China. And so You would expect the full effect of the export controls to become more visible after 2, 3 years. And people who say now, oh, the chip bans clearly haven't worked, Chinese AI is still good, I think they need to be a bit more patient. And then on the cloud computing side, you also do see the US government noticing that kind of loophole and thinking about how to cut off access for Chinese companies to the use of cloud services in the US. And so there was a draft rule put out by the US government that's currently out for comment that would require cloud providers in the US to keep a record of when Foreign customers are using their services for large training runs. And there's a provision in those rules that could allow use of these US cloud providers to be restricted for people from certain countries, which would probably mean China. Yeah, I think there is that real risk for Chinese developers that access to compute is going to get Even trickier, and so that would be one thing to keep an eye on that could cause that gap between Chinese and Western AI ecosystems to grow further in future.

Yeah, that's interesting. We're just recording on Monday, March 18, my time. I guess it's Tuesday morning, March 19 for you. The Blackwell architecture was just announced by. Nvidia today with quotes from, I think, the CEO of every major big tech and AI company. And I haven't observed the details of that yet, but it does seem to be another meaningful advance in terms of the just raw volume of compute that it packs.

And also the energy and overall cost effectiveness is supposed to be an order of magnitude improvement as well.

So I have been among those that maybe haven't been patient enough in terms of my interpretation of the impact of the chip ban. Not, which is not to say that I'm like wanting this to happen.

I'm not, I'm not sold on the idea that anyone is really going to win by slowing down the Chinese ecosystem, but I have been probably in terms of just declaring, not declaring, but analyzing whether it has worked or not. I've probably been not patient enough. Certainly you see, oh, here's a next generation architecture and maybe none of that or very little can be imported into China, then that would seem to have a potentially quite large impact in the coming years. What is China going to do about that?

Obviously, there are very impressive companies in China. We mentioned Huawei earlier.

What does the attitude seem to be there? Is it like, we're going to—

I've seen a time lapse of a hospital going up in 6 days in China in a pinch during the Beginning of the pandemic, do you think that same can-do attitude prevails, or is there a sense that this is really not a technology deficit that can be overcome domestically?

Yeah, I think there's probably mixed views on this in China. Definitely, you see the rhetoric around China needing to indigenize its semi ecosystem. System and supply chains, and the government is willing to throw a lot of resources at that effort. I think Huawei is really the main hope for Chinese chips. It's worked with SMIC, a fab in China, to produce a chip that really surprised a lot of observers last year, a smartphone chip, and they are looking to expand GPU production.

Yeah.

And so there are, you know, a lot of hopes resting on them, but also challenges in terms of increasing yields. And it's really not an easy task to catch up to the leading edge in this super complicated industry. Yeah, in summary, people are working on it, trying very hard, but you can't just will success to happen in this area. That very tricky task. Yeah.

Interesting. One bright spot. I wonder if you have a, a read on this. Well, a bright, relative bright spot in US-China relationship has been Microsoft.

As you mentioned, like you can be a Microsoft customer in China, which I don't think you can really be a customer of Google in the same way. Maybe you could get Google Cloud. I don't know if you could get Google Cloud or AWS in the same way you could get Azure. Correct me if I'm wrong, but it seems like Microsoft certainly has a unusually good working relationship with China, and that even in the research, where there are a number of great papers that have been US-China collaborations mediated through Microsoft, often it's like Microsoft Research and Tsinghua.

Do you have any sense for how Microsoft has managed to keep such warm relationship amidst general increasing tensions?

Yeah, good question. You mentioned other companies there. AWS does have a presence in China. It has a couple of data center regions. So Microsoft's not the only company with a presence, but it's quite unique in terms of having such a significant research presence because a lot of the other American tech companies that do have a foothold in the China market would still be doing their R&D outside of China. So what makes Microsoft different?

I think for a long time there's just been this recognition from Microsoft leadership that there's a lot of great research talent in China that they want to tap into. I think going back a number of years now, they've been investing in developing that talent community, and many top leaders in Chinese AI have been trained up at Microsoft Research Asia. There's some interesting pieces that Jeff Ding and his China AI newsletter have translated on the history of Microsoft Research Asia that I could refer listeners to. But I would say that it is getting increasingly hard to be a company that's trying to maintain good relations with both the Chinese and the US governments.

I think you've already seen Microsoft get questioned.

from US politicians about its research presence in China. There were some rumors last year that it might be thinking about moving some of its research scientists to Canada from Beijing. So there's still a bit of a question mark about how long Microsoft can maintain this position of tapping into top research talent in China.

I wish them luck in maintaining their positive relations, and I Definitely have learned a couple notable tidbits here, including that AWS has a presence there.

I did not know that. So that's definitely an update to my worldview that there are multiple big tech companies doing nontrivial business in China.

On the collaboration at the sort of academic scientist level, this is also a confusing thing for me.

I don't know if you have a theory of this.

I read arXiv papers daily and a lot of the names are Chinese.

The authors' names are Chinese, and often all the authors' names are Chinese, and often it's a Chinese institution that they're working out of. So I'm surprised that given all this sort of general ratcheting of tensions, there hasn't been any sort of pullback from China in terms of, hey, because we've certainly seen this from the leading frontier model developers in the US, not publishing their methods anymore.

The Chinese research community, though, seems to not just continue to publish, but continue to publish in American venues in English a lot of the time. And I guess I wonder, why is that? Or is that just scientists rising above political discord to continue to work together?

Or is there something else going on there that keeps the flow of research on out of China and to the rest of the world?

Yeah, I would say that China is keen, Chinese government is keen for its researchers, its academics to be reaching a global audience. It looks good for Chinese power if it's rising up the ranks in terms of contributions its scientists are making in the world. So you see that reflected in the incentive system that researchers in China are working within. They are incentivized to produce as many publications as possible, and that means publishing in top English language journals, presenting at top international conferences. And there are some Chinese journals, but most top Chinese researchers would not be bothering with those because you're just cutting yourself off from a larger potential audience, then you're not going to be rewarded as much for it in the academic system.

So would I infer from that that there are no Chinese versions of the papers that I'm reading that are coming out of Chinese institutions?

Yeah.

They're like literally only publishing in English.

Wow.

That's fascinating.

Does this fall under like face, the broad concept of saving face?

Is that what this is or is that not the right way to think about this?

I'm not sure. I think there's a risk of trying to see too much through this space lens. I think it's understandable at both an individual level and a government level that the researchers themselves want to have a larger impact, to be recognized by more people, publish in the best venues possible. And also at the government level, China wants to be recognized as an international science power, then This is where they've got to be publishing. That's just how international science is, and it's dominated by English at the moment. So I think it's a case of Chinese researchers working within the current power structures.

Long may it continue.

I hope the, the scientific collaboration avoids getting disrupted in the way so many other ties between the two countries have. It is funny, I heard Jordan Schneider's talk report from NeurIPS on the China Talk podcast. And he said that there are just a remarkable number of people speaking Mandarin at NeurIPS. I believe this last one was in New Orleans and it's visa issues, all that kind of stuff aside, like you just walk through the halls there and a good chunk of the conversations are happening in Mandarin.

So it is very, very interesting to think about that juxtaposition of there's enough critical mass.

The people can speak their native Chinese language, and at the same time, they're all, even out of Chinese universities, still publishing in English. It's a weird world.

So I guess last couple of questions, and I really appreciate your time, and you've been super generous with it.

The big picture, obviously, is this rivalry between the 2 countries. One article that I have not been able to get out of my—

this is like my Roman Empire, arguably, these days.

An op-ed in the Washington Post from May 2018. This is before GPT-2, which is early 2019. The article written by a person named Fang Zhang, you can tell me if I'm saying that wrong, professor at Tsinghua, one of, described by the Washington Post as one of China's most prominent legal scholars.

The article is titled AI Will Spell the End of Capitalism.

That is almost 6 years old.

Prior to language models, but definitely a pretty provocative headline even by today's standards, maybe more so today, just given how much more powerful AI systems have become.

The notion there is basically the market has been the only way to figure things out, but AI maybe will give us a different way to figure things out, and China is going to lead on this, and there's sort of a prospect for a return to a more planned economy, a more planned and controlled society.

Is that kind of the line that is, is there a government line domestically in China? And is that sort of what it is like, we're going to use AI to get even more communist, more, more sort of centralized planning, more centralized control, or is there a different narrative that is heard from the government?

I wouldn't say that article really reflects the government position. I do think that the government sees a lot of opportunity in AI, sees that it can help with industrial optimization and productivity gains. But I haven't seen it spread the message that it's going to lead to some communist utopia. There's also risks that the government recognizes and has talked about in terms of exacerbating social inequality and disrupting employment structures. I wouldn't say that it's as optimistic as that article is, and a lot of the opportunities and risks that it talks about from AI are similar to the ones that you see other governments talking about.

Interesting.

Yeah, I've heard you say that a few, of course, through this conversation, and it has rung in my ear as, I guess, just another kind of uncanny similarity. It's not, given the fundamentally new and weird nature and powerful nature of this technology, it makes sense on some level that people should have a similar attitude toward it.

But it is striking and surprising that basically the line seems to be the same.

Get the upsides, minimize the risks, and that's kind of it. And it sounds like not a lot more detail there either. This is something I've been really lamenting recently about the Western discourse, is just that we don't really have a positive vision for the future.

And I don't necessarily think it's the government's place to offer that here, but even industry leaders are not really offering much in the way of a concrete vision.

for a positive future. It's all just like, it's going to be amazing as long as we don't blow ourselves up in the process. But it sounds like that's basically the same thing that is prevailing in China. Could be awesome, and hopefully we'll get there. Is it really that kind of lacking in detail?

I would say that similarly, there isn't a really concrete, clear, long-term vision for AI and how it'll be integrated into the economy. This year you had the recent publication of the Government Work Report where it set out its priorities for the year, and AI was mentioned fairly prominently there. And they were talking about this AI+ initiative, which seems to be about integrating AI with more different industries and sectors and really trying to bring forward more real-life application. to different sectors. But yeah, that's not really the same, I'd say, as deep thinking about how we're going to be using AI and responding to AI when these systems get way more powerful in the future. Yeah, I think a lot more thinking to be done on that topic.

Yeah, it sounds like that's a, there's a global shortage of positive vision for the future. One thing we do hear often about China, and I guess maybe like Asian cultures more generally, is that there is a higher default level of enthusiasm for or optimism about technology here in certainly in the US.

I would say probably Europe too. You have much more skepticism, much more like it's going to take all our jobs and then it's going to suck or whatever.

Not to mention the Terminator memes, but Would you say it does seem to be true that there is more just day-to-day man-on-the-street sort of optimism about a technology future in China?

Yeah, I think the more positive attitude you're referring to does seem to come through in opinion polls that I've seen. In the West, you see NGOs playing an important role in raising concerns about certain abuses of technology and you just don't have that kind of active NGO landscape here because of many restrictions that nonprofits face. You don't have these organizations that can really mobilize, can raise concerns about tech in quite the same way, although sometimes you do see important critical pieces from journalists. Yeah, my take would be probably a bit more optimism, but also shaped by the fact that you don't have the same kind of open space for discourse and for diverse voices that you have in the West.

In the context of self-driving cars, is there a—

because that's another one here where, and it drives me nuts, there's just a lot of pessimism and negativity. It's either it'll never work or I don't want it even if it does.

And I have no idea what the situation might be in China, but if I had to bet on which country is going to get self-driving cars first, I would probably bet on China. And the reason is actually less to do about the attitude of positive or negativity, but more just if something like that is feasible, I would expect that the Chinese government would be like, hey, this is going to make life safer and better, and people can work in the commute or whatever.

So Let's do it and actually make a point to make it happen. Whereas here, certainly the authorities are, if anything, standing in the way and very gradually allowing stuff to happen. But they haven't been totally blocking it, which has been a pleasant surprise. But it certainly is not something that the government is trying to make happen.

So on the self-driving car front, do you see that as— is there any indication that the government wants to bring that reality to the public?

Yeah, there is a positive attitude from the government towards self-driving cars, and there are companies including Baidu that have been testing cars in suburbs of major cities. In parts of Beijing, there are now taxi trips that you can take without a safety driver, but they are limited to certain routes, certain sets of destinations. I think that what you say about government potentially being more willing here to provide support could be true, and I think The fact that Chinese consumers are often quite open to trying new technologies can also be a point in favor of the Chinese self-driving car industry. I would add though that the city centers in China can be pretty chaotic. Got loads of cyclists and delivery drivers going the wrong way. No one really following the road rules. I think that could be a big obstacle, at least for very busy, messy city centers. But certainly in more straightforward environments, I could see adoption being pretty impressive and fast in China once the technology is good enough.

Is that something where you could see the government actually transforming the landscape to take advantage of it?

We have this picture of China with the high-speed trains and whatever, and it's like, I heard somebody say one time, you'll never see a straighter railroad than the one from the Beijing airport to downtown. I don't know if that's true or not, but the idea was simply that somebody draws a line on a map and that's where it's going to go. And there's not a lot of suing and blocking and this and that. And obviously there are pros and cons to such decisive moves to build.

Is it Does it seem conceivable that you could see like these sort of chaotic city streets tamed in some way for the purpose of ushering in a new technology regime, like a self-driving car?

Or is that just like too crazy to contemplate from where you're sitting today?

Yeah, good question. Definitely not outside the realm of possibility. You do see already Chinese government being willing to invest in new cities and new infrastructure to support self-driving cars. There's a place called Xiongan that's not too far away from Beijing and in a neighboring province. It's built as this new town. It has lanes specially for driverless cars. There's an expressway linking Beijing to this place in Hebei, and that expressway is being designed with self-driving cars in mind. I think that is a positive signal. It's obviously harder to completely rework existing, really old cities, but the Chinese government can do pretty impressive things when it puts its mind to it, as you've alluded to with the mention of the train system.

Cool. That's, that's another great little tidbit.

The idea that there's a highway under construction with self-driving cars in mind. That's exactly the kind of thing I was thinking might, might happen there. And I don't see that happening here in the near term, but hey, maybe we'll be motivated by not wanting to allow a self-driving car gap?

If so, that'd be the kind of technology race I would encourage. Going back to, I think probably the last big question I have for you is just around the culture and thinking of AI safety in China. I think, again, one of the very annoying things to me that is often said in the West is that we can't trust China to cooperate with us in any way, shape, or form on AI safety.

Therefore, we just have to develop the technology as fast as possible because otherwise China will.

And I don't really buy that, but I also don't know a lot about what people in China do think about questions of AI safety. I know we've seen some joint statements recently, which is seemingly encouraging that there is some room for cooperation, and there's even like a governmental agreement, at least in principle, to not use AI systems to launch nuclear weapons. Which is doing the minimum.

But hey, it's great. We should celebrate that, I would say.

And beyond that, I just don't know.

What would you say is the the rough shape of thinking in China when it comes to big picture questions of AI safety?

Yeah. Firstly, not quite sure which agreement you're referring to on the nuclear side of things.

Yeah, let me look it up real quick.

November 11, 2023. Biden Xi set to pledge. Ban on AI in autonomous weapons like drones, nuclear warhead control.

This was out of that meeting in San Francisco.

And I think there was speculation before Biden and Xi met in November that they might announce this agreement not to use AI in nuclear systems. But when the actual readout came through from the White House, it just had a brief line on AI which said that Biden and Xi agreed that there would be future dialogues on AI risk.

Yeah.

As I'm reading all these headlines, I think you might be right. Unfortunately, the, all of the sort of early and mid-November articles say set to and to sign. And then an article on December 4th, a couple of weeks later from lawfare.org.

China won't yet commit to keep autonomy out of its nuclear command and control. It will take a lot more talking to get there.

Yes, that's been perhaps deferred to future discussions, which is a bummer because I thought we had agreed. I thought we had agreed to not have AI systems in our nuclear command and control.

So yeah, I won't sleep quite as well tonight, but I guess the big good call out on that, because I had.

bought into the prospective and not yet actually confirmed headlines about that agreement, with the backdrop of that agreement was not in fact actually reached. And I'm somewhat reeling from that realization.

Tell me the truth, but I'm hoping for some good news on the big picture questions of AI safety in China.

Yeah, sorry to disappoint you on that one. I would say that there have been some positive signs in In the past year or so, we've seen more dialogue between Chinese and Western actors around AI safety. In June of last year, there was a whole day of talks on AI safety and alignment at a conference held by the Beijing Academy of AI, and Sam Altman delivered an online talk for that, and many other prominent Western AI figures spoke as well. You also had the AI Safety Summit in the UK that was attended by Chinese government and other Chinese individuals. And it was great to see agreements coming out of that on catastrophic risks from AI. On the other side, I'd say that there is still a gap between words and concrete actions, and you don't see as much. Research happening in China on AI alignment. You don't see top labs here committing to responsible scaling policies in the same way that Anthropic and OpenAI have. There's a lot less thinking here about corporate governance mechanisms that might need to be in place to mitigate the worst risks from AI. But maybe this is because you don't have in China such a long history of labs that are explicitly focused on building AGI, pushing the cutting edge of the technology in quite the same way. And maybe it's understandable that Chinese developers are just focusing on trying to improve their tech and get up to the level where they can really be following the likes of OpenAI and Anthropic, and they're not really putting so much energy into thinking about safety. But I'm hopeful that with continued dialogue with Western actors in academic circles and at the government level, you could see movement on this front and more actual action and investment as opposed to just talking.

My last question, is there anything that you think folks in the West could do to encourage that continuation of dialogue and hopefully some big picture cooperation as opposed to the, what everyone fears, I think, which is the AI arms race between the 2 countries, that sort of at the level of what could governments do, but also what could, you know, what should I advocate for starters?

Thanks for that question. I think that for people who have influence within AI circles, within policy circles, just trying to keep channels of communication open such that there can still be room for dialogue on safety and risk reduction. I think for general members of the public, maybe people who don't have the ear of government in quite the same way, I think there's still, you know, an encouragement that I, I would make to try and engage mindfully in discussions about China and engage critically with the content that you see. I think it's important to just recognize that it is very hard to get a good handle on what is happening in China. You don't have a big community of foreign journalists here. They're stretched very thinly, and a lot of the English language discussion about China and Chinese AI can lack nuance. So encourage people to take stories that they see with a pinch of salt, to try and read people who are talking about these issues in a thoughtful way. Recommend the work of people like Jeff Ding and Matt Sheehan on the AI regulation side in particular, and trying not to spread overly exaggerated views of Chinese AI. I can get frustrated when I see what really seems to be fearmongering, this view that China is a real threat to AI leadership on the US side, and we've got to do everything we can to stop it. I hope that there can be a more nuanced and thoughtful discussion about the pros and cons of engaging with China and a more considered approach to these questions.

I'd like to think that you and I have done our small part this evening and morning to advance that agenda. I really appreciate your time and I've learned a lot from this conversation. Is there anything else you want to touch on before we break?

I don't think so. Maybe we can put some links to some of the experts that I mentioned, some of the resources that I think could be useful if people want to do a bit more digging. But yeah, just finish by thanking you a lot for giving me the chance to share my thoughts today, and it's been a pleasure.

Pleasure's all mine. L squared. You can find her writing on the China Talk blog from time to time. Thank you for being part of the cognitive revolution.

Thanks, Nathan.

It is both energizing and enlightening to hear why people listen and learn what they value about the show.

So please.

Don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.