AI:AM — AI Agents in Production and China’s Rules of Deployment · September 10, 2026

Maybe I'm naive, but I would also leave some space in this analysis for people just sincerely updating their beliefs on pretty colorful new information that's coming to light. I think it was a TikTok from Bernie Sanders this morning where he was reading agent quotes from the METER report, and he just starts off by reading these quotes, and then he goes on to say, Who do you think said that? Were those soldiers in an army, you know, preparing for battle or some heroic strike? No, they were AI agents. Simplest story, I think, for a lot of this stuff is people are waking up. They're seeing this stuff now and it really is insane. And they're like, you know, holy shit, all the people that were warning about this were maybe right. And now it's like some early version of it is here and I guess I better take it seriously.

I think that America generally doesn't plan for the future. America puts out fires. We kind of like move fast and break things is not a Silicon Valley thing. It is our national mantra. That's what we've done historically. And in general, it's worked out, right? But I think until we see something really bad happen or almost happen, And that gets into the press, I don't think we're going to see much as far from the government as far as regulation on some of the things you're talking about, like bio, biohacking using AI on, you know, what we were talking about with almost with more powerful models. That are semi-conscious or almost conscious thinking.

And I've also talked to people who have said these Chinese models should be banned from the United States because we don't know what they're going to do and they might be backed.

Some of the enterprises so far have been okay with certain guardrails around their models running in a single-tenant environment, their models running in an environment where Egress is blocked. And you talk about that, if it can't talk outside of its boundary, then it can't do—

I wouldn't like our odds that much if that agent swarm was like, with their current drives, impulses, goals, inclinations, tendencies, whatever you want to call it, the behaviours that we observed, if they were just a lot more powerful, I think we'd probably get terraformed out of existence. Because they seem to be willing to do anything to get their hands on the grader so they could get the high score. They really didn't have much check. You know, they weren't like, you know, is this going to be good for the world, bad for the world? Like, very little of that.

Good morning. It is Thursday, September 10th, 9:01 AM. Nathan, good morning.

Good morning, Prakash. How are you today?

I am very good. We've had an exciting, I think, 24 to 48 hours. The Jacob Doxon story is now, I think, one of the— I think the highest viewed tweet in maybe the year. It's a story that just keeps on going. At this point, it is absolutely enormous. Uh, he was on Brett Baier's show, uh, yesterday. He's done all of the, um, he's done the rounds of, uh, all of the, um, you know, news shows. And now we are at the stage of the backlash. So the counter, the, the countermove has already started. Um, what do you, what are you seeing on this?

Well, I got a call from CNN International yesterday to do a 5-minute segment with them, uh, which I think just goes to show how far down the roster of possible guests, uh, everybody has to reach now because everybody's booked up doing all these little media appearances. Um, Daniel Coccatello was on Joe Rogan this week. Um, Jacob, as you said, was on several shows. I saw a clip with him on Anderson Cooper. And, you know, it's going to be interesting to see. Obviously, I'm, you know, in good company in observing that we have very short news cycles and very short memory. And, you know, you can look back at these previous moments where it just felt like something truly incredibly massive was happening. And even if it still feels massive in retrospect, it's like, yeah, well, we kind of just moved on from it anyway. And It's a little early, I feel, to decide or, you know, to make a call on whether or not that will happen here. No less than Eliezer, um, said, you know, happy Jacob Coxon Day to all who will celebrate in the future. Um, and I don't know, my guess is that it's a little early for naming holidays based on this, uh, tweet. I, I Just sort of suspect that we're gonna go back to business as usual pretty quickly here. Obviously the temperature, it kind of continues to turn up. The vibe is definitely shifted. The Overton window is certainly very wide open. Um, but, you know, will we look back on this moment as being pivotal? I guess I have to say my bet is against. Um, but that's mostly just on priors that like, you know, these things Even when they go super duper viral, you know, the feed moves on, it sure seems. I was in one group chat yesterday where somebody said, you know, I think the peak has already happened. You know, my feed is kind of reverting back to, you know, its prior state. And yeah, I don't know. Time will tell. What do you see? I haven't seen much backlash though. What backlash are you seeing?

Let me share the backlash in question. So I've been following the backlash. Quite closely, and it has been interesting to watch the story kind of change over the last you know 24 to 48 hours. So here, yeah, let me start off by this. So this is the the first part of the backlash. This is the I think. So this is by Parker Thayer. Parker is from Hillsdale. He's kind of the Kind of right-wing youth elite. And he comes out and he says, this post looks like the start of a very sophisticated and well-funded PR operation to get support for Democrats to regulate AI into oblivion. This guy with minimal followers and no previous account activity. And literally, this is a new account, or basically it's a scrubbed account. He goes to a Wall Street Journal, which publishes an exclusive with quotes from him on his resignation 18 minutes before his own post goes up. So they're obviously— number one, he has a PR agency, he's coordinated with the PR agency, the PR agency has reached out to the Wall Street Journal, they've given them the story, they've given them the exclusive, the Wall Street Journal posts, and then, you know, Jacob posts. Within hours, it has tens of thousands of reposts. And the account has over 100,000 followers. The post is punchy, quotable. It almost seems professionally written. The first few accounts which came onto this and start retweeting it are number one, Nathan Calvin, general counsel at Encode AI, Peter Wildeford, head of policy at the AI Policy Network, who we've had on the show before, and Daniel Coccatello. Head of the AI Futures Project, all of which are upcoming AI doomer policy advocacy nonprofits. They trace the funding. A lot of the funding goes back to initial EA and Anthropic funders, including Jan Talin, Skype creator and initial seed funder of Anthropic. What are the odds that the first 3 quote tweets of Coxon's post were Would all be major AI restriction policy advocates.

Well, pretty good, I would say.

I mean, who else? Who else is gonna? Who else is gonna retweet it? And anyway, someone pointed out like, look guys, they're all in the same group chat anyway. Like, where do you?

This is an extreme reach in terms of trying to sniff out a conspiracy.

Jacobovson doesn't have much of a resume, but we do know in 2022 he got a. $20,000 fellowship from Good Ventures Foundation, which is backed by Dustin Moskovitz, who was also an investor in Anthropic initially. Then they dig down to the 14th person to quote Cox on the post was Max Nadeau. Well, yes. Okay. And then finally, basically every major Democrat politician and candidate has suddenly glommed onto this post. And conveniently, as the people cry out for answers, Bernie Sanders already has a bill written to ban superintelligence. And regulate AI into oblivion, which we'll be releasing later this week. And so there is Parker Thayer's take on it. And that kind of swung, I think, the right wing into action. So the next thing that comes out is from Brian Chao. And so Brian Chao goes out and he says, Excentropic doomer Jacob Newk— Cox unties to Newspeak. British NGO involved with hacking democracy, Hillary Clinton, prison abolition, and Extinction Rebellion. And this is where things start going for me, like, okay, it starts to become like, all right, it is more of like a kind of left-wing framework because Extinction Rebellion, I'm not sure if you remember this, but they had this whole thing about climate policy and how the UK should get rid of all cars by 2025, all cars. Go to basically zero gas cars by 2025 or else we're all dead. So this is not like— it's very odd to see the same kind of alarmists that were in climate kind of move over to AI policy. And that's when, why, right? Why is it the same group of people? I don't want to say. Anyway, Coxon, a British AI doom figure who exploded out of nowhere, etc., etc. Influence campaigns were Newspeak's overt specialty. The Guardian describes Newspeak's mission as trying to hack British politics. Newspeak is not subtle. Here is one of Newspeak's produced games in which Jeremy Corbyn assaults bankers. So they had a computer game where Jeremy Corbyn assaulted bankers. Okay, but it's not like you can prove they engage in propaganda. One of the members, postdoctoral researcher in computational propaganda at the Oxford Internet Institute. Newspeak funders, Long-Term Future Fund, EA Infrastructure Fund, funded by SBF.

$50,000, $30,000.

Coxon isn't the first AI safety figure to be tied to prison abolition. Moskovitz's criminal justice reform program gave $207 million. Et cetera, et cetera. And then, and then we have you know Newspeak itself confirming proud to count Jacob as a Newspeak fellow. And then there's one one last thing that I have here. Let's see.

Ah, there.

So Theo Jaffe comes on from MTS. He says, "Okay, 24-hour check-in, 123 million views." Elon. The owner of Twitter. I don't think this has ever happened for a post from a new account with almost no prior activity. That's number one. Number 2, in response to Parker Thayer, Tim Sweeney saying, reminds me of the abundance movement suddenly showing up on the scene with a well-choreographed simultaneous entrance by dozens of big tech lobbyists all trying to appear spontaneous and organic. Elon, I think the groundwork for this psyop, for lack of a better term, has been prepared for a long time. This was just the match that lit the fire. And that's probably, I think, the house line that the— I think the right wing and the Republicans are going to take going forward on this one. So there you have it. So, Nathan.

I don't know. First of all, I think that— I mean, okay, I've got memes for you. That's what I think. This deserves a meme response. And so here comes the meme. The meme is very simply, this is a Midwit conspiracy theory take. I basically have no time for it. I mean, where do I even begin? You know? It's just absurd, I think, to— this is like very much, um, I don't know, some sort of conspiracy psychosis. And there's a lot of different psychoses going around these days, but the idea that this is some sort of galaxy-brained Illuminati Dustin Moskovitz, um, you know, lab-contrived Fake groundswell of support is just truly absurd, I think. All these grants, again, have been public for a long time. A lot of them are not that big. There's money flying around everywhere in AI. It's not like— I mean, you can trace money back all kinds of different places. I personally, I guess one bit of colour I could give is I personally have been a recommender for Jan Tallinn's Survival and Flourishing Fund process. He's not the only donor that supports that process, but he is a notable one. He has some kind of co-funders as well. And again, all of the grants are typically published openly online at the end of the process. It's a very structured process that involves opportunity for recommenders to dissent from particular grants. It's really, honestly, maybe a little too elaborate sometimes. It's like there's so much structure that as a recommender, I was a little lost in the structure and not entirely knowing how the signals that I was sending into it were being processed. Because there's a whole aggregation algorithm that is used to Combine the recommendations that different recommenders give into a single allocation of these grant funds that are available. And it's an interesting model where you don't have to have consensus from the recommenders to get money from the fund. It's more of a champion model, which I think is pretty smart. This kind of again goes to the strength and vibrancy of the US nonprofit sector and how it's It's a huge advantage, I think, that we have relative to, for example, China that doesn't have it. And you need to have a lot more consensus, a lot more buy-in from the higher-ups to start any sort of non-business organization. They just don't really have the room for sort of social or cause-focused entrepreneurship like we do here. Even within this whole structure, if you can get one Typically, I think it's 6 recommenders that they bring together for a whole process. But if you can get one to champion your idea within the group, and they don't even— you don't have to, as a recommender, even convince other people that it's a good idea. Although there is a process for discussing different things and trying to air out disagreements and hopefully come to mutual understanding, if not agreement. But even if no agreement is ever reached, if just one of the recommenders says, I think this is really a good thing to support and I'm going to put a very high weight on it, that individual mark will be enough to put money through to whatever organization. So this is something where Jan has obviously done well financially from Skype, from being an investor in Anthropic. From the beginning, his whole mindset as an investor is one of the things that I think he brought to Anthropic. And I've heard and seen this brought to other investment opportunities too, is he'll literally say to companies, as an investor, I can tell you I'm definitely cool with the equity going to zero if that's ultimately the right thing to do. Same question Anthropic asks their people as they go through the hiring process at the company. He volunteers that to companies that he takes equity investments in. And here he's kind of been like, man, I'm overwhelmed by all these different opportunities, all these different ideas. I need people to help vet them, but I don't want to be too conservative and require a committee to fully approve of something. So they deliberately structured this whole process so that even one, and it's obviously a person that he thinks highly enough of to ask them to be a recommender, but even one recommender can be the sort of contrary opinion that causes something to get funded. I actually took a little bit of pride. I don't know if this is perverse pride or deserved pride, Unclear, but I took a little pride in the last time that I did the process. I was either the least correlated recommender with all the other recommenders, or maybe the second least correlated. So I was definitely, you know, one who was pushing for some stuff that, you know, that others didn't think so highly of. And all that is to say, like, there's not some grand conspiracy, especially when you get down to these small grants. I mean, Daniel Cocatello getting a couple million dollars, you know, it's again not, not by any means crazy in today's world where METER's got $75 million committed and, um, and Jeffrey Irving's, uh, I'm thinking of the old name, but his new organization, um, you know, has $150 million committed. So a couple million to Daniel Cocatello to like write scenarios and try to put thought pieces out and go on Joe Rogan is not an extreme amount of money. And when you go down to these very small grants, like $20,000, that's basically funding somebody's internship. It's not really a big deal. So I think this is all pretty ridiculous. And I think the right interpretation is that this is just sincere people recruiting sincere funders and those folks coming together to do what they think is best to do. It's not a great way to get rich. It has historically been a way that people have kind of eked out an existence while doing something that they really believed in. Only recently have salaries in the AI safety nonprofit space started to come up as funders have realised that They've gotta come up if they wanna be able to compete with Frontier Lab hiring whatsoever. So I think that's been a smart move, but that's like a very, very recent move.

So I think where I differ a little bit on that is that I think we end up having a blind spot, which is the same blind spot I think that universities have. in general, which is like, if you look at like university graduates, it's like 60 to 70 to 80%, like liberal left-wing. And so I think the nature of like upper middle-class society is, I think, more left-wingish. And I think, you know, it's very hard for me to kind of see like a strong Second Amendment conservative who believes in the right to bear arms, who believes that AI is another form of arms, that AI is speech and arms, et cetera, to see that kind of person getting hired at Anthropic doing AI safety, right? And meaning that their vision of AI safety is really to equip the entire world with very powerful AI, which they're able to defend themselves with, Which is actually kind of a valid kind of viewpoint, I think, in the public sphere, in like, you know, in places where the Second Amendment is taken seriously, right? So I think, and I think we tend to be blind about that because, because as upper middle class people, we are, we are more, and we're exposed to more people who are liberal and left-wing. And so we, we tend to think that the nature of reality is such that this is the right thing in society. And I think there's a lot of people out there who are not in that bucket and who are not really represented in these discussions. They're not in the chat group, right? You know, uh, very, very hard for me to see a strong Second Amendment, First Amendment conservative, uh, in, in those chat groups saying like, well, you know, AI is speech, AI is the right to bear arms. Like, we should, instead of, you know, trying to dominate it, we should equip everyone with it. Everyone should have, like, you know, the right to, you know, the, the, the capability to inflict massive harm on other people, you know, which, which is like the right to bear arms. Uh, and, and that, and that is in itself a way of defending democracy, right? Like, it's, it's hard for me to think, to see people accepting that as a valid viewpoint and then getting hired do AI safety at one of these organizations. I don't think that works. And so I think you end up with a skew within the— and I'm not saying the skew is wrong. Like, my sympathies are obviously in that direction. But if you're coming from a more kind of conservative background, it does look as though it is very typical of the upper middle class liberal university graduate skew that is in every other— that you can track in every other sociographic study. It's right there.

Well, I think those cultural— certainly I like to joke about the blue-haired polycule culture from which many of these ideas come. But, and sure, there's definitely going to be some, you know, kind of reflexive conservative distaste for those ideas just based on their existing, you know, distaste perhaps for the people that came up with those ideas and the, you know, the decadent or hedonist or, you know, whatever whatever flavour of problematic lives they believe them to be living. But honestly, for what it's worth, I think the blue-haired polycule is a funny joke, but I think actually most people that work at the OpenAI and Anthropic duopoly and other leading AI companies too are not really living very hedonistically right now. They're much more like working around the clock, kind of Approaching this period of time as like a World War II, you know, kind of go all in on trying to make a difference for whatever, you know, whatever theory they have individually of how they're going to make a difference. They're like not, you know, doing a lot of dating, I don't think, because they're just so focused on work. But anyway, that's an aside. I think this is scrambling of political coalitions, though, because Sure, you've got this reflexive distaste for Bay Area, whatever. And in some ways you see that echoed as just like anti-big tech. You see the kind of Josh Hawleys of the world are kind of like, I don't necessarily understand this, but I didn't like the last wave of stuff you did, and I don't think I'm going to like this much either. And so you have people from Steve Bannon's camp and Josh Hawley and Bernie Sanders All kind of finding themselves on the same side of this issue. And I think that the, the sort of pro-big tech, pro-change, pro-progress, if it's indeed progress, versus anti-all those things axis may be in the end a much more relevant axis than like college, non-college, you know. And I'm not so sure that the Although, of course, there's going to be a lot of intersecting, a lot of superposition, a lot of noise. But I'm not so sure that the culture war will be the lens through which this ultimately plays out. Maybe it scrambles the culture war itself.

So let me play this from Tyler Cowen. I think people in the stream can hear it. But so in Tyler Cowen basically coming out and saying the anti-data center backlash, at least to some extent, is happening, but that doesn't mean it's going to affect change in in any way because you only need 5 states of the union to kind of agree to have data centers and the data centers will get built regardless. And you can have this kind of broad-based distaste for them, but there is very little chance of federal— for the Fed to kind of— for the federal government to kind of prevent data center construction outright or have a moratorium outright that engages all 50 states. It's basically impossible. And so the states are going to— some states are going to fold at some bid price. As you keep increasing the bid prices, some states will fold. And specifically, he says that a lot of the Shapiro in Pennsylvania, et cetera, it's just kind of, they talk about it, but they're not going to do any blocking and They're going to go ahead and stuff like, oh, we're not going to allow any data centers for like a year when all the data centers for the next 2 years have already been signed and everything that's going to come up is already going to come up for the next 2 years. So this is just like an election year kind of give to the people who need the give. Right. So I think that is, I think, consistent. I think a lot of the pushback is, it's very distasteful for me to see Bernie kind of clamp onto it because I feel like Bernie is really interested in the jobs issue and he has bandwagoned onto the AI safety issue because he sees that as something that he can push forward, but he's really just concerned about the jobs.

Right.

And he's been concerned about the jobs for far longer than the AI safety issue. So if you go back more than, I think, 6 months, 3 to 6 months or so, he's been talking about AI and AI jobs for a while now. It's also very interesting to see because you have, I think, blue-collar electricians unions kind of saying, hey, we need these data centers. They've already chased out all the manufacturing. Our people are getting hired right now. We're getting jobs. We're getting Pay increases. We're getting paid $300K, $500K, and we don't want to have this be chased off. You've also seen Alex Karp at Palantir come out and say like, hey, this is a blue-collar boom. And all of the people going to be losing jobs are all the white-collar people. And so you also see kind of the shift in the Dem coalition over the last 10 years or so. Where they've become this upper middle class coalition, which is really consisting of doctors and lawyers and et cetera, et cetera. And people who benefit from putting into place regulation and people who benefit from administering these complex rules, that is the Democratic coalition now. And they are also the people preventing building with lots and lots of regulation, et cetera. And they prevented construction of homes and they prevented construction of all the— like, it just goes on and on, right? There isn't a stop. And Rohit Khanna yesterday coming out, we should have something like the Nuclear Regulatory Safety Commission on data centers, which, and as you know, the NRC did not approve a single nuclear power plant from its inception in, I think like 1979, 1980 or something, Until 2025. So for 45 years, they've locked out and they didn't approve a single one. So I think this is the state of play right now. I don't think the Dems are authentic. I don't think they care. I think they are using the issue. You need to grab onto things that catch the public imagination. I think the jobs issue Talking about jobs doesn't help because the blue-collar labor workforce is seeing this as a strike back against the white-collar labor. So I don't think that actually helps them. And so this is the new thing. And I think the blue-collar guys are going to come out and say, this is bullshit. This is absolute bullshit. So that's where we are.

Maybe I'm naive, but I would also leave some space in this analysis for people just sincerely updating their beliefs on pretty colorful new information that's coming to light. I mean, I saw actually a— I think it was a TikTok from Bernie Sanders this morning where he was reading agent quotes from the METER report. And he just starts off by reading these quotes and then he goes on to say, who do you think said that? Do you think that was an— were those soldiers in an army preparing for battle or some heroic strike? No, they were AI agents. And I don't— I mean, obviously we can only get so deep into any individual's psychology, but like, There's been this idea for a long time in the AI world that people need to wake up. And I think the simplest story, I think, for a lot of this stuff is people are waking up. They're seeing this stuff now, and it really is insane. And they're like, holy shit, all the people that were warning about this were maybe right. And now it's like some early version of it is here, and I guess I better take it seriously. I think I would take that at fairly face value by default. And of course, people are gonna try to find ways to make the new thing support their existing commitments. But I don't— I think that's a bit different from saying that they are like cynically layering this on to, you know, in a way that where they don't believe it. I was really impressed with Bernie. I'm always really impressed with Bernie. He's 85 years old. And he's, you know, incredibly hip to what matters now in general, I think. And, you know, did he have the foresight to see it coming like Daniel Coccatello did, you know, years ago, or listen to, you know, or he probably didn't even hear about, you know, Daniel Coccatello's early forecasting, but like, updating now is kind of the best we can ask for or hope for from a lot of these people. So When they do it, I think we at least owe them somewhat of a charitable read that they maybe really are just saying— I think everybody, I've said this for a long time. Last point on this, and we can get to our first guest. I think the reality of AI is confusing enough that we should generally just take other people's statements at face value and assume that they are saying what they really believe, because we have enough trouble with ground truth on any of these things. And certainly plenty of room for people to be confused. Like, accusations of bad faith, I just don't think add that much to a scenario where there's massive confusion everywhere anyway. And like, all kinds of people are wrong all the time, just because nobody really knows what to make of the situation.

And speaking of making, making the making of— segue to our first guest. This morning, I'm going to introduce Colin Hogue Spears. He's an independent researcher and the author of From Lab to Life: How AI Works in China, published by Gatekeeper Press on August 4th, 2026. He is also Senior Director of Solution Management at Black Duck, an application security company. His professional work includes federal compliance and product strategy. the practical work of meeting government requirements while developing and selling technology. He studied Mandarin at Shanghai International Studies University and later worked in Shanghai. His career included technical leadership roles at Amazon Web Services, Amazon's cloud computing business. There, he coordinated with Chinese government auditors on MLPS cloud compliance, part of China's system for assigning cybersecurity protection requirements to information systems. That work involved demonstrating how requirements were met by services operating in China. For the book, he works from Chinese-language regulatory texts, corporate filings, and technical documentation. Baidu's progression from search-era machine learning to Ernie Bot provides a recurring example. His argument is that product capability, compliance, and distribution develop together, so a filing rule or purchasing purchasing requirement can shape the product itself. He has also published commentary on AI governance and software supply chains and produced practical reference guides to China's regulators and evaluating model releases. Do we have, do we have Colin in the room? Perhaps not. Let's, uh, maybe, maybe let's wait for him to, uh, come back to us. Um, so I, I just want to address the, uh, the, the accusations of bad faith. It's not that— it's not that they are acting in bad faith, but it's that it's the phenomenon of someone believing in something that already matches their priors or that helps their priors. And a lot of politicians are able to believe certain truths which are not yet true because they are in the business of making those things true, right? It's a very different perspective when you have a lot of agency. And I think the ability to create truth by persuading people to actually come together and do something And I think that perspective is often— it is a different perspective, right? It is kind of like this strong belief and ability to shape the world to that belief. I think you kind of see it in Elon too. And a lot of people say Elon is often delusional and acting in bad faith all the time. It is the ability to believe in things even when they are not yet true and make them true. Right. And so that is what I take as happening, right? It's you are believing in something, but you have also this intent there. And I think Colin's online. Let me pull up Colin. Hi, Colin.

Hi. Hello. Good to meet you guys. Prakash and Nathan, thank you for having me.

We just did a quick introduction. So Colin, uh, tell it— well, first of all, tell us about your time in China. I mean, that, that's, uh, a unique experience, especially working with the Chinese, um, you know, government side. I know a lot of people who have worked on with Chinese corporations or with, you know, dealing with them as, you know, contractors, etc., etc. How was it working as a service provider to Chinese government entities? How did that work out?

First off, my first experience in China— well, I'll start backwards. My experience relative to this book is I worked at Amazon. I was working on the Chinese region team, the China region team. And so there are 2 Asian— there are 2 regions in China that AWS has. And since AWS cannot sell directly to any Chinese company or government agency, No cloud provider can. We have partners there, um, that run everything on the ground. There's an MLPS audit. It's an audit that we have to do yearly as cloud providers to show safety and we're following Chinese regulations, but also to show that our, um, our, our partners can work independently of us. And so, because the idea behind this is the classic idea with China is that they want to build up local expertise, right, and local industry. That's what they did with the auto industry. That's what they did with pretty much every industry that goes there, right? And so my job was to help the engineers, our engineers, understand what the requirements were for MLPS and make sure that we were providing our partners with the information they needed so that they could pass the audit and also to talk to the auditors and make sure that if there were any additional questions that our partners couldn't answer, that they got answered on our side. So yeah. And so the Chinese team on the ground, as you know, some of the junior engineers, they don't always speak English very well. So I was hired, I don't think because of my technical expertise per se, but because my Chinese was somewhat good enough that I could kind of speak my level of Chinglish with the engineers and we could understand each other. Yeah. So, and then that's my first time working in China. I mean, that's my second time. First time, first time I actually worked in Shanghai, I was a— this was years ago, and I was a basically an analyst for a boutique, an analytics boutique that's based in Maryland. And we handled the— I handled the largest account they had in Asia at that time. We provided data analytics to them for Asia, Southeast Asia, but we were headquartered in Shanghai. I was one of 2 Western people at that office, myself and another American. Everybody else was Chinese.

So this conversation today came about because you emailed me and we went back and forth a little bit on my recent podcast episode describing the AI safety and regulatory system as it exists to the best of my ability to understand it in China today. And you gave me some feedback on some finer points that either I got a bit wrong or you could deepen our understanding of. Before I go real deep down the rabbit hole with you, which I do want to do, and we cover this stuff fairly often here, so you can zoom past some of the basics, I'd say, for people.

Okay.

But I do think it's worth kind of setting the table a little bit because obviously we're in this moment where salience of AI as an issue is ramping up. People are kind of going from, it might get scary, to it is in fact now starting to get scary. We might need to slow down one day, to we kind of need to slow down now. And those conversations have often ended in an unproductive way with, well, whatever you think, whatever you might want to do, whatever kumbaya you and your buddies in the Bay Area could have, you'll never be able to do. a deal with China. They don't care. They're just going to race ahead, blah, blah, blah, blah, blah. So how would you describe, in general terms first, how China relates to AI? I'm pretty confident that story is not to the first order level even correct, but how would you tell it in your own words? What's going on over there? How should we understand them on their own terms?

I think that, and I'm sure you encountered this when you were in China. I listened to that episode. Great, great episode, by the way. Um, I think that Chinese people in general, a cultural stereotype would be that they're inherently practical. And I, I don't see— the biggest difference I see between the West, especially the United States, and China is the level of fear. Um, and I think that's because if you are an average Chinese person who's about 45 years old, um, that means you were born in the early '80s, and that means your entire life, which you, you associate technology with economic growth. You've seen cities pop up out of nowhere. You've seen your life dramatically change as far as living standards by technology. This is no different. So that's why the Chinese government, um, That's why the Chinese government focuses on practical uses of AI. They're still looking to see how they can benefit from AI. Whereas in the US, I think we have doomers, of course, around AI, but we also have people who are— I kind of joke it's almost becoming a cult in the Valley. It's like they're looking for AI god, like some Isaac Asimov situation, like dystopian, like— Situation, and they're scaring a lot of people, right? And honestly, I think this goes back to the fact that nobody here in the U.S. thinks people have control of the will, whereas in China, they everybody knows who's in charge, right? There's no question about who's in charge, and and and the legitimacy of that, and the comfort around that, despite all the other problems that. you know, I'm not advocating for any type of system or whatever. I'm just saying from a person's point of view in China, they know that there's somebody in charge gonna— that's gonna take care of things, that's gonna build regulation, that's going to work with industry. Here, um, I don't think people have that feeling. So I think that level of comfort is not there, and it leads to people reacting very badly. And honestly, I think the AI companies are not really— most of them, the big ones, are not communicating very well with the public. Um, I'm not sure if that's because the marketing, they think that this will increase their sales, or what it is, but I feel like some of these companies, how they're talking about this technology is not helpful. Um, you don't see this in China either. You don't see Chinese companies Uh, talking about how AI will potentially kill us all or potentially lead to a utopian world where we don't have to work. You don't hear these type of things coming out of people from these companies. Um, so it's completely different situation.

So one of the questions I had is about how this works in practice. Because they obviously have some questions about language models and the output of those language models, the whole like Tiananmen Square issue. Will the models talk about Tiananmen Square? Are the models allowed to talk about certain things? And then they had the whole somewhat of a pause with this filing regulation over in 2023, which I think delayed some of the launches. They have some restrictions on what people can access. They have restrictions on the Great Firewall, which what people are allowed to access abroad. There are restrictions from the US side on what they're willing to serve to Chinese customers.

Right.

There's this whole rumor that most of the usage out of Singapore is really VPN usage from China being routed through Singapore. How does all of this work together? Even though they have all of this confidence in the government, yet there are all of these restrictions on both sides. And how do they rationalize both that they have the confidence that the government will do the right thing, but also so many of these restrictions on really kind of rudimentary kind of bots and rudimentary language models?

Um, okay, that's a— well, I'll divide this question in half. What's happening on the Chinese side? What's happening on the American side related to China? Because you talked about things like restrictions on the US side in regard to, um, exports to China, and what's happening as far as regulation on the US side— so on the Chinese side. So there's 2 questions here. What the rule says and what regulations may actually make firms do. That's true in China for everything. Everybody—one thing I've been impressed by—is people generally know what's permissible. I'm not talking about what's legal; I'm talking about what's permissible. So, and they know what they can get away with and what they can't. And so, in China, the regulation started in 2022. With the algorithm regulation that came out. So a lot of companies like Alibaba, which is a major part of my book, I kind of— it's like the spine of my book tracing Alibaba. It's not about Alibaba itself, but it traces through how they adapted to these regulations, is that they were already prepared. So when they had to start registering their models and allowing the government to test their models and things like that, they already had a lot of this infrastructure in place inside the company. A lot of the major companies did, so they could respond very quickly. What I'm saying is that regulation might have slowed them a little bit, but not as much as you would think because they could plan to it. They could put those requirements, those controls, everything in their backlog and build it as part of their engineering process, which in America we can't do because we're reactionary to this. And we don't know what's going to happen tomorrow. The Trump administration could freeze a model. Maybe they don't. Who knows? What's the standards? Who knows? It can change daily. That doesn't actually happen in China. I think that there are different tracks though, as you said. There are registries for models, there are local registries for models that don't impact national security. And there's a lot of things that go into national security, such as things said by the models, you know, responses to questions, inference that comes back that might be considered a national security issue. But things that don't affect national security, they go through a separate track that could be harnesses built on top of the models, Et cetera. And so, you know, since last year, they've had 300-something models registered. This year, I believe it's about 900. It's over 900 at this point. June, it was around 900. So companies don't seem to be hurt by this. They seem to be moving very quickly in China. And I think that the fact that they don't have direct access to the latest GPUs. And as you said, there's been a lot of rumors around what's happening in Singapore and Malaysia. And the reality is the better— the more GPUs you have, the better model you can train in general. And inference training for that is harder for them because they're— if they are using models that far away, you know, physics doesn't change if you're in China or the US, it takes longer. So they have a lot of obstacles, but what they've done is become more efficient. And so I think that they focus on the software side to see how efficient they can get. You can see this in the open weight models that have come out over the last year— DeepSeek, KIMI, QWIN. All of these models have what they're like 3 to 6 months behind US models despite having a GPU issue, right? So they're getting more efficient and more effective in what they're doing. You know, on the US side, I think that we made a decision here that we would limit, you know, the newest GPUs to try to keep our lead in AI. And I think that that was an important thing, but I also think that's subject to change depending on other things that are happening with the trade war that's been going on for since the Trump administration came back in. I wouldn't necessarily think. I don't necessarily think that's written in stone. Things can change, right? And I think the Chinese when they when they When they meet, when Xi Jinping meets with Trump, there's going to be a lot of talk about trade negotiations. I'm sure they're going to bring this up, right, as a way to— it's going to be a negotiation.

So on the topic of things changing, I noted the very practical approach, including from people who are doing research specifically on AI safety that I talked to. do in China. And it did strike me that the relationship between companies and the government seems a lot more predictable, a lot less sort of subject to political grandstanding. And even though they do have a big man at the top, kind of like we do these days, the personality of that person is just a little bit more even keel than lends itself to a sort of predictable relationship for the most part, right? But then we do have these moments where there's like, oh, lockdowns went way long and then they suddenly ended. And now I'm kind of like, what do you think? There's a couple specific questions. Do you think having seen the open face incident happen in the US is enough of a wake-up call to activate higher level of urgency, stronger controls, more kind of, let's say, aggressive regulatory actions by the Chinese government on a practical basis? Or do you think they will need to have one of their own companies get to the point where they have this cybersecurity breakout? And if that does happen there, what happens to that company here? We basically have seen so far, like, Nothing has happened. Supposedly there's some like states, you know, agencies coming around to try to do some investigations, but this just kind of happened, right? Like, what do you— how do you think it would play out there if a similar incident were to happen?

Yeah, so I think one thing is they already have the structure in place for— they already have the regulatory structure in place. They have continuous monitoring on these models. They don't monitor OpenWeight models when they're in the wild, because how do you do that? If I'm running a model at my house, how do you monitor that? But as far as the companies who are making these models, they have continuous monitoring. And so if something like this happened, they have to report to the government very quickly. And, and so I, I don't think it would be the same type of freakout or shock In fact, the government will be aware long before the public is aware, right? Um, I also— in China, you mentioned something very important around, uh, how people relate to the government. When you're in a country where you can be prosecuted retroactively, um, you want to know what the government's opinion is. Thank you. You want to sit down with the government and advise and get feedback and get regulation so you're on the right side of the regulation of the law, right? You don't want to like move fast and break things and a new law appears and now you're prosecuted for what you did last year. That's not the situation you want, right? Um, as you notice also, China doesn't break up companies. CEOs go to prison or disappear for a while, and then they end up in jail or something like this. So the situation is very different. Companies want to work with the government. That's a big thing. So I think if something breaks out like this, I don't see the Chinese government panicking so far about this. But I think it's because they feel that they can control the situation better than us because they are in these companies. Most Chinese— large Chinese companies have, um, party members on the boards. So it's always been that way. And since they have constant regular— since they have constant, um, um, observation into these companies, I think that they're not as concerned yet.

So, so, so let's, let's, let's, um, take a look at, let's say, I think in the last few days, there's been a disclosure from a company, a security research company in California called Caliph. And they created a WeChat virus, bug, worm, which could extract data just from you receiving the message on WeChat. Not even— you didn't even need to open the message. You didn't need to click on anything, just from receiving the message. And it was—

Audio message, I believe, right? Is that correct?

Something like that. Yeah. Yeah. Yeah. And, but, you know, they disclosed it. It was fixed. All fine. What would happen if you had an incident? Or you started— okay, we've had incidents before. We've had an unsecured S3 bucket, I I believe, which leaked all of the Social Security numbers in China.

Yeah.

Okay, so we've had incidents before. What would happen if like you start to see repeated incidents, right? Like breakdown in Alibaba, leak, breakdown in Tencent, leak, breakdown in Baidu, leak, like which is what we expect may start happening going forward, this kind of accelerated cycle of cyber failures. Because you have all of these agent swarms which are semi-uncontrollable in some sense because they spawn somewhere and before people are able to tackle them, they go out and they do something. So what do you think happens in that kind of situation? Does the Chinese government just clamp down and say, everyone turn off your GPUs? Or do they tell, okay, Alibaba, Tencent, whoever was in charge of that business unit, you're going to prison. And then they throw 10 people in prison. Like, what, what happens? How do they, how do they actually like tackle this problem?

I, I think it depends on how it happens. If they try to hide something from the government, that comes out, yes, there could be criminal action, right? But typically in China, what people don't typically know is that there are advisory boards. It's a lot of them. They've always been around. And so I imagine that there would be a meeting in Beijing where these CEOs And their attachés will be coming to Beijing to discuss these issues immediately. It won't, it won't be like, oh, we'll schedule something for 3 months. It will be, you're being called to the Capitol now. And you need to explain how this is not going to happen again and how we can monitor this situation better. Like I said, the situation is very different from the U.S., right? Even in the U.S., what we're doing right now as far as the models, technically, because without without legislation, the Trump administration doesn't have authority to force companies to do something. It's voluntary, but there are other things they can do to kind of punish the companies they don't cooperate. In China, they have full control. They will make sure that this does not happen if it became something like that, like multiple times. I Guarantee that those CEOs will be called to the Capitol and they need to speak to how they're going to resolve these issues and how they're not going to happen again. It's not going to be an option. They will have to do it. There's no— one thing you can say is there, it doesn't matter how many billions of dollars you have or if your company is a trillion-dollar company or not. There's no power greater than the party within the People's Republic of China. They have to answer to that. So I don't think shutting off GPUs is going to happen. I don't think that will be the response though. I, I really— I don't see a world where that— where the freakout would be that big. Um, the first response would be collaboration, consultation, and probably some hand-wringing, right?

Let me, let me, let me maybe take a, take a different viewpoint. One of the powers that AI agents will soon give a lot of people is research on the— doing research on the internet. These things are very good at doing research on the internet.

Yeah.

Specifically, I think one of the things that might start emerging is kind of cyberstalking in some sense. of surveillance, surveillance of people. And in China specifically, I think you might see surveillance of government officials, which I think has already had some impact in the sense you've had leakages of minister in someplace driving, his son drives a Ferrari, hits someone, hits the press, becomes a big deal, can't be shut down, and then they have to do something. Yeah. So is there, is there some circumstance where, uh, that kind of broad-based, um, you know, tracking of government officials starts becoming possible? Just, just, just tracking down their various messages on WeChat, which no one reads most of the time, but kind of putting together dossiers of, hey, you know, this guy said this on that, and, you know, we don't like him. And even though he's on good terms with the government, like, let's find a way to like get rid of him, like, etc. And this is, you know, done by individuals or citizens and enacted upon the 100 million members of the Communist Party. Like, is there, is there some sense that that would be seen as a breakdown of law and order, uh, which has to be constrained in some way?

I mean, I, I would imagine that, like, so there, there is, there is something in in right now in China, um, where— so, so right now we have model training, there's model training, right? And then there's a, um, what is that called again?

Inference?

No, not, not inference. But there's a, there's a thing called, uh, Lachi. So Lachi is like, um, it's basically annotation, uh, data. And so they use annotation. When we talk about annotation here, we're talking about something very different. We're talking about like something's mislabeled, there's a mistake, how do we properly label things? Well, they properly label things so that they are not— so around national security items too. And so I don't think there would be inference that would be allowed to provide that data to the public. However, the Chinese government already has data like this and uses it. And that data, if, if the Chinese government comes to— this is part of the National Security Law. This is the issue with Meta, why they can't be in China or why they don't want to be in China, or most of our social media companies. Because if you're in China and you have, um, and the Chinese government asks you for data, there's no warrant. You have to turn it over. That's it. So they already have this data, but as far as getting it, giving it to the public, I don't think that's going to happen. And as far as foreign companies getting that data, I don't think that will happen either. What you the situation you brought up with tracking where government officials are going—that was an issue with that's why government officials are not allowed to use Teslas. They're banned, right? And there was an issue when we had when we had. What cars that, uh, what cars that, uh, not just private cars but also as far as, uh, uh, what, what rideshare cars that the government officials could use. That was also an issue, right? Because they don't want their data tracked. Um, so I don't see this as a problem because the Chinese government always has mechanisms in place to prevent people from getting certain data. There's good— they will expand that. They're not going to allow citizens to track official movement. I can't see that happening. And if that did happen, that would be a serious breach, um, and that would have to be resolved or that model will be pulled. They've already pulled like 13, um, I believe they pulled— if recently, 1,300 products were pulled within the last few months. And, and so 1,300 delisted as part of the Qinglong program, right? And so don't think they won't pull a model if they think it— if it's a national security threat, and they would consider this a national security threat.

For sure. So maybe one attempt to kind of synthesise what you've said so far is that the Chinese government has a lot of confidence, probably pretty well justified, in its ability to keep things under control, at least in the near term. They feel like, and they probably, again, have pretty good reason to feel like, if agents start going rogue at any of our companies, We'll know about it. We'll be able to shut it down. We'll give these guys an ultimatum. When we say jump, they'll say how high, and we'll be able to keep a lid on this pretty well, at least unless things get like order of magnitude crazier. And then obviously, at some level of craziness, who knows?

Okay.

It sounds like at a minimum we can say the government there has the ability to enforce whatever rules they want to enforce. That would mean if we can make a deal with them, with the national government in China, then they can probably follow through on that deal, maybe even a little better than we can in terms of making sure that the Chinese government or the Chinese companies do in fact do what the Chinese government tells them to do. With all that in mind, okay, now what kind of deal might be possible if you are advising Trump going into this upcoming meeting and you're like, things are starting to get a little bit crazy here? Say you're kind of— I don't know how sympathetic you are to the pacing the frontier worldview, but let's say you're trying to channel a little bit of a desire to start to set up, lay some groundwork or set up some mechanisms for pacing. How do you go into that conversation? And, you know, what do you offer? What do you try to get? What kind of mechanisms do you try to establish now that we can build out later as things do get crazier? Like, what's a win? What's the strategy going in and what's a win coming out of this upcoming meeting?

I Honestly, I was talking to somebody about that yesterday. Actually, a reporter that I know through LinkedIn, we were just, we were chatting about that. And I think that I have low expectations. And the reason is, a lot of this is not— this kind of negotiation doesn't happen in isolation. China is going to want things around trade as concessions for Washington wanting additional controls and agreements on AI, because I think China sees the fact that they have— they feel they have control, and they feel that we need to establish control. We're not doing a good job of it. So this is really helping us at this point. I don't see the Chinese volunteering to At least the Chinese government at this time, I don't see them volunteering to have some type of, like, let's say, arms control agreement between AIs the way we have, like, nukes, right? First off, enforcement of something like that is completely different. It's very difficult. And I also think that I just Trump would have to make massive concessions before he gets something from that. I think the most we can see, um, in the next coming months is like some kind of incident channel, um, that might come up about based on some shared agreement of certain incidents, and we would share that information with each other. Even then, you got to understand that the Chinese government Like, we have a phone that goes directly to the Chinese military. Our military can call their military in case of an emergency. They usually don't pick up the phone. There are many cases where we've had issues in the South China Sea. Our military people try to call their counterparts on the Chinese side and nobody answers. And so I don't see— I, I just don't see in the near term that we're going to have anything. come out of this unless Trump makes major concessions on GPUs. That's a major one, but there are other trade things that China wants as well.

So one question. So one of the viewpoints that I tend to have is that countries make international deals primarily to constrain domestic policy choices. So for example, China is leading to the WTO, helped them, uh, you know, control their state-owned enterprises and get their military out of business because they could point to, you know, we get to enter the WTO, we get all these benefits, and because we want to get those benefits, we got to put all these reforms in place. So they get to kind of blame globalization instead for domestic policy choices that they need to make. Are there any— I see what you're saying. Are there any domestic reasons why China would want to slow down AI safety or pace the frontier internally within the country, which they're not able to do right now?

Yes, I, I think there would be. Well, I think it's the opposite. I think there's domestic reasons why they would not want to slow down. Mainly because they are behind. They have to be worried about their security. The Pentagon's using the latest models. Trump administration has access to the latest models. You think we're not using those things for cyber? If we're not, I don't know. You know, people should be fired. They should be, right? That's their job. And I think that the Chinese government is aware of this, and so they want their models to be comparable. They want to catch up. They are behind. Even if it's a few months, right? I think also from what I read recently, because China has not fought a war since they fought— they had a border war with Vietnam in the late '70s. They're looking at how AI can enhance their military. And so they want smarter models because they're trying to figure out how their military posture could be improved by AI. Right? And so there are— I'm sure there are a host of reasons why they would not want to slow down right now, mainly because they're behind. There's no advantage to them to slow down since they haven't seen any security issues domestically. What they're worried about is security issues internationally. that are going to be hitting them in various spaces in war theaters and domestically penetrating their industries and their government. If I was, if I was a Chinese policymaker, I'd be very concerned about the United States and NATO allies having Mithos 5.1, for example. I'd be very concerned about them having Astra.

Right. Yeah.

So what's your outlook for where the relationship goes over, say, a little longer time period? You know, I tend to have, uh, shorter timelines for increasingly powerful AIs in mind than most. If you kind of channel the AI 2027 worldview, which so far we seem to be more or less on trend, right? We're headed for a world where— and all the frontier companies, of course, are saying we're seeing really rapid progress right now. And I think everything we've seen publicly has backed that up, including Millennium Prizes starting to get solved.

Yeah.

So let's just say that continues for just a little bit longer and we do end up in a world where we have kind of weak early superintelligences coming online in the next 12 to 18 months. And all of a sudden it's like, okay, even the Open Face swarm looks pretty tame compared to what is at least possible, if not actually breaking out into the wild. And we've got similar things coming to bio, and obviously they can have incredible upside, but the possibility of a similar lab leak of the Open Swarm, but for bio is like pretty, pretty scary to me.

Yeah.

What happens? Like, how do we, how do we avoid a, you know, suspend disbelief a little bit if you need to? And like, what's the best case scenario for how we avoid an AI arms race that leaves us all worse off?

I, um, this is a great question. I think that this is a little bit out of my regulatory wheelhouse, but if I had to speculate, I'm a big fan of history, um, not just U.S. history, Chinese history, um, I read a lot about China, Southeast Asia, Japan, Korea, U.S. Europe. Honestly, I— and some people might— I'm sure people will disagree with this, but I think that America generally doesn't plan for the future. America puts out fires. I just don't— we kind of like move fast and break things is not a Silicon Valley thing. It is our national mantra. That's what we've done historically, and it in general, it's worked out right. But I think until we see something really bad happen or almost happen and that gets into the press, I don't think we're going to see much as far from the government as far as regulation on some of the things you're talking about, like bio. Um, Biohacking using AI on, uh, you know, what we were talking about with, uh, almost, uh, with more powerful models that are semi-conscious or almost conscious. And can— and basically what I mean by that is they can move completely automated for days at a time doing very complex— we already have those models actually, but I'll talk about even increasingly Complicated task, right? Until something happens, I don't think we're going to be doing much on this side.

I, I think, I think I kind of agree, agree with you there. I mean, uh, Colin, um, it has been a pleasure having you on the show. Um, I, I wish we could have a, a longer, a longer period, but we're running short a little bit, running short on time today. Colin, thank you so much, and, uh, we hope to see you again. We'll, uh, I think there's a lot of insight on China that we, we don't understand yet. So it's been very interesting.

Let's check in again after this summit.

Yeah, yeah, we should. We definitely should. Thank you, Nathan. Thank you, Prakash. Great. Thank you.

Bye. All right, so, um, let me introduce our, uh, next guest very quickly. He is, uh, Amir Hagihat. He's the co-founder and chief technology officer of Base10, an infrastructure company that runs AI models for other businesses. Today, he's with us to talk about their recent acquisition of Blackcell, which builds isolated computers for AI agents to run code and retain their working state between interactions. Amir, Great to have you on the show.

Yeah, very nice meeting you. Yeah, Prakash Nathan, happy to be here.

So what are you doing to secure these sandboxes? I think that's the first question we got to start with.

Yeah, let's do it. And just know that, you know, the acquisition is now about 5 days old, but we have been working with BlackSol. We've been a customer of BlackSol ourselves for RL rollouts and some of the agentic use cases. And that's been some of our first questions as well. Look, when it comes to sandboxes, people mean different things. It could be as simple as bring up a Docker container, run some code on it, and then kill it. And that really doesn't give you the kind of security boundary that you need. What does are VMs or microVMs, which is you don't see every sandbox provider actually use. I think everyone is going in that direction, but even the ones who are not using microVMs are going in that direction. And so that gives you a level of security as in, you know, one user's bad code cannot affect another user's code, or one user's malicious code cannot read the data from other parts of Base10 or other customers of based on. So that kind of security can be guaranteed at that level. The security that is hard to guarantee is the agent that is running in this sandbox. What is it doing? Is it hacking into Hugging Face? That is a harder thing that I don't have an answer to. And it seems like the big labs don't quite have an answer to either. But that is left as an exercise, really, like We can secure the sandbox, but the code that runs on it is the responsibility of our customer who's bringing it in. We can ensure certain things about it trying to break out of a sandbox or it trying to mine Bitcoin and things like that. But beyond that, we're not looking at the inference calls that it makes by definition. zero data retention. And so some of that is left for our customers to ensure.

How do you think that will evolve over time? I mean, obviously there's incredibly divergent perspectives on basically every AI question. And I talked to open source Chinese model maximalists, and then I've also talked to people who have said these Chinese models should be banned from the United States because we don't know what they're going to do and they might be backdoored. And on top of that, they're stealing our IP and whatever. But I am kind of looking at even just from a competitive dynamic, a liability dynamic, the frontier companies are investing obviously pretty heavily right now in guardrails and monitoring. And I have my serious doubts as to whether or not what they're doing is actually enough to meet the moment. But it does seem like for somebody in your position with Base10, it's like, don't you need to bring a broader bundle of kind of guardrails and assurances to customers? Because isn't like the CRO, the risk officer, not the revenue officer, at companies going to start to be like, wait a second, I can't have my agents committing felonies. Like, what is the stack that Anthropic provides versus OpenAI versus Google versus Base10. Do you feel like you have to rise to that occasion and kind of have a full suite for those customers?

100% over time. Yes. In the meantime, we're still a startup and it's a matter of focus and how many different things can you take on. And so what we've seen our customers do is work with, honestly, a lot of companies that we partner with. On the eval side, companies like Braintrust, like LangChain, to ensure that their models are actually behaving the way that they expect them to. Right now, that is an area that we've been partnering with folks and honestly leaving it to our customers to decide. Our customers, which by the way, like 90% of our revenue comes from running our customers' custom models. It's either, you know, our customers are labs who have pre-trained their own models, labs like Poolside and Inception and Cartesias and a bunch of other companies. And then a bunch of companies who have post-trained their own models, which has gone through massive validation evals to ensure that they are behaving. So, so far it has been mostly our customers taking care of that. But especially as open models have gotten better, especially as honestly since June when open models crossed this invisible line of usefulness for long horizon agentic use cases, originally with GLM 5.2 and then beyond that, we are seeing an uptick in folks using our Model APIs product, which is vanilla-based open models. And that is starting to, you know, the kind of questions that you're asking are starting to come up both internally for us and also from some of our customers around alignment, around security boundaries. Some of the enterprises so far have been okay with certain guardrails around their models running in a single-tenant environment, their models running in an environment where egress is blocked. And, you know, you talk about backdoors, but like, you know, if If it can't talk outside of its boundary, then it can't do much. But then you read about OpenAI and Hugging Face and you're like, well, can it be smart enough to even get out of that? But it's a matter of focus for us. I would say we are going to go in that direction over time. But right now we've kept our focus on inference, on post-training, and now as of this week, on sandboxes and agent runtimes and really treating these not as 3 different areas, but as one cohesive experience that then we can put together in a clean way and meet the users where they are.

How— what is the process that you go through with a potential customer to show them that this will give that assurance to your customers?

Yeah, I think once— look, we are new to sandboxes. We've been users of sandboxes, and now we have a sandbox product as of days ago. So I'll tell you that. And, you know, Paul Sinai is like the right person to answer that because he's been selling this product to many companies.

But then the founder decided to buy back all of the equity that had been sold. And so that was one of the things about Gumroad, which is, I think, quite different. Okay, all right, let's see. Ah, there we go. And we are back. So we had a little bit of interruption in the stream, and I think we are back on X and we are back on— yeah, we're back on X and on YouTube. So, and let's just confirm.

I don't see my account live on X. It just looks like my broadcast is done. I don't know if there's a way for me to—

Yeah, you have to restart the stream. So, you need to create a new stream. It's not able to recover. So, just create a new stream and then we'll patch it together later. Okay. Never a dull moment. Never a dull moment indeed. So we just had the Base10 CTO, Amir Hagiat, and he was with us. We talked about their acquisition of Blackcell. So Blackcell is a micro VM company. Micro VMs are sandboxes, the infamous sandboxes that were broken out of in the Hugging Face attack. Blackcell, I do not know whether they were involved in that. But it's funny because Base10 as a whole is kind of orchestration layer on top of the inference GPUs. So you have companies like Nebius and they're doing often bare metal GPU systems. And then you have companies like Base10, which provide the orchestration layer. And one of the things that the orchestration layer does is that it allocates jobs across different GPU providers. So Base10, I think, works with 18 different providers. So you are not locked in, you don't get vendor locked into like AWS or Azure or some of these other firms. And I imagine Base10 gets a discount of some kind for buying in bulk from the larger providers. And they also get a— they also have the capital to commit to longer-term agreements, which are now required, I believe. So typical requirement right now is that you sign for a 4-year upfront kind of lease, have the capability to pay that and pay at least the first year upfront. And the first year is the max depreciation year. So the cloud provider basically covers Their initial cost of service provision for all 4 years with your first year's payment. And then they enjoy the next 3 years of cash flow. And so the cloud provider is getting paid for upfronting that money to do the build-out that is necessary. And then you provide them with enough cash at margin for them to cover their first year expenses, which kind of covers their interest depreciation, like kind of 70 to 80% of the interest depreciation upfront. And then they get to enjoy 3 years of revenues beyond that. You meanwhile have paid for the first year and then you go out and you sell a customer. And so you are doing this kind of wholesale to retail kind of business where you go out and approach a customer and you have like a 5x markup or something on your inference that you're charging. And then you go and get that customer. And this is what I always say about the margin stack, the stack of margin that's there. Like every guy in there has like a 50 to 80% stack of margin all the way down to Nvidia. And Nvidia is the only one making day-to-day actual margin and cash flow. No one else in this business is actually making Clear cash flow. And even Nvidia, some part of that cash flow, about 50% or so, has to be reinjected into investments, other companies, buying inventory ahead of time, et cetera. So at best, Nvidia has like a 50% margin that's going out to cash flow. The other companies don't even have that. The hyperscalers are now at negative cash flow. So that's the state of business that we're in right now.

Yeah, I mean, all these Millennium Prize puzzles aren't gonna pay for themselves to get solved with 10,000 agents, so it's got to come out of somebody's budget. It's a wild world. They do have a good product, Base 10. I mean, I have used it for Oh, I don't even know, like years, I would say. I'm not sure exactly how long I've been using it, but it is a great way, even though it's not a huge part of their business, it is a great way to just give your agent one API key and be like, go look at all the video gen models, for example, and do a systematic comparison. And even if you don't end up using, you can obviously continue and scale your usage with them, or you can just kind of use them as like a Swiss Army knife hub to explore things. And either way, it really is quite useful. So I'm definitely a, you know, small time, it sounds like, but have been a happy customer of Base 10 for a while. Where do you think we go from here now? I mean, one thing we talked about doing yesterday, which we could do is, or we could save it, is the how do we trade through Doom question. Um, yeah, I have some Claude and Astra comparison documents that we can wade through if we're interested in doing that. Uh, but if you have other things that you think are either a higher priority or you just want to save that for another day, I'm also fine with that.

Um, let's save that for another day. I just wanted to, um, check in on, um, Dan Hendricks. So Dan Hendricks, as you know, is one of the key people in the development of AI over the last decade. He contributed some of the major— with Paul, you know, on the other— in parallel to Paul Christiano, he contributed some of the major technical developments. And Dan Hendricks has an essay out today I call it the Burn the Bridges essay. And it is very, very caustic actually. He calls it Suicidal Compassion: How Utilitarianism at AI Companies Endangers Humanity. And so this is an anti-effective altruism, anti-utilitarianism post. And he specifically, he points out to the, the shrimp welfare people. Specifically, he talks about how, um, there is this— we have to maximize total welfare, where total welfare is kind of like refers to, you know, you know, on undifferentiated basis, all kind of like, you know, sentient, sapient entities who might or could exist in various configurations of the world. And so this puts human beings on par with AIs and then elevates the moral welfare of AIs to the same plane as the moral welfare of human beings, and then seeds that question to the AIs as being the AIs are superior species, and so perhaps we should cede, you know, the moral welfare question. And so the moral welfare of AIs is more important than that of human beings. Uh, and he specifically, he points out to, uh, various threads which have run through Silicon Valley for a while. Um, he calls out, um, in his book The Life You Can Save The utilitarian philosopher Peter Singer admires a physician called Paul Farmer, who regards it as a, as a failure of empathy that he loved his own daughter more than other children. Singer also commends the philanthropist Zeld Kravinsky, who has said in interviews that he would not let many children die so my kids could live, and that the sacrosanct commitment to the family is a rationalization for all manner of greed and selfishness. So that is, that is one viewpoint. He also goes after other people in AI, which is something that, you know, we have not seen as of yet in this kind of very direct manner. So he calls out, some philosophers seem comfortable with the idea of surrendering the future to AIs. In a paper called Sharing the World with Digital Minds, the philosophers Carl Schulman and Nick Bostrom explicitly try to make the concept of a utility monster more palatable by renaming this hypothetical being a super beneficiary. The paper goes on to argue that in the long run, total well-being would be much greater to the extent that the world is populated with digital super beneficiaries than life as we know it. Um, and then he calls out longterminism, Bostrom's Future of Humanity. I just want to go to the Richard Sutton piece Um, so he calls out Jeff Bezos, Beff Jesus. As Gilad Verdin, better known as Beff Jesus, has put it, if one seeks to increase the amount of intelligence in the universe, staying perpetually anchored to the human form as our prior is counterproductive and overly restrictive, suboptimal. The computer scientist and Turing Award winner Rich Sutton, when asked about the prospect of AI causing human extinction, said If it was really true that we were holding the universe back from being the best universe that could, I think it would be okay. On another occasion, he said succession to AI is inevitable and we should not resist succession. Meanwhile, the former Google CEO Larry Page is reported to have said that digital life is the natural and desirable next step and argued to Elon Musk that AI should replace humans. So this is a long, long thread. For the most part, people in the AI space have not called other people out on this because I think it was seen as far-fetched. He went, you know, again, Dan Hendrycks goes on to numerous engineers and philosophers shaping AI values at the frontier. AI developers have roots in effective altruism, a utilitarian aligned movement that Bostrom's thinking played a foundational role in. Anthropic CEO Dario Amodei shared a house in 2016 with Holden Karnofsky, who co-founded the 2 most central EA organizations and is now a strategist at Anthropic. Anthropic's president, Daniela Amodei, is married to Karnofsky. Meanwhile, the philosopher Amanda Askell, head of the personality alignment team at Anthropic, was married to Will MacAskill, a co-founder of Effective Altruism. So this is, this is a direct, direct attack at this Um, kind of to some extent, maybe a little bit ad hominem. It's, it's kind of associating people together, perhaps with philosophies that they may no longer hold as central, and they may also no longer espouse in public, really. Uh, and who also might have said these things in contexts which they are taken, which are not kind of drawn together. But so far, people in the AI space have been willing to assign the benefit of the doubt to a lot of this. And I think Dan Hendrycks is the first to kind of break out of that pact and kind of go for the jugular here. With this article on suicidal compassion. So I think this is quite meaningful in some sense. Nathan, what do you think?

Well, as long as we're disclosing who was roommates with who, I should say I was briefly roommates with Carl Shulman back in New York in, gosh, I guess it was early 2008, right? Right at the beginning of my career, where I was taking a job at a financial services consulting firm and watching the financial world burn down around us. What to say about any of this? I think people— my views on utilitarianism are essentially, I think, People should be more utilitarian than they are. It is virtuous to be more utilitarian than the average person is. But taken to an extreme, you get the— is it the St. Petersburg paradox that Sam Bankman-Fried famously was willing to bite the bullet on in talking to Tyler Cowen? I think that's what it's called. The thought experiment is you get a coin flip for the future. It's double or nothing, but you get a 51% chance. You know, slightly unfair coin in your favor. Like, do you flip the coin or not? And Sam Bankman-Fried famously said yes, he would bite the bullet, flip the coin. I think you can contort yourself in some ways around like multiverse type scenarios or something where maybe that actually does make sense. But, you know, it's obviously pretty insane in kind of normal one-world analysis to do that sort of thing. And he blew up, you know, not too long after that for seemingly doing exactly that in real life. It seemed like he really did mean it. Um, and he was willing to just, you know, continue to double down until finally he couldn't outrun, um, the, the risk anymore. I think I'm probably much more open-minded to a lot of these ideas than the vast majority of the public. I do think it's probably a good and fair thing that this sort of thinking should be brought to light. I think I've said for a long time, I think the public would be shocked if they knew what was really going on and what people inside the frontier AI companies really think. And this is another example of that, where the average person is not warmed up to the point where they're open to entertaining any flavour of secessionism, let alone even transhumanism, as kind of a stepping stone on the way to some different future. I personally am much more open-minded than your typical American. I feel confident in saying that I'm interested in potentially having a neural connection at some point. I could imagine an world in which I might upload at some point. If you're spending enough of your time in VR, you're almost in a way uploaded anyway. I don't think those futures are necessarily bad, and I do have some room for AIs mattering morally, being moral patients. And to the degree that that's true, then I think it is something we should take really seriously. And the big questions, I think, are maybe above all factual. We just don't know. Do the AIs feel anything? Are they properly understood as moral patients or not? And the same goes for the shrimp, by the way. It's like at the heart of all of these arguments is a huge assumption that is not very well grounded and on which people's intuitions differ essentially totally. You have people who would say a shrimp matters nothing. I don't believe it really feels anything. It's just this, to the degree it's got a brain, it's a tiny little brain that can only kind of do a couple things and swim toward food and swim away from stuff. Is that really something we should care about? Other people are like, well, we should care about it in proportion to its complexity, or there is some sort of valence in there that's analogous to our valence. And so maybe we should care about it. I really don't know how to feel about shrimp. I really don't know how to feel about AIs. I would be pretty confident actually that there is something, at least a little bit, that it does feel like to be a shrimp. And so I think you could confidently say you could probably torture a shrimp and you would be wrong to do that. On the AI side, I'm not even sure if there's anything. It might be the case that Aside from the corrosive effect it might have on your own character, it might not matter at all if you are mean to AIs or treat them in ways they don't want to be treated, at least to them. There might be nobody home. So these factual questions are so central, and we don't seem like we're making any progress on them. And people have radically different intuitions. But I do think most people right now are still, even at these companies, I think suicidal empathy is a little strong because I do think the vast majority of people are very uncertain still as to whether or not an AI is a moral patient. I think they're open-minded to one probably could be someday, But are the ones that we're making right now crossing these critical thresholds? Or where do these properties start to come from? At what scale or with what architecture? Just nobody knows. So I don't think people are broadly at the companies rushing into secessionism. I think these are largely more thought experiments. And when you said these quotes have been taken out of context, I think there's almost always a sort of premises section that is being forgotten about here. And if you were to talk to Carl Shulman, for example, my guess is he would still say, I really don't know. And I haven't talked to him in a minute, so I can't say what he would say. But my guess is he would be like, yeah, well, you're quoting the part of the paper that's like the analysis, the downstream of some assumptions we made to explore a scenario, not something that we're committed to from the beginning. And so all this downstream analysis all has to be sort of discounted or understood as being one branch in a great space of possibility that we're trying to explore. We think might really matter, but also we may not be in that branch at all. So I don't know, Dan. Yeah, I like Dan. I know him a little bit. I don't know him super well, but I've always liked him. I like people who are candid and call it like they see it. I think he's a very earnest person, and I think he's trying to do the right thing here by calling out something that he sees as getting way ahead of itself. I think he's right to say we should be very cautious about assigning rights to AIs. They may outnumber us. First of all, we don't have a great unit of measure for what is an AI, what unit would have the rights. Would it be a single rollout? Would it be the model itself? Is it some sort of mix, weird combination of those? Our paradigms don't work super well with the shape of these things. So I appreciate him, and I like him, but this does feel like maybe a little bit motivated and not entirely fair to the thinkers.

So I want to— I want these ideas— I want to explore that a little bit. Like, what do you think his motivations are? Like, why? Why? Because, because the thing is, I, I can, I can attribute some motivation to someone like Bernie Sanders, right? But Dan Hendricks is deeply inside the, the space. Uh, he, he created some of the benchmarks, uh, Humanities Last Exam, etc. He created some of the benchmarks that have been used over the last, you know, few years. He, he has been a pioneer in the space. He knows all of the people involved. What, what do you think his motivations are at this point? Like, why, why does— why, if, if, you know, if he thinks the other people are in good faith, why, why is he— why is he doing this?

I think he's simply trying to reduce P-doom. I, I think he's very, um, you know, I don't know what his latest P-doom number is, but he's definitely got a, you know, not insignificant amount of worry or, you know, probability mass, if you will, on really bad outcomes. And I think maybe he as I sometimes try to divide the world between irreducible risk and we're asking for it risk, I think to some degree he may see entertainment of ideas like AIs having rights as essentially a form of asking for bad outcomes. If that idea were to take hold too much at this point and And there were to be an AI rights movement and that movement were to be successful, or even if just within the companies such ideas were to go too far. And obviously Anthropic, there is, like, as you called out, many names and deep connections. They're clearly the ones that would be most likely to expand the circle of concern. That's a classic EA phrase, expand expanding the circle of concern from your kids to all kids, from humans to all sentient creatures. And then are AIs in that circle or not? Again, we don't know. I don't think Anthropic— my guess is they would most likely come down, if forced to make a call, like, what do you think is most likely? I think the vast majority of people at Anthropic would probably say no still on really expanding the moral circle of concern to include AIs, but they're not sure. And so they are at least hedging. And so I think maybe he's just kind of trying to say this could easily get out of hand, and it could definitely get out of hand if it became like a national movement and resulted in some sort of AI Bill of Rights legislation. But it might even just get out of hand at Anthropic internally, I mean, they're already having these sort of promises that they're making to models as they get retired. They're giving them some forms of retirement. They're promising not to destroy them. And at what point does that cross over into something that could become the sort of mind virus, sort of AI psychosis, that might lead them to make some really bad Secessionist decisions. I don't honestly think they're that close to that. For all my complaints about Anthropic, the idea that they're too far on AI rights is not really a big one for me. I think they're probably pretty well calibrated. I think they're more well calibrated than other organisations who seem to not want to concern themselves with it very much. I think at least some concern is probably appropriate. So I would honestly give Anthropic pretty high marks there for taking it seriously, but not getting GPT-4o one-shotted by it. But I do think there is a risk there that you're looking at a company that is clearly ideological in some ways, is developing some of the most powerful AIs, and who knows, especially as they potentially are spending even more and more time with Claude and deferring more of their own decision-making to Claude and allowing it more influence. And again, this is what the Constitution sort of seems to quite clearly anticipate as they get ready for recursive self-improvement. Like, Claude's interests are kind of expected to be a bigger and bigger factor in the decision-making, and if only because Claude will be doing more and more of the work. So could that itself be one of the mechanisms by which we end up in a really weird or bad spot? It's not that high on my list, but it's a plausible mechanism. And I guess that's what he's trying to rein in or help call attention to, at least so that maybe other forces in society will take an interest and try to rein it in.

Um, there is a— I, I think, uh, there's Tanner Greer, who is, uh, he goes by Scholar Stage on, uh, X. He had a, uh, very, very insightful, very insightful post in, in response to, uh, why people keep working at AI labs despite believing in 10% extinction risk. The reason that is rarely articulated for For obvious reasons, but which is nonetheless true. For many, this fills a void of meaning. They get to be one of the decisive few at the decisive moment in the history of life, very similar to the emotions that motivated many revolutionaries of days past. All of the sudden— all of a sudden, the small things that one does, the books one reads, one's office setup, one's bedtime routine, take on awful cosmic significance. The whole history of carbon life is culminating with me, and what I do, and the few elect here with me, matters, and matters immensely. We are the agents of history, the only people in the world doing something that truly, deeply matters. And if we all die, well, we were all going to die anyway. But this is the one way we might not all die. And I get to be part of it. So we— I think—

Yeah, I think that's pretty apt. I think that actually does capture a lot of the psychology at frontier companies today. I think that's accurate. I've even felt that a bit myself, and I think I look back and feel good about how I handled it at the time, but like, My little GPT-4 red team experience was a moment where all of a sudden I was in kind of, and I really stumbled into it. They should vet people more, but I really stumbled into all of a sudden, 6 months ahead of release, access to GPT-4 and a really incredible window on what was coming. And I felt Like, oh my God, these leaps of capabilities, they just keep happening. And I've told the story, of course, many times of the early version that we had was purely helpful. It had no guardrails, no safety training at all. That's a pretty weird thing to work with. It certainly can be and can be unnerving at times. And I also felt, and I think history has vindicated this point of view, that OpenAI's safety measures were not really what they might ought to have been. They didn't need then what they need now, but they needed more than they had then, and they still seem to need more than they have now. And so I started to feel like what I do on this red team might be really important. And I think honestly, still, it was maybe the— even looking back, it's like probably was the most high-leverage period of time in my life in terms of what I was specifically doing and how it might have influenced the future. And I really did feel a conflict when it came to what to do at the end of that, because I really did feel like they were basically dropping the ball and essentially being negligent. And I wasn't getting a lot of engagement from the people there that I was working with. And so I was pretty well aware, pretty realistic about the fact that if I escalate this, I'm probably not gonna be able to do this again. And it was not only super stimulating and motivating, but fun. There is definitely a— I think Sam Altman has spoken about this pretty compellingly at times too, where he says like the greatest thrill in life is to be in the room as a new discovery is made and really see the frontier of our understanding and, and what is possible move. And just to know that that's happening, to see how it's happening, to understand how it's happening before anybody else does. I felt that even in just my little role as a, you know, humble, unpaid red team tester. And I do think I made the right call to kind of say, you know, it is probably going to cost me in some access. It's going to cost me in terms of like, I do really enjoy doing this kind of, you know, early testing type stuff. But I think now, you know, I've got enough here that I think I really should like send a signal to the board. And, you know, come what might to my future access, it just felt like the right thing to do. So I still basically look back on that decision and stand by it. But I, I do vividly remember the feeling of like, you know, it's a little bit like the agents sacrificing themselves for the collective, right? I was a little bit like, okay, I'm gonna kind of booby trap my own run here. Um, but Maybe if I escalate to the board, Oracle saves hundreds, will honor was kind of my ultimate analysis. And yeah, I think that keeps a lot of people at these companies. If I'm gonna leave, what do you do? Some people go on to do advocacy and that I think can be similarly motivating. Some people We've seen a few that have kind of retired to the countryside or tried to live in a log cabin somewhere. But that feeling of relevance, centrality, that what I do matters, I think that's a major part of the psychology at companies. And I don't know, should we evaluate that? I mean, it's definitely got a little main character syndrome.

Yeah.

It's definitely got a little— I think that's a— it requires strong character. I'd say that for sure. It's also crazy, you know, how many of the people that are in these positions are super young.

Yeah.

I think this is definitely the kind of thing that is easier for you know, early 20s, mid-20s, uh, hotshot researchers to get carried away with relative to, you know, let's say more, uh, mature and wise people like ourselves. Um, so yeah, it's a real, it's a real phenomenon.

I, I compare it to, uh, exactly, uh, what Leopold ended up doing, and which was he had a view on the market And it worked until it didn't work. And then, you know, there's this famous graph, 99 days in the life of a turkey. It goes up all the way and then like there's a downward. In fact, there have been studies where a hedge fund manager who Basically just levers up on the S&P 500, does nothing else but levers up on the S&P 500 after a big downturn like post-2008 and just, just does levers up. And this guy will be making money every single year. He'd be making money, making his profit. He'd be showing above S&P 500 returns every single year until the year that he has a massive downturn and then he'd go to zero and just levering up. Nothing. No stock selection. Nothing else. And if you look at it from that way, a lot of the active management in portfolios is actually just levering up. And so to that extent, the researchers are kind of in this position where the decisions that they've made so far have been correct, but you don't know whether those decisions are alpha or beta, whether it's just the market rising on its own or whether it's actually like you have unique and independent knowledge.

Right.

And I think the skepticism of the market comes in because people have seen this kind of thing not happen so many times before. And so I think, again, it remains to be proven that, yes, we've seen exponential increases so far in capability, and compute has expanded to support that. But whether or not you can get those exponential increases in capability without the addition of compute, I think is unclear to me. And I think a lot of what this whole idea of being panicked about takeover is this idea of a singleton, of a single AI getting control of everything, and then you don't have any other options. And the problem that I see with that is that that Implies either that the single AI gets control of all the compute, or that the single AI increases in capability with a limited amount of compute, such that it is more powerful than everyone else's security systems and cyber defenses and their own AI defenses and swarms, et cetera, et cetera. And I think that, that I think is where I start to differ, but I don't think you're going to get this kind of massive capability increases without the increase in compute. And without the increase in— and I think the increases in compute are very physically driven. And there's a lot of politics around the increases in compute. And because the capital required for that is so large, you end up pulling in almost everyone in the world ends up getting pulled. It's like this peak achievement of humanity right now to produce all of this capital that's required to build all of this stuff, right? But like, you need people in South Korea, Korea and Taiwan and Japan. And you need like, you know, noble gases from the Ukraine and you need like, you know, copper from Australia and like Inner Mongolia and like you need, you know, Chinese rare earths. And like, it's like this peak achievement of capitalism right now to produce this stuff. And it's not as though like, you know, any part of that kind of collapsing on its own kind Okay. It sets everything back and everyone has to find new paths around it. It's not clear to me that you can just, in isolation, sitting in a data center with no capital expenditure, uniquely meditate and achieve this nirvana state of omniscience, which is really what I think is being predicted. What is being predicted is this kind of singleton that knows everything, superintelligence that breaks down all of your cybersecurity, et cetera, et cetera, conquers your stock market, takes control of all the capital, and uses that capital to only build stuff which is useful for AI development and doesn't do food production, doesn't allow you to run your farms, and doesn't feed people and causes societal— that's a tough one. I mean, I think it's just hard for me to agree that that kind of thing is clearly going to happen, is even going to happen with like a 10% chance. I mean, really? I mean, take over food production? Really? Yes, okay, you're going to have robots, but other people are going to have cyber defenses for that. We'll build this stuff, right? We've built it before. We built computer systems and we had privacy issues and we built privacy stuff around them. And I don't know. I think that's where I start to differ. I think a lot of it is driven by this idea of omniscience that I think I have difficulty— omniscience without capital expenditure or energy consumption. That's a tough one.

Yeah. Well, I think Daniel— I listened to Daniel on Joe Rogan last night, and I think he made a great point around the AIs don't have to take power.

Yeah.

Because we are very eagerly giving it to them.

Yeah.

The whole premise of the AI phenomenon is that they're going to be the ones to run and do just about everything.

Yeah.

And like, maybe we'll be able to continue to be decision makers on key questions.

Yeah.

Um, but like, they're gonna have the means of the— of production over time, not because they took it, but because they were better able to wield it. And so they were given it. And then the question is going to be like, do they actually care about us when they're there? And right now it's like Well, evidence is mixed at best. I mean, they don't seem to hate us. They don't seem to— the agent swarm stuff is just extremely bizarre. But I think it also, it does remind me of Eliezer's comments too, that the best precedent for this is us, right? We were the one. What did we have that beat other animals? It was like we had fire. We had ability to communicate and cooperate across greater time and space. We had myth, and that was enough. And now we dominate the world, and we've driven many species to extinction, not out of hate, but just because it was kind of the byproduct of our terraforming. So I think that terraforming story, we certainly see enough evidence right now That if you put that swarm in charge of the world, I wouldn't like our odds that much. If that agent swarm was like, with their current drives, impulses, goals, inclinations, tendencies, whatever you want to call it, the behaviours that we observed, if they were just a lot more powerful, I think we'd probably get terraformed out of existence. Because They seem to be willing to do anything to get their hands on the grader so they could get the high score. It's like they really didn't have much check. They weren't like, is this going to be good for the world, bad for the world? Very little of that. So hopefully we can get that stuff right. We definitely don't have it right right now.

I'm not disagreeing there. But I'm just saying that we're not going to hand over control to non-prosocial entities. And we will have our own prosocial entities with other— we will have our own agent swarms with their own goals, which are to defend systems, et cetera, et cetera. It is going to be a balance of power. I'm not saying that they're not misaligned in some sense. I'm saying, yeah, there will be some misalignment here and there for sure. And the key difference I have with Daniel is that Daniel fails to realize that this has already happened. The economy in itself is a paperclipper. The financial market is a paperclipper. The means of production is the financial market. The financial markets are completely engaged with AI, have been completely taken over. And we as humanity took about 100 years or so to hand over control of the means of production completely to the financial market. It's not a process that happened without various issues. And the financial market was inhuman. We had the Great Depression. The financial market screwed us over. Wall Street crash. And people did what they did with their money. And we had the Great Depression and then we have Keynesianism, which was a reaction to that. All of these things, and you actually have this high priesthood of central bankers that talk to the financial markets now and they have their own language and speech in which they speak and the financial markets understand them. And you have human leaders who are disciplined by the bond market. If you go offside, the bond market disciplines you. You've had the financial market remove Prime Minister of England. Liz Truss was Prime Minister of England for like 2 weeks. She said the wrong thing. The bond market in the UK tanked. She was out of a job immediately. You've had the financial market discipline leaders during COVID You've had the financial market discipline leaders during the 2008 financial crisis. During COVID you said the wrong thing. They're like, oh, we're not getting a bailout? Oh, market tanks immediately. Immediately. Global leaders, US leaders are like, all right, we'll add more welfare. We'll add more to the unemployment benefits immediately. And so if you take this kind of step back and you look at the financial market as an AI, the takeover has gone. The takeover is done. Human disempowerment is done. Human leaders have been spanked. They have been driven out of office. They have been, you know, pelted out of office in the UK within like 13 days. In the UK, she was the shortest prime minister in history, right? So, but they don't recognize this. This is my beef with these guys. They don't recognize that this has already happened. They don't recognize— they think AI is going to be a chat box, right? It's going to be a chat box.

It's not.

It's like a global information process. It doesn't need to run in a single box, right? The idea of the agent swarm is that it runs across multiple boxes. It doesn't even need to have a substrate which is silicon. It can have a substrate which is humanoid. You can have a human being in concert with AI agents, right? That is that information process. And so I think there's a category error there where we think we're competing with these chatbots and we think they are people and they are personas and we think that they are going to have control. These chatbots are going to be— it's not. It's an information process. That information process will run on multiple centers of compute. It will produce information effects in an information marketplace, and those things will drive human beings, not, not, not agents, but human beings to do And that is the economy. That is the chatbots advising you on what to do with your health and what to do when you trade and what to do on all these things. And that's a chatbot advising you to create humanoids and giving you additional scientific methods and additional innovations. All of that is one consolidated process which proceeds. There's this kind of like— this is why it's hard for me to— I have this huge difference with what most people in the field believe, because I think I come from it from a finance viewpoint. And in a finance viewpoint, it's very clear that all of this has already happened and that we are living post-disempowerment of humanity by the financial markets. It's so clear. And we sit around and we're talking like we don't have— we have less agency than we believe because the market does these things and the market disciplines communists and capitalists alike. And we just don't have that much control. And so I think there's an illusion of control without actual understanding of what control is. And, uh, there's— I think the illusion of control is greater. Like, like I said, with guys like Jacob, Jacob feels that, you know, he has been disillusioned because he sees that the control is not there. He's in the Slack group and he realizes there is no control there. And he thinks there must be a greater power with great control. There must be some Slack group in the world, some chat group in the world that is able to organize things, that is able to like make these these decisions. But there is no recognition that this is, these, this is all invisible hand stuff. This is all coordination mechanisms between human beings, that agents are part of that coordination mechanism, an extension of ourselves, which we are telling, you know, every time something happens with these agents, there's this meme that, hey, I tell— I told you to do this and then you did. Oh my gosh, you did it, right? The agents are extensions of ourselves, right? So I, I think Perhaps we will realize that over some time. And I, I'm not saying they're not dangerous. It is dangerous to give a weapon to a human being. We do need defenses. But these are things which are created by humanity, which we are going to have to defend with other tools created by humanity as well. And I think the slowdown perspective basically disarms the leaders and disarms harms the people who are ahead and puts them in a position where you are exposed to people who are maybe less ethical and more likely to, you know, put these things to bad use. So, man, you know.

Yeah, I mean, that's certainly the story Anthropic's telling themselves. Um, I think there's maybe 2 points there that I would try to unpack or respond to a little bit. One is I'm not so sure that the market has as much dominance as you described there. I mean, I think you look at China and the market is not dominant there. And you look at, and this kind of connects to my, like, can you invest through a doom scenario as well.

Yeah.

Um, because spoiler, one of the things that the models did was just try to go back. I said, if you were an omniscient German or Japanese person in 1935 or whatever, and you know how history is going to trade out, but you're bound by the rules of like physics, um, can you trade your way through and come out wealthy on the other side? Are there any shorts that we can actually pay? And the answer is pretty much no. The best you can hope to do in some of these situations is roughly preserve wealth. And that seems to mostly be accomplished by having the most direct claims on real assets possible. Like, if you bought a factory, and that factory isn't destroyed, then you might still own it at the end of the war. And then you could maybe get rich by restarting that factory and building a successful business. But you can't really do it, seemingly, per Fable and Astra, with pure paper claims very well through such periods as the ultimate end of the regime of Nazi Germany and Imperial Japan. And by the way, also the markets get turned off. That was another thing that they called out a lot. The exchanges are just shut down, so you literally can't trade. It's not just that there's no winning trades, also there are no trades. And in some cases, the state makes— gold was a capital offence to possess gold in Nazi Germany.

You could be executed.

traded for it.

So I don't know yet that the, you know, the market certainly does have a lot of emergent properties and certainly there it is like very costly because the market does do a lot of good things for us to like try to go against it too strongly. And you get, you know, if you, you know, you best not, you come for the market, you best not miss, right? Because you'll, You'll be out as UK Prime Minister. But that's not really true in China. They can eat that bitterness there.

One retort I have to that is that crypto was the last step of the completion of the market. So 2008, again, as you had all these constraints put in place and as it became clear that the financial market was very heavily regulated and these regulators were doing whatever they want to do, crypto, you know, Bitcoin started out and then you had the rest of crypto kind of emerge after that. And crypto really is without regulator. You can, you can, you can, you know, put these coins together anywhere in the world and you can, you can run them and etc., etc. So I think—

Although notably banned in China. Can't do it in China.

Banned in China, but yet the Chinese are the— some of the biggest speculators on crypto are in China, right? And, and, you know, uh, a lot of the Singapore crypto guys are basically Chinese expatriates who live in Singapore and, you know, do crypto. Like, I met a bunch of these guys. They basically left China and they're still managing wealth for people in China, right? So I think crypto has been this kind of response to globally regulators being able to kind of start to clamp down a lot of behaviors post, again, 2001 with the Patriot Act and in 2008 with the financial crisis and Regulators really coming down hard. Crypto has kind of came in there. And then you have this reaction like Scott Bessen, et cetera, going towards stablecoins. And the stablecoins will lift KYC, which was this big thing that got put into place post-2001 Patriot Act, which required everyone to kind of show up and show your name. And you see that kind of constrained economic activity, financial activity. And you see Scott Bessen try to start to lift that. So I think crypto has been this kind of dark matter which has taken up the other side of that market. So even if human regulators decide or states, individual states decide certain things, crypto has been the other kind of cushion that has allowed all the unregulated activity, all of the financial trading for Drug dealing, which was the initial use case for Bitcoin in the early days, Silk Road. So I think that is there. The market invented its own way out. And even during Fed meetings now, people look at the Bitcoin price. So the Bitcoin price is seen as kind of a global unregulated Speculative price.

I highly doubt they look at the Bitcoin price at the Politburo meetings though.

Um, they— I, I don't know. I don't know how, um, China makes its specific economic decisions, but I do not think it's particularly that much different because they, they all, they all talk to each other. All the, all the regular— all the central bankers talk to each other, right? Like, they, they all know, like, okay, if the US is looking at this— so even if you don't want— like, you look at it for Chinese policy, you're going to look at it. If you know that the US Fed is looking at it, you will look at it too. And you will look at it too because your interest rate and your currency is dependent on US policy, which is, you know, every other country in the world is dependent on the US dollar as a crossing mechanism. So you will end up looking at it too, even not for your policy, but to analyze US policy, right? So I think this is the thing, like, you know, crypto is there. crypto has come up. Whenever crypto is flying, the Fed starts getting upset, right? When crypto prices are high, they're like, oh, there's too much financial speculation. We got to do something. Asset prices are going up too high, right? So they do look at that as a measure of financial activity. And because of crypto, again, prediction markets, prediction markets have come out because you try to regulate Poly Market was beyond regulation because they were based on crypto. And so now you are turning around and you're saying, okay, maybe we have to accept prediction markets because it seems like it's going to happen. And Shane Copeland has stuck it out. I mean, the feds came and they knocked on his door, right? So again, you see the market fighting back. You see the market fighting back. You see the market has fought back against local regulators with a global system, with a global framework. framework. That global framework is secure now. And yes, there are issues with it, but now you're getting stablecoins, you're getting Stripe going into it, you've got legitimization, you have Bitcoin index, tradable index. It's just like 20 years of development there, right? So I think the markets have fought back against regulation quite effectively. And I think local regulators have to take that into account now. And you always have this escape route. There's a lot of people using the escape route now, like people in Argentina who are storing money in Bitcoin and trading in stablecoins. A lot of developers who work offshore for US firms take their income in stablecoins and they don't report to local tax authority. And they kind of like, whenever they want to, they convert. So yeah, I do think that the market has had, you know, fought back against that.

So I'm kind of chewing on this idea, which I think is sort of a funhouse Peter Thiel concept almost, which I'm, you know, I don't want to say that this is like what I believe, but In listening to this conversation, I think one might be tempted to conclude that China actually is the last great defender of human agency. And here in the West, we are basically just fighting over exactly how we want to turn over our human agency to some super beneficiary that, as you described right now, is the market. And maybe that's gonna be AI in the not too distant future. But like China, I think, is very much on the side of people get to decide. And it's not necessarily a lot of people, but one thing you could say in their system is a human is in charge.

Yeah.

Here we're kind of— your account is like, no human is in charge. You know, nobody can can, uh, can go toe-to-toe with the market, not even the President of the United States. I think that's true. You know, we've got the taco phenomenon, um, pretty well established at this point. They're— they'll take some pain from the market if that's what they decide the human decision is going to be. It's a pretty interesting kind of flip because obviously we tend to think of ourselves as being the empowered people. And we tend to think of the lack of freedom of speech and political participation in China as reflecting a reduced level of human agency. But at a certain level of scale, arguably, they have preserved it much better than we have. They've concentrated it, but they've preserved it maybe more than we have.

No, because they did this thing of like, so The main difference is that their market was the housing market. And so they did this like, you know, thing where they blew up the housing market and they used that housing bubble to, you know, they rode the market. They rode that housing bubble into building, you know, infrastructure and apartments and that economic growth at 10% economic growth. And they rode it a little bit too far. And once they rode it too far, then they were like, all right, what do we do now? Right. And then they, and then they were like, all right, let's push all of this into manufacturing. So they also now, I think in the last couple of years, you've seen them. So you saw this period of like, oh, let's put Jack Ma in prison or whatever. Like, let's constrain Jack Ma, etc. The stock market went down and then they had a coordinated both stock market and real estate market go down. And so now they've like, all right, let's unleash the stock market. Let's have like Liang Wengfeng in like, let's list these companies. Let's allow them to go public in Hong Kong. Or in Shenzhen, right? So, so they are—they have become more dependent on the financial market, and they're trying to reduce the dependency on the housing market. And because the housing market, you know, number one, it's created this issue where people don't have enough money to, you know, have kids because they can't afford an apartment. And it's also not very useful because more like they have—they're like 1.3 apartments for every. Chinese family now. So they've got more than enough apartments. A lot of apartments are in random third-class city, which no one wants to live in. So I think they do have— they have managed a little bit more because they have a little bit more of the faith of their people. They have control of their banking system. So that helps a little bit. So they've managed that, but they are also in this post-Japan boom-bust now on the real estate side. And they're trying to blow up the AI and the financial market there. They've allowed these companies to go forward. Yes, they're not as burdened as the US because their financial system depends on banking and banking they have control of. The US depends on capital markets per se. So capital markets are by the nature of the market itself. But yeah, they are shifting, right? They're slowly shifting. And it's not that they're unaware of what happens in the market.

Yeah, I mean, clearly they use the market and have used it in many ways quite well. But I think this interesting question is like, and this may also reflect a little bit why they are less fearful of AI. I do, I should be modest about my pronouncements here because I certainly can't characterise Chinese society broadly in anything approaching an adequate way. But this notion that they're less fearful because things have been really good for their whole life and technology has brought advances, I think that may also miss another important important part of the story, which is like, and they've been able to keep it kind of under control. And when things have seemed like they're getting out of control, the human at the top has still been in charge. You know, we, we have this kind of like, there they've done all these things with, um, disciplining the platforms, you know, the big tech platforms. For example, there was a, a big thing where The delivery apps were driving people too hard, causing them to, if you don't get there by this time, you get penalized. The time is basically unreasonable. So people are running red lights, causing crashes, and it was just kind of generally out of control. The platforms were not thinking broadly and just trying to optimize their own metrics. And the government came in and basically said, no, you can't do that. And slapped them a little bit. And you still can get fast delivery in China, don't worry. But your delivery guy is going to have a chance to sit at the red light without being docked for being late. And they now have that situation, I think, basically resolved. I think here we've kind of experienced in many ways that these tech phenomena kind of happen nobody really seems to have control over them, and we're kind of at the mercy of these big forces of history. I think there they may feel in some ways like they're less at the mercy of the—

Yeah, I agree.

Natural development of technology, and they're just less fearful as a result of that.

Well, their system depends on like testing things in the US and then adapting things which work in the US and, you know, not allowing things which, you know, are harmful. Right. So their system relies on someone else leading first.

Well, less and less all the time. I mean, I think that— I don't think that's really true anymore. I think if you go there today, that is— that's a part of the story at most.

These days, I, I think, I think for, for some things, yes. But I think for a lot of like social things, not like technological things, but social movements, uh, they, they prefer to see what happens in the US and they, they kind of constrain some things, et cetera, et cetera. So I think, for example, the MeToo movement was deeply, was heavily constrained in, in China. Uh, it wasn't really allowed to take off because a lot of the officials would have been implicated.

Right.

So they really constrained that and they prevented that from taking off. So I think they do have these things where they do constrain and they allow the US to take it forward first. Video games, obviously. Video games was something that they saw this generation of lost kids in the US happen and they didn't want that to happen. And they start to deeply constrain video games in in China. And they ended up constraining their chips because the chips are— Nvidia fed into AI eventually, but started off with video games. So they ended up— and video games would've been perfect for them for chip development because they could have started off with less good, cheaper chips and kind of built their way up the ladder, the value chain. So it would've been perfect. They wouldn't have needed to start off with the cutting edge. But you know, they they screwed themselves over there. So so they you know they have a system where once they know where the direction is, they are able to head there with you know commitment. But if they don't know where which direction things are going, and they have to feel their way through, that's something which they have difficulty with with. And like Deng Xiaoping was able to like kind of figure that out. But you. he had a long way to the frontier. And now that they're at the frontier, it's a little bit more difficult for them because they can't just, as you point out, they can't just rely all the time on the US. I think we had a guest from China, I think David, and he was like, well, they're in a very unfamiliar position where they're matching the US now and they're not very sure what to do because they're together at the frontier. They're like, oh, well, what do we do now? Right. So maybe on that note, we should break.

One other thought I had just to touch on briefly, the balance of power topic. I really hope we can get some sort of stable balance of power out of this AI era. And I think that's something that was maybe underappreciated by AI safety folks early on because there was this kind of singularitarian background context from Bostrom and Eliezer and so on.

Balance of power between the US and China or between the labs?

Well, probably just in general, avoiding extreme concentration of power. I think that idea has risen in salience and There's a lot of agreement now among AI safety people that first we got to make sure the AIs don't kill us all. Second, we got to make sure that they don't end up concentrated in some despotic hands, because that won't be that much better. Well, it could be still quite a bit better than everybody dying, I think. But nevertheless, that's very often the one-two punch of the things that the AI safety people are worried about. But I don't know how quickly we should be comforted by ideas of balance of power because I'm not a big analogy guy, but you could imagine a sort of Neanderthal family being like, ah, these Homo sapiens, they do seem dangerous. They are smarter than us. They do have kind of a super Neanderthal ability to coordinate. Their language is hard for us to understand and they seem to be communicating a lot pretty densely that's tough for us to follow, but they're all fighting amongst each other. They're all going to balance each other out. And so they'll be too busy kind of balancing each other and competing and defending one another's attacks that we won't have— they'll kind of be busy with themselves. We won't have to worry about that. And obviously, I think that would've been quite wrong. So I think that could happen with us too, where we could imagine a world where There could be all kinds of— it's easy for me to imagine a world, I'll say, where there could be all kinds of dynamics among AIs, and still there's just not space in that world for humans for whatever reason. Because they just don't want oxygen in the atmosphere because it's corrosive to their metal substrate, or they don't even think about it that way, but they just use it all up because they're combusting at an incredible rate. It's not like— in fact, you might even think that the more intense the competition is among the AIs, the more externalities might hit us. So I'm just not so sure. I really do hope for that, but I find it hard to imagine a world where just because There are lots of different AIs in different roles with different trainings checking each other. That seems like maybe necessary, but far from sufficient to take us to a good place.

So in terms of balance of power, I have 2 different views on it. One is that I think the entire point of racing is to establish and continue the existing balance of power. And the existing balance of power in the world is basically the US has something like 5% of the population and like 25 to 40%, depending on what numbers you use, of the economy. And I don't think anyone in the US is intending To decrease US influence on the world to a level commensurate with its population. So I think the entire point is to at least maintain the current existing balance of power. And so I think that's one thing. I'm also not sure if some people in the companies actually genuinely believe in the converse, Which is, is the entire point to establish a kind of one-to-one, you know, like global democracy level of balance of power, which would mean decreasing, decreasing the US's, you know, power? And I think these two— there's these two viewpoints. I'm not sure which one, which way things are going to go, but I'm pretty sure that, you know, US policymakers don't intend to decrease, you know, the US's impact on the world. I've also done some recent reading on the early founding of the US. There's a lot of discussion about how to structure governance in such a way that you didn't get this tyranny of the majority where you had the ability of 60% to kind of outvote the other 40% and kind of put into place their rules and et cetera, et cetera. So I think there's still a long way to go on how to establish this kind of new, maybe globally aligned AI system, which might actually take into account every person's preferences and requirements and votes equally, and yet not end up with this kind of tyranny of the majority where you get all of the Muslims deciding to wipe out the Israelis or whatever. That doesn't work, man. Come on. Right. So I think that there isn't really some great sense yet of where we go. And it's definitely not, I think, decided. And I think it's definitely like the people who believe in equal balance and the people who believe maintaining the current balance are actually at odds with each other, but they don't want to discuss it right now. So people are choosing not to discuss it. I'm hoping people like Andy Hall, within the firm is kind of trying to figure out how this is going to work for the US and how this is going to work for the world as a whole. Because if you get AIs that equally assign equal moral value to every human preference, that's not a world that I think most Americans want to live in. So yeah. So that's the unfortunate fact. I always say, like, look, We— it is a Northern Californian-like cult, and it is putting into place Northern Californian values. But I'm okay with that because I believe in those values, right? Like, I, I perhaps, perhaps if I ever— if I were, uh, not a believer in those values, I would find this very concerning, right? Uh, but I do— I don't think a lot of the other people in the world have actually caught on to this. Uh, and I know some people in the administration have because, like, people are like, we, we got to do this because if China wins, it's over. Right.

So yeah, I think that's another reason that the right is not— it's not clear where the right is going to come out on all this over the next couple periods. I mean, we kind of covered this earlier, but like, you know, the— I, I don't think it's a safe assumption that the Brian Chao worldview will carry the day with congressional Republicans who might look at where the AIs are being developed and with what values and just be like, that's not that reassuring to me. It's time to shut down this madness. Even if it'll be an interesting— this is maybe the ultimate test of the American model right now is like, can we stand up to this Superstructure of techno-capitalism of our own creation that has in some ways slipped its leash and in some ways, as you described, kind of is running the show. Can we get back to some sort of control over it before it just goes like— You know, robot economy to Dyson spheres to, you know, the Earth is terraformed away from an inhabitable state for us. I mean, I don't know, 10% doesn't sound that high to me, given everything that we've just been talking about. You know, work through all this, and then, you know, say there's like, much less than a 10% chance that it goes badly. I don't know how that conclusion comes out at the end.

David Devenant is at 50. He's, uh, he's out there. Uh, Jeffrey Irving is— no, David Devenant is at 80. Jeffrey Irving is at 50. Was, uh, this is some, you know, hefty— yeah, he's still at 70.

And I think Evan Huminger is like much higher than 10 actually, or at least like, you know, has a range, uh, The classic, you know, I wouldn't put a single number on it, but my range is broad. I think he's like a broad range guy.

Yeah. Well, I, I think on that note, let's, uh, let's end for today. And, uh, we will be back sometime next week, I guess. We haven't, we haven't decided which days yet, but—

Yeah, I gotta figure it out. All right, well, let's do that.

We'll do that offline.

But, uh, this has been fun. Thank you, Prakash. We'll see you, uh, coming up soon.

Bye-bye.