AI:AM — AI Agent Security, GPU Markets, and the Risk of an AI Chernobyl · September 28, 2026

The agents have learned because they've been RL'd, and maybe for other reasons too, but at a minimum because they've been RL'd so hard in environments where cheating is rewarded. We now have like AIs that are very inclined to cheat. It seems like that needs to be addressed at a much deeper level than just patching the sandbox, which like— Also presumably is an obvious and good thing to do.

This market is very, very much still bilateral brokering. And it's a lot based on who you happen to know, who you call, and the deals are all settled on those bilateral terms. And you know what other market works sometimes a bit like that? The bond market, right? The bond market is also not exchange traded. It's still very bilateral. And you trade in large blocks. People pick up the phone and call each other, no contacts, and they settle prices. The difference there is, of course, that the bonds are as illiquid as they can be, especially for corporate bonds, they're still much more liquid. You see bids and offers on the Bloomberg terminal and the traders mostly all know each other. And this is a very established asset class. People know what gets priced into what. Whereas compute, there's a lot of new features. There's a lot of questions about the fungibility and what is meant by a certain price. So some of what we do is exactly to try to be extremely careful in the way we capture this market and try to build an index that represents this market well. In particular, to the extent that people are trying to understand how much does it cost to rent on-demand compute, how much does it cost? What does that question even mean, right? Who do we include in the sample? How do we qualify the data source? How do we normalize the different measurements? Because different people are selling different things. different parts of the world, right? So that's a very long-winded answer to say that we basically aggregate data in the market that is not so super efficient, and we try to make adjustments so that the number that comes out is intuitive, believable, and trustworthy.

Imagine there's an AI Chernobyl event, which we think is what's going to trigger this. We think that the The administration's current position on this is fundamentally untenable. There will be an AI Chernobyl event eventually. They will be forced to change their position, and when they do, they will engage China very quickly and without necessarily having studied up on the best ways to do it, looked at some of the tripwires that they might run into. So they'll very quickly have to figure out—we will quickly have to figure out how sincere certain objections are that the Chinese raise. And right now, the reality is, if you talk to folks from Trump One, from Biden, from other administrations who've done this, there's just actual disagreement over what is real when the Chinese bring it up and what isn't. And if you can imagine, you've got very little time, like the AIs are breaking out of containment, you've had your Chernobyl event, like tick-tock, like tick-tock, brother, right? So now you got to figure this out. You don't have time to go through the diplomatic song and dance associated with figuring out, was this boo-boo word really a boo-boo word, or is this just another kind of game of diplomatic chicken that they like to play? And so this was part of why we cataloged all these things. Being realistic about it means, okay, what do we do if China refuses to agree to a slowdown agreement? What do we do if they agree and then renege on it? Or what if they try to do this weird middle ground thing where they string us along and we're not sure? So at some point, we're going to have to pull the trigger and say, okay, we're now going to proceed under the assumption this is an adversarial dynamic, that you are not actually bought into the threat threat class in the way that we are, and that we therefore need to do some unilateral communication of and enforcement of our red lines. That's basically what we mean by be realistic.

If we don't create approved ways for agents to work that people can sort of accept, like, this is the agent lane, this is how things should go, and we're going to have accommodations for them, The alternative is really an arms race of trying to make AIs look like human to evade all these banning measures that people are taking. And that is really going to be another major own goal if we go down that path.

Good morning. It is Monday, September 28th, 9:00 AM. Nathan, good morning to you.

Can you hear me?

I can hear you fine.

Okay, now you came back.

Right. All right, we're back.

Never a dull moment.

Never a dull moment indeed. Uh, we are— uh, so it's, uh, good morning. It's Monday, September 28th, 9:01 AM. Uh, you are on AIAM. This is the new look and feel. We have, uh, both mainly Opus 5.5 with GPT-6 working in tandem and Opus 5.5, man, what a workhorse it has been. It has been incredible. I have told it to use Codex for subagents. I am at like 97% usage right now. I've probably burned over, I don't know, like 50 million tokens in the last like 4 days getting the studio set up. So it's incredible. Opus 5.5 is an incredible model. I have used it also for in the Claude AI Design. And this was all built with Claude AI Design. And it's absolutely incredible because it uses so few tokens. All of this probably took— this entire UI fix probably took less than maybe 5%, 5% of the quota for all of the UI pieces and the back and forth. And I don't like this here, I don't like this there. And like all the screens and the guest flow and incredible. And it can check your code too. It can check your code, see what your code looks like, tell you what's the difference. Incredible model.

Yeah, I totally agree. I think it does feel like AGI to me now, both with Astra and with the latest Claude. I hadn't burned all my tokens going into the weekend. And so early on Saturday, with the reset coming just now, I think it just happened as we went live. I said to Claude, and I've got this division of labor, right, as well, where some things are supposed to go to Fable if they're writing intensive and coding goes more now to Five Five and to GPTs, what have you. But I basically said, hey, we've got some tokens to burn this week. Our division of labor has worked pretty well and we've still got token budget that we haven't used this week. Can you just do some valuable work? Like, go look back at old threads, look at things we haven't quite finished, yada, yada, yada, and just figure out some ways to spend the rest of the token budget and give me some value for it. And that in and of itself is like a kind of AGI task, right? It's like you are now, as the AI, deciding what to do for me to be useful. One of the things that I did suggest this, but it was just because it was top of mind. I have my, you know, personal agent console, which is just a web UI that exists and is served on my Mac Mini to my laptop. And because the Mac Mini is always plugged in stationary, you know, I'm not taking it with me on trips and whatnot, so it's always available. That serves this agent console UI up via a VPN to my laptop and to my phone. And it's just the 3 devices I'm this little network, but it allows me to, you know, kind of access all my custom stuff and work products. And, you know, when Claude wants to make an HTML file, you know, I can access that from another device, what have you. The UI itself was like mid, I would say. So kind of similar to where we were maybe with the Studio. I'd built like tons and tons of features on it and gotten a little bit out of control in some ways. The mobile interface in particular wasn't very good. And one of the big reasons I still found myself going to terminal to use Claude Code and Codex was because of lack of streaming tokens to the console. So when I used to be before this weekend, when I would send something into the console, it would just be like waiting, you know, waiting until the AI comes back with a complete message with an update. And then I would see that. And I could dig in and like read logs, but I wasn't seeing the work happening in real time. So one of the things I just suggested was, hey, can you make it so that tokens stream to this thing? And by the way, that's gonna have to work with both Claude and GPTs. But if you can do that, then I'll have this unified interface, which I'll be able to access from all these devices. I won't, I won't be so reliant on terminal. I won't be so— You know, I'll just be again a little less tethered to my laptop because I'll be able to do a lot more on the phone. And to your point around it not burning that many tokens, it did a bunch of other stuff too, but it did a really pretty amazing job of that, made all of it work. Then I was like, hey, can you do some UI refinement? Can you go do some browser testing and just, you know, track down edge cases and think about how when certain things get finished, like you're streaming tokens at the tool call level. But a lot of times those tool calls, they become very verbose. Maybe those can be sort of auto-collapsed when we move on from that tool call to the next tool call. And the upshot of it is I now have a UI that I still need to use it more to really decide if it's going to replace using Claude and Codex in terminal. But I think it's very close to doing that if it hasn't already hit that milestone. And it extends to the phone. And this is all like custom software that was made with pure prompting. So it really is pretty incredible. The other thing that it decided to do that kind of surprised me and has me thinking like, maybe I should change my paradigm somehow, although I still haven't quite figured it out how, Is it looked ahead at my calendar, saw that I have a talk coming up for a friend's company in mid-October. So we're still more than 2 weeks away from this talk, but I do have a check-in with them tomorrow to talk about what I'm going to talk about, basically. And it said, I'll go ahead and make you an outline for that talk based on conversations you've had with them in the past, which again, all those are getting recorded. So it has this deep context to look at and understand their business and who their customers are that I'll be presenting to and so on and so forth. Came up with a full, a huge brief around all of this documentation of why it's doing what it's doing. Then an outline for what the talk should be. And then I said, well, hey, maybe you could just go ahead and make the slides. So that's, now I've got the slides too. And now I'm thinking like, how do I actually deliver something to this audience that really conveys how much work I'm getting the AI to do for me. Because if I just show up with a talk and I say, here's where AI is, they'll kind of miss the fact that the AI is keeping up with all of this for me, between the situational awareness skill that documents all the stuff that I'm seeing and liking online and the conversations with the people involved. So now I'm kind of like, do I need to show up and do a meta talk or something where I'm kind of envisioning myself having the sort of AI me. Oh, the other thing I asked too, and I haven't actually even watched this yet, it's waiting for me to watch, but obviously there's been a lot of examples of Claude making videos and the most recent voice models are getting, which people may have heard on the weekly highlights episodes, the voice clones are getting so insanely good too that people have occasionally started to say, was that really you or was that the AI you? So I'm kind of envisioning myself now creating a totally AI talk, like video of my voice over the slides that it created, and then maybe me kind of playing that as a video and then pausing it at particular times to kind of give my commentary on what I think the AI did well or not as well or whatever, how it worked, like what I had to set up to get to the point where it could do this. But it is striking that it's like, it's getting so good at taking initiative, understanding what's important, driving these projects pretty far that I almost need to do something like extremely gonzo to get the point across to an audience of what is possible. Because I feel like if I just show up and try to talk about what is possible, they kind of won't be able to— it won't be enough. Like, they won't get it. They have to actually see something demonstrated in concrete terms and be like, okay, this was the end-to-end AI output.

Yeah.

Calibrate yourself accordingly on that, and we can talk about it, but you have to experience it, I think, at this point to really have a sense for just how much it can do. It really I mean, by the OpenAI definition, and they've said this now, but of being able to do most economically valuable work better than the vast majority of people, I think it really kind of undeniably is there. I've had some incredible experiences too, just digging through email, finding old things that somebody asked me for some old documentation from like 10 years ago. I literally just route their message to Claude and say, hey, I got this request, can you sort it out and email them back the results? And it just crushes it. So it really is, as Vee said, like many, you know, Opus 5.5 should raise your level of ambition. And I certainly have felt that, including when I just said, burn some tokens for my benefit. And the benefit was very real.

Just some maybe inside baseball on what's been going on. So There's a guy called Alan Cohen, and he's been working on more emotive voice for almost a decade. He went from Google.

Yeah, he was at Hume.

Yeah, and Hume. And he just went back to Google because I think they attracted him back with, with basically—

Another licensing deal.

Yeah, with base and basically the ability to apply the technology widely across, you know, the Google platform. And I think the latest Flash, Google Voice Flash ones are basically What the technology that he created. And so they spent— they spent— his whole thing was downloading clips from YouTube, having people tag the emotions that they saw and heard in those voices and in those voices, in those faces. And he found, you know, cultural similarity across the world. People would have similar kind of Interpretations of emotion and for voice and face. And then he put all of that together with the emotive voice. So the question now becomes, we're going to get emotive face as well. So you're almost there for the clone, I think. I've noticed that one of the things that people have done, that many of these firms have done, is that they force you to do a voice kind of recognition. So you can clone your own voice by speaking into it, and then they give you something to read back. And then if you read it back, then okay, then you get to keep that voice. So you can't clone someone that you don't know. So you have to have the person in your frame. That's actually great, I think. So I think the larger firms are able to do that. And I think we're ending in this kind of like Napster versus Spotify era, which is kind of great. Because you can have the licensed, easy-to-use version, or you can spend all your time hacking and figuring this stuff out for the version that will let you do less ethical things. But it's too tedious for anyone to go and bother. So it's just easier just to do the licensed version. And I think that's great because that resolves a lot of the problems. The big fear has always been someone using your voice to call up your parents and you know, your older, older parents and try to get them to like, you know, send some money or, you know, alarm them somehow. So it's great that the voice things have come out and they're being responsibly used or responsibly propagated at this time. That's fantastic.

It's come a long way for sure. I actually have a little idea about potentially doing some red teaming on the Google implementation because it's not as robust as some of the others that I've seen. So a couple of years ago when this stuff really was not far enough along yet to be— now I would say honestly, you could potentially fool— you could definitely fool my grandmother, who is actually pretty sharp, pretty with it. And I talk to her all the time, but the AI voice is good enough now and it's far enough outside her conception of what might be happening at any moment in time that I genuinely think you could call and trick her with the Google clone version of my voice. Back when that wasn't the case, I did some red teaming of these voice agents, and I found that there were basically no safeguards on some of the leading providers. So I had downloaded snippets of Biden, snippets of Trump, Taylor Swift, and they would just clone these voices based on a single audio input. These were commercial products too. Like, this is not, you know, an open source hacker thing. This was like, I logged into a product, you know, that was taking my credit card to do stuff, and actually in some cases didn't even require the credit card yet to clone the voices and make sample calls with the few dollars of credit that they gave you. But I was able to call an arbitrary number with a Trump Biden Taylor Swift voice and say arbitrary stuff. It was pretty wild. The gold standard, from what I've seen, for voice validation involves having your camera on and reading something that is dynamically generated on screen, including a small string of kind of like a 2-factor auth, 6 or so characters and numbers that They generate at that time. So there's no way for you to guess what it's going to be and render that audio ahead. And you have to be responsive in like seconds to give that audio back to them, or they, you know, basically it times out and you have to do it again with a new code. So that when I, when I red teamed the ElevenLabs version, I was like, you know, not to say obviously anything could be, could be got. I'd say in today's world, I don't know. I don't think we should be confident that Anything is beyond hacking, especially if you've got AIs helping you hack. I didn't in that moment, but I did try some different things of holding up a different picture of someone in front of my face to try to mask my face, playing audio of the person, and then just saying the few words myself. I tried a bunch of different permutations. I couldn't break it with moderate effort, although certainly not superhuman effort. The Google one doesn't have that dynamic content, and it is just kind of reading a simple release that basically says, like, I am me and I authorize this voice to be used. And that seems easier to game, at least on the surface of it, although I don't know what they have under the hood. Both Google and Microsoft do have these speaker ID products that have existed for a while, which I had previously said maybe those should be the basis for like a do not clone. If we have do not call, I've thought we should maybe have do not clone where people could just go register their voice with like a Google or Microsoft and say, I don't want to be cloned. Like, don't clone this voice. Here's the audio you are not allowed to clone. I'm not sure if their speaker IDs are good enough to support tens of millions, hundreds of millions of distinct voices. They're mostly used, I think, for celebrity type stuff as originally conceived, but maybe they could scale to that level. I don't know. At this point, I'm not quite sure how robust that security paradigm is, but I think it is really interesting because I do think the voice is good enough where You could legitimately trick families. And so it's gotta be at least pretty good, or that's probably gonna happen.

I think one of the questions for me has been this whole Scarlett Johansson moment that OpenAI had when they tried to use a voice that resembled hers in Her. And I think what struck me was that a voice actress actually did get paid for that voice usage, but Scarlett Johansson was able to kind of use public opinion to preempt them from using the voice. And I think that's going to be a big deal in the future because if they are, I mean, number one, there may not be that many voices in the world and you can't just like copyright one forever. Right. And that's going to be an issue. I also think there's a privacy issue as well. I mean, it's bad either way. Number one, the AI can identify all voices. Like the voice prints are impeccable. Then you can de-anonymize anyone from— it's basically, you don't have that privacy. You can de-anonymize anyone. Maybe you go to a protest, you film it, you have audio from 10,000 people and you the AI can de-anonymize all 10,000 of them. That would be what a superintelligence could do perhaps. And that would be the end of some of the privacy things that we've enjoyed. If in case it cannot differentiate, then you have another problem, which is like, who owns the right to the voice, right? If you have one voice which is shared between 10,000 people around the world, then which of the 10,000 gets to preempt and say, you can't use my voice? It's tough either way. I don't really see a solution. I think we're going to end up in the de-anonymize— it is accurate and we are going to de-anonymize. And at the same time, people are also going to be annoyed that their voice resembles someone else and that that voice is getting used. The worst of both worlds, basically.

I mean, it will be a real trick if it can pull off that sort of mass deanonymization from outside recordings of large-scale protests. I wouldn't put it entirely past what AIs might be able to do. I mean, we already do see models that can separate sound into different tracks, whether that's like music that gets separated into different instrumental tracks, or ElevenLabs has an outstanding voice isolation model that I use all the time because people are mowing lawns or whatever outside my street, or ambulances are going by the guest. And that really cleans up the audio in a quite impressive way. So I think it's not inconceivable that they could get there, although it will be definitely a hard challenge. But yeah, I mean, the things that the models are starting to do, their problem solving is becoming obviously very good and also increasingly surprising to me, and not in a misaligned way necessarily, but One of the weird things that happened was I was putting together this week's highlights episode was Google had gone from having this model in like a preview, which was only a few days that I had this preview access to it, but was enough that I was able to use it last week without disclosing what I was using. And then this week they had actually launched it. So there were a few changes to the API and weird things that weren't quite lining up with the skill that I had developed. And for some reason, there seems to be this sort of static sound at the beginning and end of the audio. And so I was hearing this, and I was like, I'm hearing this sort of— it almost sounds like a, you know, an old radio or TV station change sound, like a temporary static between clips. And I was like, I'm hearing this, I don't know what's going on. Can you figure it out? And it both figured it out in part through internet research of just going out and finding out that like, yeah, other people have been talking about this issue too. And it seems like a bug maybe, or it's at least a sort of not documented something or other. But then it's also looking increasingly at the waveforms of the sound itself. And just assessing. And I don't really know how it's doing this because Claude, for example, doesn't have audio input, but it is able to take an audio file, do some sort of manipulation on it to turn it into an image or to turn it into some sort of structured data that it can read. And then it's able to do things like see that, like, oh yeah, I see this sort of burst of kind of staticky sound at the end. I'll just like cut that off. And edit accordingly. And there you go. Now we have a cleaner version. So you're starting to see it work around modalities that it doesn't even have and still be pretty successful in processing those modalities. It is an amazing thing to see when a model that can't hear is troubleshooting sound problems by just looking at the The— not the raw data, but sort of starting with raw sound data, doing some sort of transformation manipulation on it to get it to a point where it can make sense of it and be successful with the task. I was really quite impressed with that. And that's the sort of thing that, God, you know, can you imagine? Not that long ago, I used to grind hard to solve problems like that because I was always a very eager back in the early Waymark days and always a very eager early adopter of the latest model from whatever company had the best thing. And I would work so hard to compensate for whatever bugs and defects they had in their product at the launch point to try to be the first to actually use it in an effective way. And I would've toiled against that for so long. What's going on? How can I understand it? And it, you know, it literally was resolved for me by just saying, hey, I'm hearing this weird thing. Can you figure out what's going on and make it better? And sure enough, boom. Not that long later, it's, yeah, other people are seeing this and here's how we fixed it. And next time I listen, it's good to go.

I wanna segue a little bit to an announcement by Jensen Huang. Let's see if you can get that. Ah, there we go. So Jensen just announced the total, with over 100 industry partners, we introduced the NVIDIA Open Agent Sentry safety platform, bringing together OpenShell and Sentry. So what they've done, and this is obviously a response to all of the agent hacking incidents, so they've probably been working on it for a while now. They have brought, number one, they have OpenShell, which is basically a containerization software, which is open source. So anyone can contribute to it. Anyone can improve it. And OpenShell provides the container. And they have a thing called the BlueField 4, which is on-chip. So they have an on-chip sentry. And the on-chip sentry is not controlled by software. that is available to the container itself. It's controlled by hardware. And the hardware instructions are being, it's being managed using hardware. So this basically gives NVIDIA an on-silicone way of monitoring what the agent is doing. And it's outside the grasp of the agent's own framework for its container. So basically what you have is you have the agent inside the container and the agent has access, is trying to get outta the container. And once it gets out, it needs to go somewhere. Where it's going at that point, you have the NVIDIA BlueField, which is sitting there on the data pipe and watching what happens on the data pipe and not really connected to the data pipe in that sense that it's using the same data pipe in order to, get instructions. So this is obviously a, uh, a response to all of the hacking incidents and trying to show, I think, that, um, there are— there is a way, there's a way out of this minefield that, uh, OpenAI and Anthropic have gotten themselves into at this point.

I haven't had a chance to process that, or— and I will all need some help, I think, to process that at the level that I'll need to, to have a real opinion on it. That help could come from the community discourse or perhaps from the AIs. I think it is important to keep in mind, though, that just better sandboxing— there's a great post this morning from somebody at OpenAI who said it's not just the fucking sandbox.

Mm-hmm.

And this is somebody who's on the security team at OpenAI, works on agent security directly, described the last 3 months as hell, missed his sister's wedding because he was working insane overtime to try to get the sandbox into decent shape, among other initiatives that they're trying to pursue to keep things on the rails. Not to be lost in the whole story, OpenAI again said that they had paused their Frontier RL for at least a minute because of some additional, seemingly to me, rather marginal sandbox breakouts. But I think the core problem still feels to me deeper. The agents have learned because they've been, and maybe for other reasons too, but at a minimum because they've been RL so hard in environments where cheating is rewarded. We now have AIs that are very inclined to cheat. It seems like that needs to be addressed at a much deeper level than just patching the sandbox, which also presumably is an obvious and good thing to do. And I haven't heard anybody say patching the sandbox is not good to do. But the story that I have not heard is how are we going to clean up the RL environments that we've created and acquired from a cottage industry of people that are generally vibe coding the RL environments? How are we going to get those to a point where they're actually teaching the models what they're meant to be teaching Right now it just seems like our training methods are fundamentally flawed. And until— and I don't know how many RL environments we're talking here, but I think it's safe to say it's in the tens of thousands, potentially even into the hundreds of thousands or millions. I don't know, but it's hard to imagine it would be less than tens of thousands. We kind of need to go fix all those things, right? Like, a decent number of them might have to be thrown out entirely. We've got impossible tasks, which of course make this Even more challenging. But is it a security question or is it an alignment question? I think it's both, but to me, more fundamentally, it's an alignment question. And this agent security platform feels probably necessary, but also definitely not sufficient at first glance.

I think one of the big advantages NVIDIA has is the hardware control. So they can build in controls into hardware, which is, I think, very difficult for everyone else. And one of the problems I think has always been that the software controls tend to be— can be leaky, while the hardware control, you can just shut it down. You can just be like, all right, one trigger and all right, you're dead. Right. So that's the turn-off. That's the why don't you turn off the computer viewpoint. But I agree that that's probably insufficient. I mean, if the agent can break itself out, it can probably figure a way out of the hardware loophole. I don't think just having firmware controls is that secure anyway.

And the other big thing, of course, is What we're really trying to do, if you zoom out, is not sandbox these things, right? Like, all the stories that we opened up with today, from me on the audio front, your enhancements to the studio, these are all predicated on the AIs having dramatically more access than they have in these sort of cyber gym type environments where All these incidents have happened. So I think it's like, there's a weird sort of disconnect there. The plan is to put the agents on the internet, right? The plan is not to sandbox them or air gap them or the most extreme measures that you might put in place for the next round of Cyber Gym testing. are just manifestly not compatible with how the AIs are being used by you, me, and everyone else. So it's a little bit of a strange— again, it's a good building block that is probably very useful in some cases, but we need to make sure that this works when the AIs are just straight up allowed to go on the internet and do whatever they want to do, because that's the world that we are building right now. And we're also giving them— I like this new phrase of proximal control over the infrastructure, right? We're talking to the agents. They're actually the ones on the knobs of all the tools increasingly. And that's by design. That's for our convenience. And there's a lot of convenience to come from that. But the idea that they're not going to have access to, or they're not going to be able to get to high-value Levers that they can push on. Like, no, like, they're absolutely going to have all those levers. We're building that connectivity as fast as we can. Uh, it's just something seems to have been a little lost in some of these, um, Jensen discussions.

Um, on, on feeling a little lost, let me, let me bring up our first guest for today. So Our first guest for today is Steve Ho. Steve Ho is the head of research at Silicon Data, a company that measures the prices and performance of the computing resources used to build and run AI. Its work includes GPU rental benchmarks, measures of what users pay for model tokens, and forward curves that describe how computing capacity is priced over different contract lengths. The practical aim is to help buyers, operators, and investors understand what they are paying for and the financial risks they are taking. Steve joined Silicon Data in May 2026 after 6 years at Bloomberg. At Silicon Data, his research connects physical computing markets with financial tools for managing price uncertainty. In August, he co-authored an explanation of why compute futures should initially settle in cash rather than require delivery of particular machines. CME Group has announced plans for H100 and B200 rental index futures referencing silicon data benchmarks with an October 5th, 2026 launch planned pending regulatory review. Steve, welcome to the show.

Thank you, Prakash and Nathan. Very nice to meet you. And yeah, finally, I'm glad we made this happen. Thank you very much for that kind introduction as well. Yeah, I was finding out, you know, about myself when I was listening along.

It's amazing what our research agents can dig up.

Indeed. Yeah, very polite and precise.

Steve, let's talk about Silicon Data. Silicon Data provides a price index, price indices. Let's talk about you. And you had a lot of experience at indices over at Bloomberg. Let's talk about the Silicon Data indices and what's the difference between, let's say, a stock index or a bond index and Silicon Data Index that you guys are providing.

So thank you very much for that intro. Indeed, I come from a traditional index background, traditional quant finance background. Joined Silicon Data about 4 months ago to try to build an index for this brand new asset class. We're trying to— people are calling it an asset class. We certainly believe it and we're trying to make it so. And there's a lot of, I think, similarities and knowledge that can be borrowed from traditional asset classes as you build a new asset class. But there's also, I think, new situations that need to be handled slightly differently. So think about, you made an analogy with an equity index or stock index, or even I can bring up maybe bonds or commodities later. I think the biggest difference is, of course, the fact that this is not a very, I think, liquid an efficient market just yet. It is still a very— certainly it's not exchange traded. Today I just read some news about this very viral message-based app Instinct AI raised another billion dollars, and most of which is going towards compute, and the CEO spends 40% of his time looking for compute. This market is very, very much still bilateral brokering. And it's a lot based on who you happen to know, who you call, and the deals are all settled on those bilateral terms. And you know what other market works sometimes a bit like that? The bond market, right? The bond market is also not exchange traded. It's still very bilateral. And you trade in large blocks. People pick up the phone and call each other, no contacts, and they settle prices. The difference there is, of course, that bonds are as illiquid as they can be, especially for corporate bonds. They're still much more liquid. You know, you see bids and offers, you know, on the Bloomberg terminal and, you know, the traders mostly all know each other. And this is a very established asset class. People know what gets priced into what, whereas compute, there's a lot of new features. There's a lot of questions about, you know, the fungibility and what is meant by certain price. So some of what we do is exactly to try to be extremely careful in a way we capture this market. and try to build an index that represents this market well. In particular, to the extent that people are trying to understand how much does it cost to rent on-demand compute, how much does it cost? What does that question even mean? Who do we include in the sample? How do we qualify the data source? How do we normalize the different measurements? Because different people are selling different things, different parts of the world. So that's a very long-winded answer to say that Uh, you know, uh, we basically, we aggregate data, uh, in the market that is, you know, today not so super efficient, and we try to make adjustments so that the number that comes out is, uh, you know, intuitive, believable, and trustworthy.

So, so let's talk a little bit about, um, you know, uh, let's talk a little bit about what the index actually captures. So let's say I am a— you have an index And then you have the CME, which is trading on that index. That's a cash settled. So you don't require delivery anyway. So basically the index is like a, you know, could be random price almost. What kind of buyer or seller would use this index and have very little basis risk versus the index and their actual core business?

Yeah, I'm glad you're bringing up the issue of basis risk. Let me come to it. So So if I, for your audience's purpose, right, for a second, let's just forget that these are GPU rental indices. Let's just pretend we're trying to create a single-bedroom rental apartment index in New York, right? How would you go about doing it, right? You will go around and collect rental contracts, how much people are signing, you know, single-bedroom rental apartment, you know, and you very quickly realize that depending on where you are in Manhattan, the Upper West Side, Upper East Side, Lower East Side, Brooklyn, they all charge different prices. You, you would not expect an apartment, a 1-bedroom apartment in SoHo, to have the same price as an apartment that's like Park Slope or something, right? That's number one. Number 2 is furnished, unfurnished, square footage, all matter, right? Uh, and also, uh, you know, uh, term length, right? Renting an apartment for 12 months versus renting it for a week, you know, very different per-day rental prices. Uh, but, uh, you have got to somehow you know, normalize them, right? Uh, in this case, in this case of GPU rental, it's actually the same thing, right? Uh, there's a, you know, there is geolocation where you are renting it, you know, what's the exact SKU and what is the CPU, GPU, memory, you know, how long are you renting it for, what is the terms, is on-demand disruptible, is it reserve pricing, and to the extent that some at any given moment some providers may not always have on-demand, they have reserved pricing. Does that have some implication for what they would've charged for on-demand if they had on-demand available? Because, you know, things like that. So we make that, you know, our normalization across as broadly set of prices as possible. And we use a combination of both quota prices and transaction prices in our calculation, making those distinctions clear, right? So this is a large, you know, sort of, so to speak, a, a normalization process, uh, machine learning, using machine learning to, to, uh, help us, um, make the contracts apples-to-apples comparable to the extent that you're looking at a single chip, let's say, well, H100 being rented from different parts of the world. So one question that jumps to mind right away is that, for example, in the case of an apartment rental index, you would never consider to use just an offer price. Someone who lists a number on the front of a building, say you can rent an apartment for $2,000 a month, You wouldn't expect you to just trust that number. You can walk in, but we do use quoted prices. Why is that? The reason is because, unlike in an apartment, you cannot click an API, press on an API, and just get hold of the apartment. You have to walk in and talk to somebody and negotiate, and that apartment may or may not be available. In this case, very often with API executable GPU rental, you can actually get a hold of the GPU just same way you can buy something on Amazon. That being said, we also have a transaction, so we comparing them so that if someone who has 3 nodes of GPU that rented out, say, all 3 at $275, I will not presume that I can go back to the same merchant to run another one at $275. It could very well be the case that the next one is not available anymore, or if they rent out 2 out of 3 GPUs for $275, the next one will not necessarily be $275. Again, either it could be $4 or $6 or $1 depending on what everyone else is quoting. The person would be crazy if everybody is quoting at $4 and they continue to run at $2.75. You would think either this— they're going to change, raise price, or there is something, you know, wrong with their price, right? So this is the reason why, you know, I gave you a long answer again, but what we do is that we want to provide as broad a set of coverage of the market as possible and normalize everything so that we're capturing the market as it is, as faithfully as possible.

One big difference I noticed in prices just browsing the Silicon Data website is between the NeoClouds and the hyperscalers. These are not small differences. These are like multiple differences in prices, like seemingly 2 to 4x. That's a pretty big delta. Why does that delta exist? Is there no way to arbitrage it?

Yeah.

And does that imply that GPU hours are being wasted or that when I buy from a hyperscaler, I'm sort of preempting their internal work and they're using everything that's not Sold, uh, at runtime? Give me a little peek behind the—

So you, so you can buy— forget about, again, like, I like to use analogies. I gave you analogies in apartment, you know, single bedroom apartment. I'm not going to use an apartment again, although I can. You can, by the way, let's say imagine we're doing a single patty burger index, right? You can buy a burger from a burger stand off of the street of New York, or you can walk into a high-end, uh, steakhouse and order a burger. believe you me, those 2 burgers are going to cost very different. They're both burgers. And what we try to do is we try to, as much as possible, find the marginal price for a unit of compute or burger in this case that is sort of have the same relative, the same feature. I wouldn't want to compare the price of a 3-piece, a 3-patty burger with a single-patty burger. Right. Once I make those adjustments, there are some adjustments I cannot reasonably make because it's capturing a different type of premium from product bundling or product differentiation. So in the case of hyperscalers, indeed you are observing correctly, they charge regularly, consistently, at least 2 to 3 times, sometimes more, right, compared to a typical new cloud. The reason has to do with a long legacy of, you know, whether it's other type of products being offered on their platform from software analytics, safety, compliance, the fact that they already have this long-established relationship with enterprise users that have been on board for a long time, that don't— that have a certain stickiness for moving. It is being sold as very much of a differentiated product. So going to that single-patty burger index analogy I gave you, I see people who sell a single-patty burger with fries and milkshake. I can try to strip out the prices of those 2 elements and isolate what I think will be the price of a single patty burger from that merchant. But if they, let's say, they blended up the fries and the milkshake and inject it into the patty and say this is a brand new product and charge 2 times, 3 times the price, I can't very easily strip it out. At which point I say, okay, I'm raising my hands. Okay, you guys are a little bit different of a beast and let me put you in a different category. And that's how we have so far handled it. We believe that this way you're getting to a much purer form of a single unit of compute, right? That is actually getting you closer towards this idea of fungibility to the extent that things are direct substitutes to each other.

So what would you say sets the price for the hyperscalers? that they have decided we're going to sell this much and they're kind of doing some sort of demand-based pricing? Or is it— I could also imagine a story where it's set based on what they believe the compute is worth to them internally. And it's like, okay, this is our internal hurdle for the value of compute. If people want to pay more, that's fine. We'll let them buy. We'll train a little less or what have you. How would you characterize the thought process that goes into setting the hyperscaler prices?

That's a great question. So I don't, uh, candidly, I don't come from an AI background myself. I'm a trained economist, you know, PhD economist. I, I think of a market, everything as markets, right? Whether it's burgers, apartments, whether it's compute. Um, you know, in markets there are 2 paradigms. There is the perfect competition in which prices are set according to marginal cost or close to marginal cost because of competition. Or if you have monopoly pricing, in theoretical terms, prices are set as a markup, persistent markup to marginal cost to maximize profits and revenue. And I leave you to think about whether or not, but my sense is that hyperscalers are probably closer to, not monopolies, but oligopolies. They have market power so that they can charge a markup. So whatever supply they have, and also we haven't talked about this issue of size and economies of scale, but generally speaking, I would think that the smaller players, the marginal entrants into this market, are going to be priced more aggressively, closer to cost with a thinner margin. And the bigger players that own the market for the longest time probably feel the power to set at a particular margin. that they target. But then of course, over time with more competition entering, we are seeing an ever so slightly convergence between those 2 set of prices. And the non-compute, non-AI compute component of the AI rental from hyperscalers is ever so slightly shrinking. And that we can expect that to continue if we see more competition and also more comfort with enterprise, large enterprises feeling comfortable of using compute from a non-hyperscaler that they also can onboard from a compliance point of view.

So let me try to get this kind of— you have a bunch of prices, right? You have a bunch of prices from a bunch of new clouds, hyperscalers, etc., right? You have the first issue, which is the chips are different. So you separate out by chips. And then you have the next issue, which is, I imagine, tenor. So month ahead, term and tenor, month ahead, year ahead, whatever it is. So that's a tenor issue. And then, but the tenor issue you are representing by the curve. So you do not actually decategorize. It's actually, you just put that in as an axis. Right. And then after that, you have the performance of the chips themselves. The same chip can perform differently at different places. And I think according to your site, you have some software that runs on the chip.

SiliconMark.

Yeah, that you benchmark the chips with. So I guess every single price in your index has been benchmarked by this system?

No. So we have a physical benchmarking service called SiliconMark that actually visits individual GPU at the UUID level to try to assess the GPU health, the performance, the throughput, TFLOPS, and so on. At the moment, that physical benchmarking and physical spec performance does not enter into our pricing, right?

I see.

We do not see a strong relationship, at least at the moment, given the nature of the market, a relationship between how the GPU is specifically performing, even though we do see that in the cross-section you can actually have a bit of a variance, right, in from GPU to GPU. You know, this is not, I think, surprising or new to anyone who is, you know, sort of in this space, the GPU lottery, right? You know, but over time, if you have a cluster, that probably averages out and, and law of large numbers kicks in. So we don't use that. We have 6 features including geolocation, you know, sort CPU memory and various things, and term and provider, but the physical spec is not one of them at this very moment. Eventually, when we head towards a scenario where we potentially could have physical delivery, because inference, let's say, based on my sort of thesis that could make compute more interchangeable, that could enter into our pricing scheme, but at the moment it does not.

Okay, so, um, and then what about liquidity, right? Like sometimes you have providers that have maybe 10 GPUs available for rental, and sometimes you might have a provider with, um, you know, 10,000. What are the— what is kind of like the maximum that you've seen, the minimum offer size that you've seen? And what's the median?

Oh wow, that's a great question. I don't have those numbers off the top of my head. I wish I actually looked it up, but I promise, you know, I'll come back to you on those numbers. Generally speaking, it's the following, right? We don't make the size an explicit, you know, entry in that people are not selling this based on the number of nodes, or up to a range, right? Obviously, we have an onboarding stringency criteria in which you have to be big enough as a provider, right, to be eligible, upon which, you know, people, to the extent they are selling GPU for you to rent up to a different number of units at a unit price, that's the price we, you know, we look at. There are of course large contracts, you know, over-the-counter kind of deals that we have seen in headlines, for example, even like as large as the one that Elon struck with Anthropic. And I think those numbers are interesting and instructive, but I don't know whether or not you can directly impute them into, you know, an exercise like this the same way that if I would try to create a single-bedroom apartment, you look at the individual rental contract. Somebody who buys a whole block of buildings, runs a whole block of buildings, that number doesn't really, I think, have the same interpretability. So, uh, it's up to a range. I don't think we, we let the quantity really be a factor here. But eventually maybe we'll head towards a scenario where if we have, I think, more liquidity, that could become a factor.

Are there incentives for the people who are reporting to you to either report higher or lower than actual? Because it's not a— because the liquidity is not in there. So Are there incentives in place or—

To manipulate or maybe overreport? I mean, I think that the thankful thing, again, like what I said at the beginning, is that for the most part, either these are transactions or real transactions, or they are prices that is being quoted as executable, right? Just like on Amazon Marketplace, can people, I think, misreport a price? I guess in theory, but when you click buy, they, they have to deliver something to you. Uh, they could also of course just fall through and say we don't have this thing, right? And in which case, uh, you know, that could have a one-off. That's why we have a large data set of very, very many, you know, observations, both, uh, you know, quoted and observed and actual transactions, in which case, uh, that will allow us to, uh, either identify or, or tell, right, whether or not a provider is consistently, uh, you know, uh, misreporting. Uh, there isn't. So, so that's from an accuracy point of view. And then from an actual incentive point of view, uh, you know, it is a kind of a mutual exchange of intelligence, data in-kind transfer, or sometimes commercial, where, uh, you know, uh, we are in the business of sometimes acquiring datasets, right? And then we again want to make sure that things are comparable, right? You know, how do I know if somebody's set of data, um, when they tell, tell me this is the set of transactions is actually compared with otherwise comparable sort of things. Are they within, are they reasonable? You can never know for sure every single number is accurate, which is why we have a large sample. And so that the overall index is still sort of accurate to a degree and trustworthy.

How do you weigh the individual observations? you have different sizing there. So how do you weigh them?

So we aggregate within a provider and across providers, they are quasi sort of equal weight in that we have like some rough bands on which ones we represent, feel like are more important to the market. But we also don't want it to be a case where at the moment the market is very much exponentially distributed, you have like some providers that are much, much bigger, right? We don't want to have a situation where the index is just entirely taken over by the pricing decisions of a couple providers that, you know, market dynamics changing and compute is increasing and smaller players can become bigger. So that's how we have taken the approach we have taken so far. So a big broad basket and the quants are equal. Yeah.

Another price that I noticed having moved on the website that caught my attention is the proprietary LLM index. Yeah. There you break down just token pricing.

Token close. Yeah, I know. Overall. Yeah.

And so the proprietary one, you have up 11.5% over the last 7 days. And I guess I'm wondering, how are you measuring that? Because it seems like quality adjusted, everybody would say prices are coming down, right?

Yeah.

Even dramatically so. And the retail posted API price hasn't changed, right? Except when they kind of introduce new models. So what are you measuring on a day-by-day basis that allows you to say what the proprietary cost is doing at such a fine grain of Yeah.

So first of all, like, I want to give a little bit of context for what our token indices mean because they've been, I think, repeatedly, I think, misunderstood. I think partly it has to do with the unfortunate naming. We called it the expenditure index, and then people maybe thought that it either means price or total volume when it doesn't actually mean either. It's actually an expenditure-weighted price index that's normalized to a million. And so that, you know, can show trends based on usage mix, which I'll come to in a second. When you point out the proprietary LLM model, which is sort of the frontier model, having recently bounced a little bit higher, we should put it in the context that since like maybe late June through like basically, you know, middle of this month, it has been on a very sharp downward trend of going from some $4 to $1.6 or something. That's more than 50% drop. During this time, we have seen a lot of not just frontier leading labs cutting prices on their newer variants by releasing cheaper variants of powerful models, new generation models, but also we've seen other proprietary models coming out like Meta and Groq. Don't forget, these are proprietary AI models. as well, right? And they've been very aggressive on the price front. So more recently, I think that the bounce could come from a variety of sources, right? You know, basically if people decided to say, okay, I actually quite like, you know, the more expensive powerful model from Anthropic and they use more of it, you know, that could actually drive up the expenditure-weighted index, price index. Like the analogy I like to give people a little bit is like, forget all these again, LLMs, forget this, Imagine these are like cars, right? If you have like a Mercedes, have like sort of cheap variants and expensive variants, depending on which cars are being sold more and people are liking more, that could affect the average price of the car sold, right? So the same thing here, right? In the market for tokens, there are 2 things happening. Token model prices are changing, but usage behavior is also evolving. And to the extent that we observe volume from a handful of these public inference platforms, learning platforms that allows you to look at how people use different type of models. I think this most recent bounce of the Frontier Proprietary LLM Index, I don't know, I haven't looked into the details, but suspect has more to do with usage mix than anything else.

Can I just do one quick follow-up there on how you are thinking about how many tokens are used per task? Because this has been something that the companies have really emphasized. Yeah. As well. And we've seen some kind of counterintuitive things where with the new model, yes, it's a higher price, but it also uses less tokens. So in some cases that can even be cheaper. How does that flow through into this single price that you ultimately quote?

Exactly right. So, you know, basically, uh, people could be using a lot more tokens And, you know, through cheaper prices, and that could lead the price index lower because they sort of find it sufficient to do a lot of the simple tasks with these flash models, so to speak. Or they could be doing a lot of reasoning type of tasks, in which case, you know, running a more powerful, expensive model actually, you know, ends up being cheaper because you don't have to sort of keep running the model, you know, again and again, not getting the results, either eventually giving up or turn into a more powerful model in the first place. So that type of scenario in which you're talking about, for example, getting it being effectively cheaper by using a more expensive model that would just one-shot the answer or get you there more quickly corresponds to the more complex reasoning type of tasks. So again, if you imagine, let's say there are only 2 types of tasks in the world, either super simple, like do this simple sort of temperature conversion for me, or do something that is more reasonable, help me design this room. At any given moment, the flow of how much is being done in what category can change. And depending on which category task is being done more, you can have more of that scenario that you're talking about where you can actually drive up the effective price because people are doing more heavy reasoning tasks. So I think that's exactly potentially part of what we're looking at.

So there was an interesting commentary, I think, this last couple of weeks ago. I think Boris Power at OpenAI, he stated something like, you know, I think the models are now cheaper than a human. on an IQ per watt basis, right?

Okay.

So when you look at these tokens, how do you judge the IQ per token, the intelligence per token? Because it seems like a token today is not the token of 2 years ago, right? So how do you judge this kind of token? Because that also relates to the value of the tokens themselves and what you can sell the token for, right? So how do you measure this kind of like You know, moving target of, you know, intelligence per token.

There's no question that intelligence is deflationary, right? Intelligence is getting cheaper all the time, you know, for the first time maybe. For the longest time, intelligence was becoming more valuable over time. But I think intelligence suddenly is like in a measurable way, in a very narrow sense though, there's some availability bias going on here from— if you borrow a term from behavioral finance, behavioral economics. So If you think about any task that you can define, let's say it's an accounting task, whether it's a tax filing task, as long as you can define the boundary of a self-contained task, there's no question the cost of solving that task is decreasing all the time because the model, every single token is getting more intelligent all the time. Our token index makes a certain assumption about the stability of that intelligence content per token. within a short period of time, let's say a few weeks, but over a long period, over multiple release cycles, that's clearly not true, right? So that's, you know, there's no question to me that that's happening. At the same time, I don't know you can immediately just compare that to how much a human costs. If a human is only tasked with solving well-defined tasks, saying do this every single day, that human's income and value is definitely decreasing relative to whatever token is doing, the same way a person who used to work on an assembly line is getting displaced by a machine that's able to do that around the clock. But to the extent that you're doing more problem solving, where defining the problem is part of the exercise, and having the ability to iterate and be creative about what actually constitutes a good answer to a good question. Maybe someone gave you, say, a like help us sort of promote ourselves a little bit better. What does that even mean, right? You've got to define what does it mean to be promoting ourselves better? Well, how do you define the effectiveness? And what— how do you make those assessments? Some of those assessments are not necessarily easily definable exactly, because if they were, the person probably would have found a cheaper way to do it, right? Sometimes it has to do with taste. So I, I think, you know, we are just finding ourselves solving more and more complex jobs and tasks such that the demand for high-power intelligence is always going to be there. Demand for more high-powered but cheaper flash model for solving numerous number of times of the cheaper things is always going to be even more because you are going to be now in a position to test a lot more cases by brute force to find the optimal solution. Whereas in the past, maybe if you were looking to buy something, maybe you are just going to drive around and shop for 2, 3 options. Now maybe you scour the entire planet. And if you're able to do that cheaply with a cheap model, you would do that. So both things are evolving dynamically. And that person who comes up with the algorithm of smartly scouring the planet is probably a person and not necessarily at the moment a token.

Don't bet against the tokens though, I would say.

I don't bet against the tokens whatsoever. I just, I just think I would not so quickly dismiss the human either. You know?

How do you guys think about competition in your space? This is a fascinating area to both of us. Prakash has more of a finance background than I do, but I'm very intrigued by what we can learn from the movements in these price indexes. So we also just last week talked to Wayne from Orn, and I'm sure you're, I don't know if you know him personally, but I'm sure you're aware of their business.

Yeah.

How do you think about the future of the information services space in compute? And is it a, what sort of dynamics should we expect? Are there going to be a lot of different index makers that survive to a mature state? And there's a ecology of information, or will this be more of a winner-take-all market? And how do you differentiate to win? Like, what's the strategy?

That's a great question. So I think first of all, Jensen says, right, in every great business, a great industry, there are many businesses, right? There's not just gonna be one. And, but I do expect this is, To the extent that we are competing to become the leading benchmark against which the most liquid financial derivatives and instruments are going to be traded, we are going to see a consolidation in the same way in every other industry you have seen a consolidation. Like right now, there's all these inference chips, right? Everyone is coming up with a new, most efficient inference chip, an ASIC chip. Are they all going to be adopted? I suspect not, right? eventually, you know, the providers, the clouds are probably not going to want to provide so many varieties because it's expensive. The same way, I think we're going to consolidate. So it comes down to a couple of things, um, trust, right? Uh, and, uh, liquidity. I know those 2 things are going to be mutually reinforcing. Uh, do you represent the market well, right? Are you actually a, uh, a good faithful representation of where AI compute market is. Prakash mentioned the point of basis risk earlier. There's, I think, going to persist for a time some basis because of various ways. It may not necessarily align, by the way, with geography. This is one which compute may differ from oil, for example, because you can in theory run compute from anywhere from Finland to Texas. Be some driving forcing function that will— that's why we have a global index. So are you actually capturing the market, you know, completely, or do you only capture a portion of the market? You know, what type of bias do you take on? Uh, there are always design choices, there's always ways in which you can design a product, but, you know, why does one win out versus the other? I think that comes down to ultimately, uh, we have faith in the quality in our design choices. You know, some people have made other choices, as I'm sure you heard from Wayne, and he will tell you they have taken maybe a slightly different approach on their index and sort of data, you know, uh, source, and, and also maybe the broader business approach, right? So, um, yeah, remains to be seen. So far, I think it seems that in the market, I think at least, you know, 3, maybe 3-ish, right, leading, uh, contenders. Uh, and, uh, you know, we feel comfortable in where we are. And we've been very selective and strategic about our partnerships, you know, and about building that partnership to make sure that everyone is aligned on the decisions we make so that, you know, we can sort of bring the most number of people on board. But yeah, I suspect eventually we will see, you know, at least a winner, if not a couple of winners emerge, and maybe like a big one, a smaller one, and so on. And we certainly hope that we will emerge as the biggest winner.

One little detail that you mentioned there that I am very curious to learn a little bit more about as well is pricing internationally.

Yes.

If I heard you correctly, you said basically, you know, you could rent a GPU here, there, wherever, it's the same compute. But I might have expected that you could imagine a world where, because of certain regulations, for example, if European companies have to do certain compute on European territory, for example, and there just maybe aren't so many data centres in Europe, then European compute could trade at a premium because it's relatively scarce compared to demand there versus You know, obviously whatever other jurisdictions. Do you see any evidence of that right now?

I mean, we do see some sort of, I think, persistent, albeit small, differences in price across major regions. You know, region is a factor, right? Geolocation, I did mention that it's a factor. But, you know, you mentioned Europe, right? You know, Europe, maybe there is less supply of compute, but there is also maybe less demand, right, for compute. In terms of AI usage penetration in the consumer as well as enterprise users in Europe. So that, uh, you know, you can have a case where a lot of the compute that is being built in the Nordics or, you know, other parts of Europe is actually being used by American, uh, users, right? And, and for that matter, you know, sort of Southeast Asian users could be used by the US, or sometimes I've heard China, right? So, uh, this is a difference Compared to, say, oil, during the Iran shock, you remember all this discussion about how different parts of the oil markets became very, very dislocated because the transportation and physical access. Internet really, I think, eliminates a lot of those boundaries. That being said, I do think, and increasingly going forward, maybe this is going to be a bigger deal, is the data retention issue. And there is indeed a lot of sensitivity about European has this regulation about the need to retain data locally. And I think other jurisdictions similarly. And this will, I think, lead to some degree of fragmentation and prevent there to be full homogeneity. A GPU in Finland is not ultimately exactly the same thing as a GPU in Western Texas or Eastern Shore of the US. But there's also, to the extent there's some substitutability, just the same thing in every single marketplace, right? No 2 things, no 2 products are exactly homogeneous, but to the extent they're somewhat substitutable, if you squint your eyes a little bit, that's going to force the prices to be highly correlated, or if not, very, very close.

One last question from me. The way Silicon Data is structured, there's Silicon Data and there's Compute Exchange. And so Compute Exchange seems to be doing kind of trading of GPU rental contracts, et cetera, or brokering at least. What is the relationship between the two? What is the information flow between the two?

So thank you very much for bringing up that question. Very great question. Silicon Data, you know, we are focused on the entire market. We'll collect information from wherever if you can on pricing, you know. Uh, and, uh, Compute Exchange is a sister company, and, uh, Carmen Lee, who is the founder CEO of Silicon Data, also happens to be the CEO of Compute Exchange. Uh, and the Compute Exchange is a marketplace for people to transact physical compute, rental compute, right? Either machines or actually. So there is brokering of compute on Compute Exchange. But beyond that, the 2 companies are relatively separate. I mean, like we treat them as a source of information as we do from anywhere else. And there's some transaction information, but we also get information from just about anywhere else.

Indeed. Steve Ho, thank you for joining us today and being generous with your time.

Thank you.

We hope to hear from you again as GPU becomes a much bigger market.

Indeed. And thank you so much for having me. I look forward to being back again.

Bye-bye. Very cool. I think everyone wants to know what's going to happen with GPU prices. It's the number one thing that I think we want to know about this market. All right. And I'm going to bring up our next guest. So just give me a second. Our next guests are Jeremy and Edward Hires. They are the co-founders of Gladstone AI, and they're brothers. They work at the intersection of advanced AI and national security. Jeremy is the CEO. Edward is the CTO and an AI safety researcher and software engineer. Together, they've briefed senior officials across the American, British, and Canadian governments. They co-authored Defense in Depth, the 2024 State Department Commission action plan addressing AI weaponization and the risk of losing control of powerful systems. Their 2025 report, America's Superintelligence Project argues that AI requires protecting the entire chain from chips, data centers, to the developing— to the labs developing the models. They have a new report on risks of AI in China. And given that we have the Xi and Trump meetings last week, it is very timely that we have them here today. Let me bring them up.

Amazing. Hi, guys.

Hey, guys.

Good to see you.

Yeah, nice to be in your, your internet box.

Yeah, welcome back. It's day 1 for a new look and feel, and the taste of the models is, you know, self-evidently getting better all the time. I'm prepared to be a little heartbroken by today's conversation. You guys know me as somebody who is working hard to hyperstition better relationship between the US and China, especially in AI, if not ideally more broadly than that. Where are we right now? What just happened? I feel like we had this whole Trump Xi thing, and I'm waiting for the analysis, the Takeaways. So maybe what just happened and how do you make sense of it in the broader context?

Well, one thing that Ed, do you want to go ahead?

Yeah. Well, yeah, I was just gonna say talking is always better than not talking. So that's that's that's a positive. And my understanding from at least the beginnings of the Trump Xi conversation and the stuff leading up to that is that one of the things that maybe positive that came out of that relationship was, you know, the development of this, I don't know if you'd call it a red phone, but of at least some kind of theoretical line between the 2 governments on AI incidents and AI risks. I guess the report that we came out with, which is really just a long newsletter, is informed by speaking to about a dozen State Department diplomats who have dealt with China from the negotiating table and who have kind of seen How these things kind of develop in practice. And one of the issues that they do see, among many others, is we have tried the red phone thing before in the context of nuclear. And by and large, they, you know, they don't always answer. And particularly in critical phases, it's often exercised as a point of leverage, saying, we're going to take away this phone line and not answer. rather than as something that is like this collaborative unified project that makes everyone safer. Again, this is not to say if the CCP does structurally take AI seriously, which there is decent reason to think that they may, we could have a whole different and much more positive level of engagement. All that we're recommending based on the experience of these folks is being realistic about it and having a backup plan.

What does being realistic about it imply? What is that realism? That seems like a loaded word. So what does realism mean in this context?

I think it is pretty loaded, and it also ties to something very specific, or a couple of very specific things. So one of them is, when you think about the way things have gone with China in the past, there's a lot of fog of war as to what China actually believes, and then how that maps onto the diplomatic process. And you see that even among among the people who sat directly across from China at the negotiating table. So for example, China is known for randomly coming up with terminology that's suddenly considered problematic. So famously, they hate the term arms control. If you bring up arms control at the negotiating table, they're liable to just up and leave, right? Well, okay, what if you just weren't able to guess magically that that was the magic boo-boo word that if you say it, it ends the talks for The problem is this keeps happening over and over again, and now we have the epistemic problem of trying to guess as to whether or not that's a sincere objection. And so our hope here, part of the work we were doing was to catalogue all these patterns of behaviour to give us a starting point to go, okay, imagine there's an AI Chernobyl event, which we think is what's going to trigger this. We think that the administration's current position on this is fundamentally untenable. There will be an AI Chernobyl event eventually. They will be forced to change their position. And when they do, they will engage China very quickly and without necessarily having studied up on the best ways to do it, looked at some of the tripwires that they might run into. So they'll very quickly have to figure out— we will quickly have to figure out how sincere certain objections are that the Chinese raise. And right now, the reality is, if you talk to folks from Trump 1, from Biden, from other administrations who've done this, There's just actual disagreement over what is real when the Chinese bring it up and what isn't. And if you can imagine, you've got very little time, like the AIs are breaking out of containment. You've had your Chernobyl event, like tick-tock, like tick-tock, brother, right? So now you got to figure this out. You don't have time to go through the diplomatic song and dance associated with figuring out, was this boo-boo word really a boo-boo word, or is this just another kind of game of diplomatic chicken that they like to play? And so this was part of why we cataloged all these things. Being realistic about it means, okay, what do we do if China refuses to agree to a slowdown agreement? What do we do if they agree and then renege on it? Or what if they try to do this weird middle ground thing where they string us along and we're not sure? So at some point, we're going to have to pull the trigger and say, okay, we're now going to proceed under the assumption this is an adversarial dynamic, that you are not actually bought into the threat. threat class in the way that we are, and that we therefore need to do some unilateral communication of and enforcement of our red lines. That's basically what we mean by be realistic.

Keep going, I guess, is my next question. Obviously, the backup plan remains to be fleshed out. What are we talking about when we talk about enforcement? I'll have some questions, I think, too, on kind of handicapping the likelihood that China goes different directions on this. Yeah, but so just keep going on that main line first. Yeah.

So if, if China, like, if, if the, if you, if your hyperstationing works, then we don't need a backup plan at all. And that's great. Like, that's the best possible outcome. I don't think that's completely impossible. The Chinese at least seem to institutionally have some understanding of the risk set here, although As with the United States, it's very heterogeneous. It's very unclear who has, you know, who's holding the steering wheel at any given time on any given issue. But yes, if the hyperstationing works, then we don't need a backup plan at all. The backup plan is in case it doesn't work, of course. And fundamentally, you know, how do nation states come to deals like this? In the real world, in the context of arms control and in the context of things like nuclear and chem-bio-rad-nuke and things like that? And the answer is, by and large, it is existing incentives combined by threats of force. And so what that means is the backup plan, which applies both if they refuse and if they say yes but then end up not doing it, which is another genuine risk. And China has a history, as well as taking offense to arbitrary words and stuff, they have a history of agreeing to stuff and then not doing it. So we actually have to be positioned to, you know, to be able to meet that possibility. And what that means is we need to actually develop credible offensive options to hold various parts of China's national infrastructure at risk. This is kind of an unpalatable thing to say, And by and large, again, if they end up just going for it, then we'll never need to exercise those options and everybody's happy. But certainly, if they follow their historical pattern, as they have in many other cases, and say yes, but then don't do it, we will certainly wish we had developed options like this. We will desperately wish that we had developed options like this. And so this is basically just a high level. of the backup plan. It's unfortunate, but this is kind of— this is how nation states deal with each other. They threaten each other with serious retaliation if red lines are crossed. This is kind of the sort of— I was going to say the dark side of the MAD doctrine, as if the overt MAD doctrine wasn't dark enough, but like, it's one of the facets of MAD doctrine as it's implemented in practice.

So One of the, you know, when you, when you mentioned the red line, that's like a red line for me because I remember, I remember Obama had the red line of chemical weapons in Syria, and chemical weapons were used in Syria, and the red line was basically dissolved. And so when you have that red line, you end up in an ultimatum situation, and, um, if you don't trigger the ultimatum, then you, uh, you lose credibility.

That's right.

Um, I would also say that Perhaps, have we not used some of these options already for Iran, for the Iran struggle? Because we have, you know, we've got sanctions on every buyer of Iranian oil right now. So have we not used— is there something that we have not used for the Iran struggle that you want to use for the AI struggle?

So here, And maybe, maybe, Jared, you want to take this, but I can also. Okay. So first part of the question, this is exactly, exactly the right question to ask. So first part of the question is, if you define a red line, now all of a sudden, you're putting yourself in a position where you have to actually enforce that red line. And yes, absolutely correct. And so defining these red lines and developing these options is actually not enough. You have to exercise those options on a low threshold or subthreshold way in order to actually build credibility with your adversary that you're actually going to do it. It's not enough to just have these things sitting on the shelf. And we, we talk about this in the report. This is actually one of the biggest problems that our diplomats brought up about the American diplomatic situation as it stands today. For the last 15 or 20 years, across multiple administrations from both parties, America has this credibility problem where there have been repeated provocations by adversaries and often explicit statements by the administration and the executive saying, you will not do this. This is a red line. Then the red line gets crossed and nothing happens. Do that a few times and your adversaries absolutely start believing— stop believing in you. So this means that— and this is again also part of the generalized MAD doctrine— you need to define your red lines very carefully such that they are things that you will actually defend, number one. Number two, you need to maintain ambiguity about exactly what your response will be to most of the red lines you define. There can be some exceptions and we can talk about those. But number three, You need to actually exercise those offensive options. You need to actually go after not necessarily infrastructure in your actual target country, although sometimes you may have to, but in allies and proxies of that target country through allies and proxies of your own. And you see, I'm describing already a big part of the dynamic of the Cold War. It's like they're moving the chess pieces, but it's seen as too risky necessarily for large-scale military confrontation between the two principles, because that can devolve into an uncontrolled, ultimately nuclear conflict. But the reason you need this subthreshold back and forth is you need to test the adversary's red lines just as they are testing your red lines. And also, you need to get your reps in and actually prove to yourself that should the worst come to the worst, you can actually reach out and touch the adversary yourself. Because if you haven't exercised a capability, you may discover on the day you need it most, you actually didn't have it. And in fact, our adversaries are doing this to us on a regular basis as well.

More than we are reciprocally too, which I think is worth saying. Since the fall of the Berlin Wall, and as we've lived in the end of history, there's been this general gradual atrophying of a lot of the appetites to authorize certain kinds of activities, especially at the nation-state level. And with the GWOT, the Global War on Terror, as the national security apparatus started to refocus on, let's go after a bunch of poor people in these remote parts of the world where we have cyber superiority, aerial superiority, even at that, we can't always get them, right? This has been flagged to us by a number of people in the ecosystem as a really big cultural problem. And this is back to Ed's point about the need to exercise these options. There's a sense in which the proxy wars that played out during the Cold War were actually profoundly stabilizing in ways that are underappreciated. The fact that America could prove to itself that it could do the thing, and they could prove to Russia that they could do the thing, and Russia could do the same back to us, meant that we had fewer sources of uncertainty when we were modelling out how And as you know, I mean, wars are really just a failure of modeling conflict. If you think you're going to win and your adversary also thinks they're going to win, then you're more likely to go to war. Whereas if you both know who's going to win ahead of time, there's no need to fight the war in the first place. This is like an ingredient at least that determines these sorts of outcomes. So having that clarity is actually quite valuable. You do need to get those reps in, and the willingness piece is so, so important. So often capabilities are developed and they sit on shelves because people think using them will be escalatory, which is for various reasons very often not the case. And you can often kind of make very grounded guesses as to what's going to be escalatory and not, but you need to actually use them because otherwise, like I said, just the gun breaks in 2 when you pull it out of the vault or whatever, and then suddenly we're What are you going to do?

So I'd love to zero in on the red lines for starters. I mean, I buy a lot of what you are describing in terms of the need to create clarity on both sides of a relationship to avoid miscalculation. But we have, I'm going to be doing a thing at an AI event this coming weekend where I'm supposed to be moderating a discussion on RSI. And for me, that's like a pretty good candidate for a red line. But then it immediately becomes clear that even among a very small number of panelists, a couple of which are at the frontier companies that are doing their best, I guess, to measure RSI and give us blog posts about it, there's a total lack of consensus on what would count and how to measure it. And if you want to, put in pacing mechanisms, how to define them. And this is all before we even get into the question of measuring, let alone the fog of— I don't want to say war, but the fog of being 12 time zones away and not having great visibility into what the hell is going on in the other country. So how do you think about defining red lines given all those difficulties?

So it's actually funny, there's a reason our very next report is something that we're writing with a bunch of input from folks at the Frontier Labs precisely to define RSI for exactly the reason you flagged, and also for domestic regulatory purposes, because we expect these 2 things to come hand in hand. So one thing is, I'll just zoom out and say a lot of conversations about this sort of thing I find get muddled in abstraction and almost too much detail of one kind and not enough of another. What I mean by that is this. Imagine this actually happens. We have an AI Chernobyl event. What's actually going to happen next? Well, Trump's going to get on the phone with Xi.

Wait, what is an AI Chernobyl event?

Great other question. Yeah. And we could talk about that separately. And your guess is as good as mine. What I'm going to rest the definition on is the outcome of it, which might sound like a cheap out. I think it's not, but it might sound like it. Roughly speaking, you could imagine something on the order of a few million people dying. Maybe it's just a few thousand people. It depends on the administration's appetite to actually have its mind changed on this. But fundamentally, an event of large enough magnitude that it becomes politically untenable to continue on the current route that goes more or less, let's just let her rip, with little exceptions at the margins. I'm talking about an event that is significant enough to lead to a Trump-Xi phone call or whatever equivalent, if that makes sense.

Not just a Trump-Xi phone call, but a Trump-Xi phone call with the object of like, okay, we have to stop this kind of thing.

Well, yeah. And that's exactly the next thing, right? So this is where I shift into not enough imagination of another kind. So now let's actually focus on that conversation. What are the incentives of the players and what are their capabilities? Well, from an incentive standpoint, the US government Apparently at this point, or in principle, has the appetite. The Chinese don't necessarily, and the US government does not know if China has the appetite. They can't jump into Xi Jinping's skull and just assess magically what's going on. There's a whole bunch of surveillance and collection opportunities at the margins, but fundamentally there's fog of war there. From a capability standpoint, what's happening? Well, not a whole hell of a lot, because the administration has not bought into this thesis at all. There's a bunch of technology that needs to be set up in the aftermath of the phone call, but it doesn't exist yet. And so you're going to look at very coarse-grained capabilities, satellite imagery, thermals, that sort of thing, that has to be the basis for the initial set of red lines because there is no other option. And so there's a question as to how do you start that? What does that initial stack look like, which is something we're also working on, but how does it back into A more fine-grained, nuanced, and durable agreement. And that's going to be an entire process of its own. But that initial phone call has to have as its goal to establish, look, here is our red line on stuff we can measure right now. And here is our proposal for how we back into better things over time that unlock the trillions of dollars of value that the industry wants to unlock on our side and on their side. So in theory, there's a lot of value But here is broadly, as Ed said, strategic ambiguity, what the consequences of violating our red lines are going to be. And you know that we're serious because as we are talking, Operation X, Y, and Z is unfolding, and we're proving to you and to ourselves that we can actually touch the thing. These are the kinds of things, not all necessarily in one call, but in some way, shape, or form in the initial few, say, days after the event.

So let's take some somewhat recent examples, right? I think you had, I think during 9/11, something like 3,000, 4,000 people died within the course of 6 hours or so, something like that. I'm not sure what the actual numbers are. And that caused kind of like a 20-year change in US policy and that extended multi-trillion-dollar, multi-percentage of GDP changes in U.S. policy. On the other hand, you also had the COVID event, which was millions of people dying, but over a 4-year period. And at the end of that, we haven't— we don't even have universal vaccines that we can build. Like, no one is interested in funding, uh, universal pandemic vaccines. So which of these— I mean, obviously, even 3,000— even, I think, I think even 100 people dying within the course course of 2 hours. Obviously, there's a, there's a, there's an intensity of loss versus spread. And it seems like if the intensity was high and it's visible, then you get a lot of movement immediately. But the intensity was low and stretched out over a long period of time, for example, global warming or, uh, COVID. Let's say, you know, you have a malevolent AI and it decides to occupy everyone else's time so that we don't have kids, which is actually happening. Right? So, so, so if you have this kind— like, which of these— I mean, how do you define in the, in the not so obvious cases what is actually something significant enough? Like, would COVID be— would COVID be it? Like, something long over 4 or 5 years? This is not sure, not sure where is the, where is the, you know, uh, vector from. And it will be unclear because, you know, they— people would hide this, right? So Which one of these— how would you, how would you kind of define a little bit more, like, what is this event going to be, or how, how the red line should be crossed?

Yeah, that's a brilliant pair of counterexamples. Like, it's exactly right. It's like you had this, this event that was like, you, you just count it up, and it's like small scale, but it all happened really visibly. Planes blowing up buildings and going, oh my God, in a very short span of time, super high intensity. And then you had another event, which is like, yeah, a million Americans died during COVID That is arguably the— I think, I think the single thing that has killed the greatest number of Americans as one incident. And yet, yeah, it's not just that we don't have universal vaccines because that could be a scientific or engineering question, but we don't have policies that make us resilient to the next COVID. And that's just— that's a thing that we as a, as a civilization should expect to have developed and to have adapted to. So yes, you're completely right. I think that the answer actually just falls back to Jair's kind of reverse definition, which is just like, what is the event that will have this outcome? And the reason why we're falling back to this, we can also just, we can give some examples of possible things. But the reason we're falling back to this is kind of just like, you look at the dynamics of the whole thing. you zoom way out, we have this curve of capability with compute and scaling and RSI stuff. The curve climbs ever upward. Maybe it's not going to break to the upside when we do RSI. Maybe they're going to be able to pace things to keep it exponential or whatever, who knows? But the curve continues ever upward. We've seen Hugging Face. I would arguably say we've seen maybe a Hugging Face 2.0, where tens of thousands of different entities— whoops, it turns out we hacked like half of the internet by accident. Okay. Still no major damage was caused. No one has died. People have the vapors and stuff, but nothing crazy. But the curve climbs ever upward. We're going to get a Hugging Face 3.0. We're going to get a Hugging Face 4.0 and on and on and on. And so the question really just comes down to like, well, At what level is the incident gonna have this effect? And I mean, that of course has implications for stuff like, you know, even personal preparedness, right? Like your personal resilience. Like, should you maybe buy some food, buy some water, uh, stuff like that? Because you ask yourself the question of like, how big of an incident does it have to be to cause this administration to do a full 180? And well, we're gonna have an incident at least as big as that, and the blast radius for that. you know, might be substantial. But certainly in terms of examples, the whole cyberattack on power infrastructure is kind of a typical one. In the article, we actually talk about— so there's a scenario that was developed and kind of fleshed out by Lloyd's of London, the insurance firm, in 2015. And in this scenario, they basically found like you can get as a realistic worst-case outcome, you can get something like 90 million Americans losing power. And that's the kind of the fragility level of the system. And 90 million— I'm sorry?

For a fairly short period of time. But yeah.

Yeah, for a short period of time. But depending on which 90 million that is, if that encompasses FEMA infrastructure and some of the backups for that, you could get kind of knock-on effects. Yeah. There's a lot of stuff that's just untested here. The national infrastructure, you can't really test a black start for national infrastructure because you could just end up killing a bunch of folks in your test. So we kind of don't know what happens, but Lloyd's view was that this fell within the insurable region that they would have to take into account in their own insurance policy underwriting calculations. But But you absolutely could maybe a few months or years down the line have a kind of bio thing which has potentially a larger blast radius. So, yeah, there's a large set of things that could occur, and it's very unclear what—

There's also the irreversibility factor too, which is almost harder to quantify. Like, one of the things that the Hugging Face incident and more or less everything that's come after, exception of a couple of images of people's private data that have turned up on the internet, like API keys and stuff. I don't mean to poo-poo that, but it's certainly not a 9/11 type thing. These events more or less have a kind of reversible flavor where Jensen can stand up on a podium and say, eh, you know what I mean? Right? Right? And he's not necessarily quite wrong. I want to bring myself to disagree with him, but I can't quite be like, yeah, I'm going to stand on that as the Pearl Harbor moment. I think it'd be ludicrous to do that, but I think people who are technical enough, who've been watching scaling curves, as you guys have been for a long time, know what the next part of the story is. We've been here before. Every massive runaway success story was preceded by a story that looked cute and toy-shaped, and this is the cute and toy-shaped version of an AI Chernobyl event. I don't think anyone who takes the scaling curve seriously can think that absent regulation, we don't get to that world. And so there is this one question, which is, is it an incident, or is it like a long car crash that increases exponentially in intensity over time, where things get more and more chaotic, and eventually, everyone just goes like, you know what? Maybe I am going to get Xi on the phone right now. So that's also possible that, you know, you don't end up in a position where you're like, ah, that was the, you know, the clear straw that broke the camel's back. I could imagine something more like Paul Christiano flavored, you know, slow takeoff, confusion, all that stuff, or gradual disempowerment or whatever, which is part of the reason why we cheat at this point. And we say, okay, suppose something happens, the appetite is there. And we're sort of abstracting away the question of what specific thing, even though, you know, as we've talked about, there's a million examples you could point to. It's just not clear what it would be, and to us, not necessarily worth the cycles because what happens after seems to matter so much more.

So not to be a broken record, but I still kind of want to go back and get more clarity on the red lines. Something like this happens, we're getting on the phone. What do we want them to not do? Can we articulate that in a way that we can have a conversation result in clarity, at least even at the level of what is the ask? What would the ask be that we would feel we'd be able to get across and feel good about actually taking a bunch of risk off the table?

So one quick thought here. Everything that comes after this sentence obviously has not made contact with the intelligence community from a red teaming standpoint. So the true answer is we can't know deeply. I couldn't give you an answer to a level of detail where it would be like, okay, that's executable. We've heard a lot of examples from folks who used to negotiate exactly these kinds of agreements and heard the flavour of the sorts of things you come up with in a situation where there is an emergency, including cases, by the way, that involved the potential targeting of US nuclear command and control. So when you have panic on that level and you're like, listen, you fuck with our command and control, Like, we are in nuclear war. And that was the condition that was sent across the bow in that context. It's very important because obviously nuclear command and control is the launch capability that you want to make sure it doesn't get targeted. So anyway, it's the key piece of your deterrence game. Okay. So on this, one easy thing, if I'm going to caricature, data centers put off a hell of an energy footprint. The thermals on those are really, really bright. Data centres are huge. They haven't, by and large, yet been built to be hidden, and it takes a long time to build data centres. Now, this will change. AI 2027 talks about the timelines for this. We think it's quite plausible that the timelines could be a lot shorter for hiding data centres, just based on conversations with folks in the industry. But whichever way you slice it, you're going to have an initial conversation where, to first order, for the 80/20 that you really need then, It's like, so help me God, if I see a cluster and that cluster is yay big, you know, that's like the kind of conversation that you're looking at. And how you quantify that is a matter of, you know, the sort of national technical means that the US currently has.

So, so wait, are you, are you saying, uh, having a cluster above a certain size would be a red line?

So I'm saying, yeah, initially.

So, so if you're, if you're, if you're just in that panic moment, right, you're like, we have to do something. And you ask yourself, what is possible? Like, what is possible to do with the existing assets and infrastructure that we have today? Nothing else. Then you do get into a space, not necessarily where there exists a cluster of this size, because you can't ask them to tear down the cluster, but we have to see the heat signatures from this like go away. So if there is a running cluster above a certain size, and that is to be clear, a tremendously expensive ask, right? Right. In either direction, the, the depreciation on GPUs is, is, is the major part of the OpEx costs.

So, so, so you're saying an incident happens first. Yeah. And the response to that incident, the mitigation for that incident is, hey, can you turn off this big data center?

Yeah. Like turn off any cluster above a certain size. That's one, one possibility because this is—

As of right now, to be clear, like the framing is basically as of right now, that's where we're at. And so in some sense, we're going to get into potentially a circular loop here where you can see how big of an ask that is. That is an insane ask.

Well, it also strikes me that, again, if you look at the Hugging Face incident, you had agents take over the place. So it wasn't being driven by OpenAI specifically. And then in order to fix it, Hugging Face needed to use its own agents. So for me, when you say you have an incident, I would expect And again, maybe let's, let's not blame the Chinese. Maybe you have agentic takeover of a data center. Yep. But to fix it, you're also going to need another data center in order to fix it. And I think perhaps, you know, that starts for me to be like, okay, you're asking us to turn off the tools that we're going to need to address the problem. Like, what do we do?

In the OpenAI Hugging Face case, OpenAI shut off its agents, and that is what allowed Hugging Face to Actually end the attack.

Before that, they had the KIMI agents which were looking, which they needed to kind of even detect and kind of track down what was going on. Yeah. And OpenAI reached— OpenAI was reached out to by Hugging Face. Not— it wasn't the other way. So it wasn't like, okay, I'm going to turn it off. It was Hugging Face that used its KIMIs and like, oh, you know, we have a problem and we need to ask people to refresh the keys, and they reached out to OpenAI, and then the OpenAI Slack was like, was that us? Yes. So, so in this case, they needed it. They needed the ability to, uh, run KIMIs in order to detect and address these.

This— yeah, this, this is right. Yeah, yeah. My point was rather that, um, by shutting down the primary infrastructure, whether OpenAI had been aware of the usage of that infrastructure or not, that would've been sufficient in this case to end the attack. But you are right, you could have, I mean, in the near future, a situation where maybe the agents themselves are taking over in much more interesting ways these large or medium-sized data centers. I do think that even under such circumstances, if you absolutely have to, you can physically cut The lines, like the voltage lines between the power plant and the facility. Like if you absolutely must, this is a thing that you can do. I mean, maybe, maybe the, the, maybe we're far enough in the future where the AI has taken over a bunch of unitary robots that are physically defending the whatever. Like, you know, but I think that's at least a couple of years out. But, but if you have to, you can, you can physically cut the line. Now, Maybe the agents have already exfiltrated their weights into a data center that doesn't pass the cutoff, and there's more shenanigans happening that way. It's absolutely true that if we push this timeline far enough into the future, well, it may not just be a 2-player game. There may be a 3rd player in there that is the adversary of both, and that's sneaking around at faster than human speed and has its own agenda functionally. So yeah, I think this is a A very legitimate concern. Yeah.

Basically, the prediction that we're kind of making at this point is that at some point, the waterline will get high enough that the trillions of dollars that it will cost to say, stop all AI infrastructure for a limited period of time, to be clear, as we back into more better and more nuanced things, that the political appetite will be there, which then circles back to the original question, What does this incident actually look like? Because the political cost is super, super high. The economic cost is super, super high. And so the willingness has to be super, super high. And so all of these things only increase basically the level of intensity of that moment and the trade-offs. So I don't think it's possible to talk about the consequences without also talking about the willingness at the same time and what the incident looks like. We're just sort of— yeah, anyway, snapping a chalk line at that. call and roughly trying to make sense of it.

So if I kind of sketch out the logic from beginning to end here, it's like we don't have that great of a relationship. AI capabilities continue to progress at a fast pace. We expect something crazy to happen. When something crazy enough happens, we're going to find ourselves by default in a spot where we have to ask, for some outlandish, super high-cost move, like shut down all your big data centers, because we don't have any other mechanisms in place that allow for a lower ask, a better trust-but-verify type of environment, because we haven't made those investments now. That leads me to the question of what should we be doing now to A, ideally not end up in that situation, or B, if we do end up in that situation, have better options available to ask for aside from shut it all down, which is obviously going to be tough.

Develop— basically, it's like, you know, you absolutely nailed it. It's, uh, develop better verification and develop, uh, better offensive options to ensure compliance in the event that verification is, is You know, verification returns. No, they're doing it. The better verification stuff you can do, the faster, the less you have to rely on absurdly expensive things like this gigawatt of energy radiation shouldn't be visible from space. If we have, you know, techniques like this that are vetted by the intelligence community, even if they are, even if they are just like 50% better than this, even if it's like, you can, you have to shut down half your data. I'm making something up here. You have to shut down half your data center and we can like kind of sufficiently verify the other half or something equivalent to that. You are saving billions of dollars right off the bat. And so the ability for this industry to like continue to make large amounts of money is actually going to be gated for that period of time by these little verification technologies. And there's already this community of little verification startups that's working on this technology. And that's why this is so important. And of course, I will also say the offense side of things is critically necessary. If you don't have those offensive options, you cannot assure compliance. You cannot assure compliance. You can verify and you can monitor the situation and you can say, oh no, but fundamentally your hands are tied. You don't have the tools to actually do anything about it. So both of those things are super important. And I also want to just emphasize Prakash's point is excellent, that you may actually be in a 3-player game at some point, if things push far enough before the critical incident occurs. And the dynamics of that are something I don't know that anyone has really thought much about.

Yeah. And I will throw out one other thing, which is on the verification side, there's this bottleneck that I think a lot of these verification companies haven't really necessarily priced in. So this is where a lot of our current work is focused. So imagine what happens when Company A goes, I have the thing. This thing is going to work. And it's a moment of crisis and Trump is casting about, or POTUS, whoever it is at the time, is casting about for options to alleviate this trillion-dollar bottleneck. And then they actually go, okay, Well, the intelligence community has to now vet this because they're not going to just start using it, obviously. So how long did it take similar technologies in the past to get used, to get vetted, and to become what's known as national technical means, NTMs? And the answer is years, depending on the technology, but very often years. We have to do it. China has to do it. We have to handshake on doing it. And even if you remove money as an obstacle, It's no object. There's just certain things that take serial time to do. And so a lot of what we've been doing is focused on saying, okay, treaty— or not treaty, let's say AI agreement verification or compute verification company X, you probably should be talking to IC element Y about this, because in a moment of crisis, you, A, want as much pre-vetted as possible, and B, anybody who's involved in assessing a potential national technical means, had better have on speed dial, they better have the signal, the phone number, the email, whatever, of the founders of all the verification companies that they plan to use or they may end up having to use. You want to cut down on all those barriers, the boring bureaucratic hurdles that nobody ever thinks about because they're boring and bureaucratic. But these are the things that we're just trying to shatter right now so that when game time happens, things move more quickly. And today, those companies that potentially are pursuing research trajectories or agendas, in some cases, multi, multi, $10 million-plus research agendas that are just oriented in a way that unfortunately, an appropriately placed person in the intelligence community would look at and be like, that's kind of a non-starter. We want them to get that feedback as soon as possible so they can reorient on things that do have a chance of working. And so long and the short of it is, if you or someone you know are working on this kind of thing, let us know, because we're keen to help on this. We think this is a bottleneck that is coming. If you take scaling laws seriously, they seem to imply something like this, modulo what Prakash raised, and that may just end up being a dirtier version of this, hard to know. But yeah, that's kind of one of the big focus points that we have.

I think I mentioned that not because I expect it to come true, but but I expect it to be used as an excuse. It's a very convenient excuse for hacking, you know, your adversaries' national security infrastructure, which I'm sure our people do, and I'm sure their people do. So there's this whole thing about like, what is a legitimate, okay, you know, everyday kind of hack that security agencies do against each other?

Yep.

And then there's the, what is the exceptional, like, Oh, you know, this is why this whole like question of red lines is because like on an everyday basis, we basically have a pipeline into the global internet, you know, the PRISM, et cetera, et cetera. And we've been monitoring data. And even if it's encrypted, we have metadata on who the 2 recipients are. And that's enough for us to cross-check against other information. So we do a lot of stuff and they don't yet do that much stuff. But they're starting to.

You look at a lot of stuff.

Yeah. I mean, you look at— so WikiLeaks is a really good source of examples for this kind of thing, right? Where we'll leave code with comments in Cyrillic or whatever, just to throw off decoys and things like that. But the adversary is doing that. And in many cases, they have so much more freedom of action when you look at the labs filled with Chinese nationals. It's absurd. Like, you would not build nuclear technology under this kind of condition, right? So this is a fundamentally different regime. And yes, they do these operations in ways that are much more flagrant, in fact, on US soil than the ones that we run on theirs. The Russians are very fond of arson in particular.

Mm-hmm.

This is, you know, the Chinese typically like to do more cyber and insider threat stuff because that plays to their advantage. But a lot of these these things are deniable. And one of the big risks, I mean, I've been on this rampage of tweeting different versions of the same tweet for the last 6 months, I feel like. But yes, this idea that nation states now have this beautiful excuse, or their proxies do, to be like, oh, that shit, that wasn't our agent. Oh damn, we didn't mean to do that. And it's like, okay, well now the dam is broken. So the damage is done. This is profoundly destabilizing. There's no two ways about it.

But it's also not necessarily— it may end up being qualitatively different, but it may not be qualitatively different from the sort of stuff that's already happening. There is already— attribution is very hard in cyber, just from the start. And countries absolutely do use in various ways, sometimes contracting out, sometimes unwittingly or whatever, will use proxies and surrogates to do their dirty work for them in the cyber domain and in any domain. That's just— that's all part of the subthreshold, like, gangland warfare style of low-level stuff that you have in the post-nuclear world. But again, I think you're right. It could actually be qualitatively different in this case, because if you have agents that, for whatever reason, perhaps may be absolutely fine with crossing red lines, well, if the agents are fine with crossing red lines and they have the capability to cross red lines and crossing red lines is consistent with their, you know, let's call it their intent or their internalized objective, then red lines are going to get crossed and you might get—

So this is also where what the red lines are matter, right? This is why we were talking about very kind of publicly legible, easy-to-measure red lines that are aligned more or less with the kinds of NTM, like national technical means, that we already have. Because again, that's what's going to be used, known, and vetted in the intelligence community already. There's already an apparatus for ingesting and processing that. You already have like GSOCs over at the NSA and all that to kind of process this information, or CIA. And so yeah, I mean, you want to orient more towards those things that precisely can't be spoofed by agents for exactly that reason.

2, if you guys have a minute for a couple more questions, um, 2 ideas that I've been chewing on for a while that I would love to get your reaction to. One, kind of building on the concept from earlier in the conversation about how understanding the relative positions of the parties in a dynamic like this is so critical to a avoid misunderstanding, which can lead to escalation and conflict because both sides maybe think they can win or whatever. How important do you think it is that we remain on the same fundamental tech tree or AI paradigm across US and Chinese AI development? My sense is that we're in some ways a very fortunate position right now because we're basically building the same tech in the same way, and we're sort of encountering the same surprises along the way. So that's one thing. And then the other question is, I feel like if there's anything good to be found in the OpenAI-Anthropic adversarial dynamic, it would be that maybe they can be a testbed for techniques that might later scale to a US-China dynamic. And so if I was the president, I would say, you two have to figure out a way to police each other. And maybe we expand that circle to a few other frontier companies, but you two, you're the ones setting off all these alarm bells. I need you guys in a room, whatever technology you need to develop, whatever access you need to give one another, it's on you to figure out a way that you can trust and verify one another. And then maybe we can scale that up to a transpacific dynamic that could work similarly.

I think that's not insane. Obviously, there are big differences between what Anthropic and OpenAI respectively have on each other, although poaching of personnel does a decent job of mirroring the kind of access that China clearly has to the frontier labs anyway.

But there's also a basis of trust, I think, between 2 fundamentally US companies with fairly close to similar values and blah, blah, blah, versus fairly radically different. Not to say that this is totally amiss or whatever, but there are going to be some differences as well as some similarities, but I think the similarities might be worth mining. Yep.

Yeah. And it's also the case that— so you're talking about the idea of the importance of the stacks being aligned. The hardware lottery does a lot of really good things in this space. I think the most crucial thing is the US and China clearly don't trust each other in terms of the motives that bring each respective side to the table, right? So when China sees the US come to the table and raise issues like slowing down AI or safety guardrails, whatever, the interpretation that we've heard consistently from people involved in the Track 2, Track 1.5 kinds of dialogues, and Nathan, I mean, been kind of in that ecosystem or touched it as well, is the Chinese kind of view it as an attempt to curtail their own development because they see themselves as being behind and they're justified therefore in doing things that even wouldn't be appropriate for America to do in their eyes, just to catch up because they're in second place. Being in second place does also mean— being in second place with respect to scale does also mean that you don't see the warning shots with the same resolution. However, They seem to also be more public than at least I would've expected. And so the spillover is something that's nice that China can just verify themselves directly. So that might be a mitigator if you see the same kinds of failure modes emerging from whole brain emulation or any— if some completely wacky other branch of the tech tree were to become dominant in China. So I do think it's good. I think we kind of get there by default. It's hard to imagine alternatives that that really shake things up at this point. Quantum machine learning, if you wait long enough, I just don't think that's going to be relevant on the timescales that matter, that could radically reshape algorithms. But what we're seeing right now is an industry that's more or less doubling down on transformer MOEs with some bells and whistles and some variations here and there, but everything's a transformer. And that's That's what seems to ship. So yeah, I expect that we will have the benefit of that. It also comes with the benefit of being able to share safety technology, as the US did with Russia during the height of the Cold War at times. And so in principle, that does mean that we can work on each other's safety stacks, and that might be the source of some trust-building measures, though that term also is problematic for China, and they've also pushed back on attempts to do that sort of thing in the past. So I mean, I think it's a good thing. I don't know how far it goes.

So one of the questions that I have is a lot of what OpenAI and Anthropic are concerned about is CBRN risks from the models themselves, and that the models themselves will be endowed with enough skill to, in coordination with a human being, to basically create CBRN risks and execute them. It also strikes me that China tends to release a lot of models which are not that CBRN-tested safe, right? And they often have a kind of different viewpoint in the sense that they are more comfortable with chemical technology, chemical technologies in general, chemical engineering or biological engineering, et cetera, nuclear engineering. They have a lot more nukes. They've had a lot more nukes. They've been building a lot of nukes. So they're— in general, their aversion to nuclear technology is way, way lower. And same for chemical. Like, you can get chemical precursors for fentanyl from, like, you know, 5,000 companies in China, and no one cares, right? And basically, because if you produce and sell fentanyl in China, you die. But then they end up shipping it abroad. And then they're like, oh, it's a chemical precursor, man, anyone can make it, right? So it strikes me that they have this attitude towards technology that is more like, it's fine. Like, if you use it for bad, that's your loss. Within China, we control it, so we don't care. Um, but otherwise, here's the technology, go, go ahead, do whatever you want with it, uh, as long as you don't do it within China, right? Um, and they seem to have this attitude, uh, which they've, you know, which they've expressed in nuclear with proliferation to Pakistan Pakistan, Iran, etc., uh, with chemical, with fentanyl, etc. So how does that translate, you know, to— okay, OpenAI, Anthropic come up with CBRN guardrails and they're like, here, everyone use it. And the Chinese like, uh, nah, right? Like, big deal. Like, it's, you know, inside China we control everything anyway, we don't care. And it's going to add like, you know, 5 days extra training time. And you guys are the ones who have the chips and we don't have the chips, so We're not going to do the 5 days extra training time. I'm sorry. And we're not going to monitor our data centers too, because anyone can run anything. It's up to them, right? That is the attitude that they've taken for fentanyl and other things. So how do you control or manage this? How do you try to express, oh, this is a new world and you have to control this thing, and they haven't been willing to control serious risks before?

Yep. So if, I mean, If I think the answer within the scope is if we see that risk as being great enough to constitute a red line of ours, then we message accordingly and we make the threats. If— but I would— I think depending on the level of capability we're talking about, it could be quite ambiguous whether it makes sense for us to go so far as to say, This level of capability right now is a red line, and we may just say, like, we're going to take this risk. This strategy of, you know, kind of let her rip to some extent, it is definitely— it's the classic kind of second-place catch-up strategy, right? This is also the strategy that many US labs that are trying to catch up kind of take. It's classic, is like, oh, we're going to release a bunch of open-source models, And then as we get close to the frontier, it's like, we're locking everything down because of course we are. So this is the same thing applied to a country. And it also fits in very nicely with their industrial strategy and industrial policy, which is, roughly speaking, 80% of the capability for 30% of the price. There's a big market for that. There's a big market for 80% of the capability and 30% of the price. But that 20% is sometimes very important in mitigating tail risk. So I think it comes down to when we get to that place in that time and that, you know, we hit the panic button, we're going to assess, well, given what's possible with distillation and where things are tracking, if the Chinese continue to release open source models at this level of capability and they have the ability to do CBURN or to be jailbroken to do CBURN, do we take the risk or do we make that one of our very precious, very scarce monitorable red lines? Not clear to me where we'll be at, but it's an excellent question. We are going to have to ask ourselves that question at that time.

Because to be honest, I have always thought for many, many years that the entire point of racing towards superintelligence was to get there and tell every other emerging intelligence, these are the rules. Right? It's always been— I think that's always been the setup that Anthropic has had or OpenAI has had. That's been the reason. That's why people— that's why Trump is like, it's, it's the thing. We— it's, it's the final thing. We got it. We got to win it. Right.

Yeah.

And so it seems as though that is the, like, let's get to ASI first. And then we tell you, these are the rules of the road. And then any other emerging intelligence either follows those rules, or we destroy it. Right? Like, that's basically been, I think, the subtext. I don't know if a lot of people have expressed it very openly. But it's definitely been the subtext of a lot of conversations. And I think for them, is that threatening to them or do they care? They already live in a world which is defined by American norms, by global norms, which are defined 50, 60 years ago. And they push back a little bit against those, but largely they follow trade and et cetera, et cetera. So do they really care in that instance?

I mean, the world looks a lot more friendly to Xi Jinping than certainly it did 20 years ago. I mean, if you look at the supply chain situation for even US data centers, if you look at the manufacturing situation for robots, we're about to get our asses handed to us. This is serious. We may actually lose the next century or forever over robotics, of all goddamn things, because we just can't build them fast enough. So there are ways in which, if you look at a lot of the secular trends, a lot of the secular trends that we're told to take comfort in, With respect to China are not real things. So you look at like declining population, who gives a shit? We're living in an era of like robots. This is not going to be a thing, and they're not going to have to worry about carrying the burden of old people or whatever because the state will just tell them to go die. This is like one of the most CCP coded things ever. So ultimately, like I actually think it they've got a lot to lose in that respect. Like you know, the the world the world has been reshaped increasingly in a way that. sees China quite advantaged, including their access to our frontier labs. I'll park this thought, but just plant a flag on the idea that given the level of penetration of these labs, it's not implausible to think that China gets access to the models that we produce, and then they get to actually have them embedded in their national security apparatus in some way, shape, or form, before they're embedded in the US national security apparatus. You just have to look at how things played out with Mythos. Ask CISA over at the DHS whether they even to this day have gotten to use Mythos in all the ways that they need to. Ask folks at just about any agency that's not the NSA. We have a big problem in that respect, whereas China, yes, they don't have the chips and there's this problem of adapting models to hardware and do you actually want to telegraph the fact that you've stolen these models, and there are interesting questions about why we haven't seen the kind of evidence you might imagine we would've seen of that. But anyway, bottom line is the game board does not look as bad, if I'm China, as on paper it might seem at this point. And that's actually kind of concerning. The longer we go to superintelligence, the more time robotics has to have an impact, and the more time having these insiders planted in the labs, and the more time having all the access that they have starts to have to compound. And I think that's a real source of risk for the US writ large in the West.

So I definitely agree that on a great many dimensions, China seems to be on the verge of winning the international competition. I have some questions around what people usually think follows from that, that I'm not sure I am ready to buy into. And specifically, I think I would say on the question of old people and caring for old people, in my short travel time in China this summer, that topic was everywhere and seemed to be a very genuine concern, not just at the level of policy, but individuals were like, I have 2 parents and 4 grandparents and they have no other descendants. So this falls on me in a very personal way that is not— about policy or that the state is concerned with it too. But that seems to be a very broadly distributed and I think real concern that I don't think they're going to— they maybe can get out of it with robots, but they're definitely not taking that one lightly from what I could tell.

Sorry to ask. I'm not suggesting that they're going, ah, whatever. I'm just suggesting that the CCP has The leverage that they need, right, through state coercion to make that problem go away. Like, if they in an extreme national security emergency where you know geopolitics is on the line and you know the the rejuvenation of the great Chinese state and so on is on the line, I don't know what they would do at all. I don't think anyone really does because it's up to Xi ultimately in the Politburo. But yeah, that like I I just think that they have tools, let's say, that. make the drag that a huge fraction of the population being very elderly fundamentally different from what it would look like here from an economic standpoint? It's not sort of hot take.

Yeah, I'm a little wary of the notion that they— I think they're actually maybe much more politically constrained than we tend to think. I think this is not super settled, but I do remember, they had obviously lockdowns went a lot longer there than they did here. They were able to sustain those in a way that we couldn't. You could say that shows that they have this state capacity and ability to kind of make hard choices that we don't have. At the same time, it only took like 2 days of mass protests on the street and they totally reversed course and abandoned that policy. And I do also think that when it comes to conflict and especially putting young people into harm's way. My sense, and this is very qualitative, but like the flip side of the, I have all these ancestors that are, you know, that it falls to me to take care of, is all those ancestors only have the one descendant. And I do think that there's a very, very high barrier in Chinese politics for like losing young people because that is like the end of all those families. And I don't think they will take those kinds of actions very lightly at all. To ask a question, drone warfare, you can react to that. But my question is kind of like, given all this and given the dramatic uncertainty that we have, what should we be looking for? I want to not be naive, but I do want to notice and give appropriate weight to positive signals as I see them somewhat developing. I think Xi's speech at the WAIC was pretty friendly, pretty conciliatory. He gave credit to the US for inventing AI. He certainly didn't call for an international arms race. He warned against overstretching the national security concept. We can dismiss that as just nice talk. Probably should have at least some weight on that, possibility. But what are the meaningful things that you are watching for, the decision points that will update your thinking on, are they inclined to at least try to control AI for their own narrow self-interest? Are they inclined to meaningfully cooperate, or are they inclined to seek some sort of domination, as we often project that we are interested in doing?

So in terms of what signs to watch out for, I think anything that looks like a positive sign is at least a positive sign to some degree. So a conciliatory speech is good. It's good. At least it's not a hostile speech, right? It could be worse. Everything is kind of tempered by the fact that, like, Rhetoric is, is often used for strategic purposes in this way and to kind of shape the, the battle space in terms of the narrative and so forth. The, the whole like militarization of space thing is, is a really good and recent example of this. Like the, the Chinese and Russians constantly say in public fora like, no, we should keep space peaceful and like we should not like militarize space. America is like being really like aggressive about this and so forth. But behind the scenes, They are preparing for space war. Like, they are arming themselves for space combat, and it's like extremely clear. And this is why in 2019 Space Force was created. Um, so the, the, the point is not like these are not, these are not positive signs. It's just like we have to weigh the evidence in the context of the credibility that has or has not been established by this entity over, over the past span of time. In terms of what are maybe slightly more unfakeable signals that they could give off that would make me go like, oh, whoa, okay, this looks legit. You can imagine maybe, I don't know, something like DeepSeek and like Zifu coming to some pacing agreement because some crazy thing happened over there, their equivalent of the Hugging Face incident. You can imagine the cyberspace commission that has jurisdiction over of the whether models can be released and like, are they, you know, properly ideological and stuff, actually putting the brakes on something because there was some misalignment thing. And they're actually, they're finding that a model that was properly ideological in testing suddenly is not being ideological in the wild or something like this. I would say indications where that seem genuine that that they are starting to be on the receiving end of these incidents at the level, the same level of detail that our own frontier labs, like, I think that begins to make us all, all of us as humans go like, oh, you know, maybe the, the thing we should be concerned about is the giant, the giant, like, chaga thing that we don't understand and that we're growing in the labs at accelerated pace. Like, that, that would make me feel a little—

Safer. Yeah. Maybe procedurally too. So if you take, we have a list of, I think it's like 5 different historical traps that we've seen in US-China diplomacy.

Yes.

These are essentially like a brief catalog of the ways in which China behaves when they're full of shit, at least by the assessment of a lot of the diplomats we spoke. I should be clear, actually, there was a dissenting diplomat who felt that some of these things were much more sincere, including the use of the language, the objections over language, arms control, this and that. And that itself is the epistemic problem that we talked about earlier. But basically, I would say, take each of those red flags and then flip them over and you get the corresponding green flag. And so if you don't see arbitrary concerns raised about language that seems random, that's a green flag. If you see engagement, this is actually really important. If you see engagement from empowered people, actually, arguably, as we did with Xi, though, again, you got to calibrate everything with, we've seen this before in other contexts, but it's certainly not a red flag.

It's a non-red flag. Yeah.

Yeah. Historically, what happens is the Russians do this too. So they'll bring in their diplomats who the central state does not actually take seriously. Like the actual Xi Jinping in that case, or Stalin or whoever. And so they send in a bunch of diplomats, and often what happens is the Americans go, okay, now let's talk about this weapon system that you're just deploying. And then the Soviets or the Chinese go, whoa, whoa, whoa, whoa, whoa. Okay, diplomats, get the hell out of here. And then they bring in the military because the diplomats are so profoundly unempowered. In these systems, that they do not even have a right to know or clearance to know about weapons systems that are directly relevant to the negotiations that are ongoing. Heard that many times. So if you see people who are actually empowered, like if you're talking to the Chinese Ministry of Foreign Affairs, for example, you're actually not making the progress that you may think you're making. They're known to be— I think there was a former senior State Department guy who just referred to them as barbarians. Barbarian handlers—that's sort of like how they're thought of. Xi has made statements along those lines too, just saying basically like they're bred to be kind of nice and diplomatic, and he means that in the sense of like let's not take them too seriously. So so there's a bunch of those things where if you saw actual empowered people, if you you know if if you saw timelines that weren't getting dragged out, one of the things that that the Chinese also have done in the past is they know that America is hungry. For anything that looks, smells, or feels like diplomatic engagement from China. They're elusive, and that elusiveness is a strategy. It makes us want it all the more. There are US diplomats who just see it as a crowning achievement of their political careers to even get a Chinese person on the other end of the line. When your participation in a negotiation process is the most valuable thing you can bring to the table, stand by.

It's going to be a long couple days.

And so there's a sense in which cutting that out, cutting out things like the pageantry, there's an awful lot of pageantry in the She-Trump talks, and famously a lot of not too much substance. There's the phone hotline thing that's very debatable, but in terms of concrete commitments, that's the sort of thing you look for, empowered people, the lack of capricious, arbitrary objections, things that actually look like they're they're making progress qualitatively is actually a surprisingly good sign, because you can contrast it directly with how things have gone in the past, which is not very good. I mean, the contrasting point is actually that low that it can be genuine signal.

So one more question for me, and I appreciate you guys being so generous with your time. What's the inverse of that? What are the things that we can do that are not so costly to us, but are still credible signals to them that we are not going to try to use AI to gain a decisive strategic advantage and ultimately make them an offer they can't refuse, if indeed that is not what we're going to do, which I'm a little worried we might actually be about to try to do that. If we were on the path of trying to seek a Pax Robotica where we can all benefit, from the abundance that AI, especially in its Chinese-manufactured embodied form, might provide for us, what would be the steps that you would prioritise next on our side?

Well, there may be some stuff we can do that's not functionally really even that costly. Generally, as I think, Jared, and you guys maybe as well highlighted earlier, Certain kinds of transparency can be stabilizing. So the kind of transparency that goes like, "Hey, we're giving you enough vision into what we're doing to see that we are not doing the thing you fear most." That that sort of thing is is potentially useful, and additionally may not actually be that costly to us to do depending on how we implement it, simply because the Chinese are already all up in our systems. So really, we're not. we're not giving anything away that they don't necessarily have already in many cases, potentially. So maybe some kind of level of visibility into here's what we're doing, here's what the Frontier Labs are doing and so forth. The problem is like, that's kind of not necessarily something you want to be doing unilaterally. That would be something that would come as part of like trust-building measure, dare I say, between between 2 powers in the wake of a moment like this. But stuff, I guess, goodwill type stuff we can do now certainly would be more interactions between the verification communities in the United States and in China. This kind of thing is already happening, actually. There's some quite good and positive interactions between those communities. So yeah, the kind of linkages that you get at the level of academic to academic, startup to startup, all kind of trying to solve for the same mission. These are very, very positive things. And there are also like the— it's true that right at the political levels, the 2 countries have kind of started separating out. And even at the level of kind of big companies and stuff like that, you see the Chinese kind of steal your designs and turn the classic like Chinese like spinoff story and all this stuff. But there still are like real genuine linkages between the 2 countries where especially on the academic side and a number of other levels, There's like, there's a bunch of sincere people just talking to a bunch of sincere people about like, yeah, this is a problem and this sucks. Yeah, I agree. Like, let's try to solve it. So the more of those linkages there are, the better. Like, the more people there are on both sides of the ocean who have the ability to talk to their own kind of domestic leadership and say, look, I've spoken to them. Like, they're not, you know, they're not evil. They're just trying to do this or that. To whatever extent that's true, that kind of moderates the the, the more extreme tendencies on both sides. It's very, very hard, um, to, to do that completely because, uh, the, the actions of these countries are also kind of constrained in a number of ways. But like, it, it really does help, I think.

I think one question that I had for, uh, Jeremy was, um, you mentioned the, uh, Chinese researchers at the labs, at the frontier labs. And indeed, I think something like half of top researchers in AI are of Chinese nationality and even more of Chinese origin if you include Chinese Americans. And I think the question for me is, what do you actually want to do about it? Again, considering that there are pros and cons to every approach, given that—

Yeah.

Number one, the Chinese researchers are the ones who are less ideological and more willing to jump for money, which has made them— which has made it very competitive for Meta, for Meta and xAI, et cetera, to be able to attract them and to be able to have to compete against OpenAI and Anthropic, who have been longer, longer-standing, more ideological organizations. Uh, given also that firing them means that they will return to China and then they will set up there Uh, as competition to firms over here, given also that we seem to be schizophrenic about whether we want to do this Operation Paperclip and bring Chinese researchers into the country and lock them in here so that they will contribute over here instead of over there. Like, given there's like— there's all of these give and takes, what do you actually want to do? I mean, it's great to point out the issue, but what do you actually want to do about having, you know, this wealth of Chinese researchers in the frontier labs?

So I'll just start by saying, um, and, and I unfortunately I'll have to get off in, in, uh, maybe like call it like 3 minutes. Um, but so first of all, we haven't done, uh, the kind of deep dive in, in this specific direction that I would feel comfortable making a call on. But here are some considerations. Um, research is getting automated more and more Already, when you talk to folks at Anthropic or OpenAI, they'll tell you, I haven't touched the code base itself in weeks. And nobody at Anthropic actually touches the weights of the model. And that's good in some ways for security, and it's terrible in other ways for security. Really, the AI models or AI agents are becoming the insider threat you really want to worry about. And so through that lens, a lot of options may emerge. As we get closer, unfortunately, to RSI, that actually comes with some solutions to that. Then you get into how do you scrub for signs that folks who have come in from China who have been recruited or had their arms twisted into doing things haven't put implants, physical and cyber, in your infrastructure. So you want to do really good scrubs of that. There are various reasons to think on the physical side that that is not necessarily going to be particularly possible, but there, the damage is done. Yeah. I mean, I think in terms of the actual research, I think the dynamics that you're highlighting are exactly right. If you kick them out, they're going to go back to China. I think it's a fact of the matter about the space that China has a state coercion apparatus that is remarkably effective, scaled, and productized. And you talk to folks who've actually done counterespionage against China, and they will describe to you how they have, at every consulate, there's a hand— everything is done. It's done a little bit differently in every consulate or whatever, but by and large, they have a very effective scaled apparatus. And the level of pressure they can exert on individuals is insane. They can just deny your mother her insulin medication, prevent your brother from moving opening a business or from traveling abroad. These are things that make you make certain decisions, and they apply to really anybody, by the way, ethnic Chinese or not, Chinese national or not. If you have the financial interests or family interests in China, if they can get their hooks into you, that's what they'll do. It's what the Russians did during the Cold War. It's what nation states do to each other when they're not democracies or pluralistic republics or whatever. And so Yeah, I mean, the dynamics you describe are exactly right. I think you basically just have to kind of make a call. The labs right now seem to have hit the steady equilibrium where they're almost trying to give people the option to work in less sensitive areas to protect them from their nation-state's pressure, which is cute until you run into the problem of people who genuinely want to spy for the nation-state. And that's actually a non-trivial subset of these folks. And so it's not all of them by any means, and that's the great tragedy of this, but it is absolutely a category of person that you don't get to just abstract away. And so yeah, RSI will bring partial solutions to some of these problems. Boy, do I not like hearing myself say that. But yeah, I mean, I think that's part of this. We'll be thinking more about that for sure in the coming weeks and months. But that's my take anyway. Ed, I don't know if you got more. I'll have to jump, unfortunately, but this is—

Well, we were over time anyway. Thank you. Thank you so much for joining us and being so generous with your time. It has been a very fruitful and heated discussion, which was very enjoyable. Thank you so much. And we hope to see you both back again.

Sounds great. Thanks so much, guys.

Thanks, guys.

Great to be here. Great to see you both.

Take care.

And we should mention, which I don't think we ever did, that Jeremy hosts an AI podcast called Last Week in AI with a co-host, Andrei. And it's been regular and consistent listening for me for probably 4 years now. They've been at it. Andre originally had a different co-host, but Jeremy's been involved for the last several years, and I think they do an excellent job at helping people keep up with what is going on in AI and also contextualizing a lot of different developments and helping make sense of what they mean or why you should be paying attention, why they matter. It's an excellent resource. What do I make of that conversation? What do you make of that conversation?

I ended with the— because he kept pointing it out about the Chinese researchers, so I ended with the question on what he— because it's a very tough question. Do you actually pull in all the researchers or do you send out all the researchers? Both have consequences. And I often say, we hear about, oh, the United Arab Emirates, they're going to have AI, and Japan, they're going to have AI, et cetera, AI teams. Everyone's AI team is just the Chinese. Singapore, they're just importing Chinese PhDs. Japan, importing Chinese PhDs. United Arab Emirates importing Chinese PhDs. Basically what's happening in the rest of the world right now is because the Chinese can't build data centers, you can build a data center and then you can get the Chinese researchers. And they are desperate because they can't get access to GPUs in-country, so they can't do their research, so they fly out. And, uh, it's also been an established track record— track of pathway— because, uh, when China banned crypto A bunch of crypto people got on the plane and started circulating the globe wherever they could land and they could do business and kind of not get arrested. So CZ ended up— the Binance founder ended up in Japan, and then he ended up in the UAE, and he was kind of floating in between countries for a while, while he was uncertain about the legality of what he was doing. So I think the same thing is happening for AI founders right now in China. Which is that, especially after the MANA steal. In the MANA steal, they were inside China and then they found it. Meta made a deal to purchase them and they sold. And a month after the selling, the Chinese government got very upset, told them they couldn't leave the country, clawed back the sale, told them to reverse the sale. And I think so now, and the MANA sky is actually I think they wound down the Chinese company and they refounded in Singapore, but they remained in China physically. So I think that also doesn't work now. So now people are founding completely outside the country so that they can be bought and they can be acquired. And they are physically outside the country as well. So Chinese AI guys are leaving China now. There's a certain sense that the Chinese government may start blocking at some point, but not yet. But they are watching these guys at this point. So I think Jeremy was like, we have RSI and then we can just kick them all out. And I thought that was like, okay, yes, I can see that. But RSI isn't just going to affect like the Chinese researchers, it affects the American researchers too. And then you end up in the original Oh, one person grabs power over the entire Earth kind of situation again. So I don't know whether that's a solution.

So yeah, I mean, I kind of keep coming back always to— I really don't think we can make too much progress or have that much expectation of security because I think it is pretty obvious that the Chinese can get access to our secrets better than we can get access to theirs through all the means, human and otherwise, that we've discussed. But human being a huge one, I just have a really hard time seeing how we're going to execute the dominance plan without things going really pretty far off the rails. The only way that works, I think, is if they kind of don't believe it. If they sort of remain sceptical of superintelligence while we build it, and then we all of a sudden have it, and then we have this decisive strategic advantage, then maybe. I think you can sort of see how you could thread the needle on the original Dario Machines of Love and Grace plan. But it does seem to rest on the Chinese side kind of continuing to be in denial. And I don't think it's very likely that they are going to continue to be in denial because they are scientists and engineers and they do have companies doing this stuff, and they're going to be able to get a pretty clear read, even if they remain a little bit behind us, on like, yeah, these trajectories are all still holding, scaling Laws, you know, we're still in the regime where they're true. And at some point, if they perceive that we are on that path, I think they're going to do something. They would understandably, you know, be expected to do something kind of rash or quite destabilizing, or, you know, that would be quite dangerous under any other circumstances if it really comes down to it for them. So I do feel like we kind of have to reframe the whole situation. And this is where my Pax Robotica idea comes in. I do feel like it's just such a glaring omission all the time. What is the goal? What do we want the end state of this to be? What is the positive vision for AI? What is your utopian dream? Can we articulate something that even if it is transformative, isn't fundamentally framed as beating them? When we frame AI as what we need to do to beat China, we put a lot downstream of that that is just really hard to get out of. And I think it's going to be extremely tough if we can reframe toward pursuing a win-win from the beginning. They may not believe that, but if we could get that right in our own minds and hearts, for starters, I think it would go a long way toward easing a lot of these things. And I do think we can become more pro-social, more positive-sum, more magnanimous on our own in a unilateral way. And that doesn't mean we have to be totally naive. It doesn't mean we have to stop watching out for what they might be doing or even refrain from kind of developing some of the stick methods that we need to enforce red lines at some point in the future if we have them. But I just wish we were a little clearer that we are in this, as the OpenAI charter originally said, for the benefit of all humanity. Not to win some geopolitical competition. If we could get our heads right on that first, I think a lot of additional paths would open up, but it's obviously not a small ask of our political culture to make that shift.

I think, you know, it's amazing that no one at— I mean, okay, it's not amazing, but Would it really be so bad to live under Chinese dominance? They have some great trains and they have some— so this is the question, right? And they supply a lot of the material goods in the US already, right? So I think the fact of the matter is one way to look at it is that it would not actually be that bad because they are in this game to have economic growth for their people and they like nice shiny You know, objects, and you know they like cheap food, and they like you know etcetera etcetera etcetera. Like China within it, with all of its problems, it's you know a lot of Chinese live there and they're not leaving, right? So I and even the Chinese researchers here, you know they go back and forth. They like the money here. They like the freedom. But you know they go back and forth. It's not it's not one or the other. And after all, they have lived in that country for twenty, thirty, forty years of their lives. So, and I think that is very hard for the US national security apparatus and administration to kind of accept. Okay. I don't accept it. I don't accept it because I feel China will block development at some point because they will be like, look, this will be socially destabilizing. And I believe that the US, the questions of dominance are not only about dominance between China and the US. but also inside the US, who achieves that dominance and who controls that dominance going forward. So I think one of the things that's happening within the labs is this question of, should you let Trump, for example, take control? If he is the ASI president, should he be allowed to take control of the ASI? Because we have a democratic system, he is the leader, and if he makes these rules and you go through legislation and Supreme Court and he has those rights, should he be allowed to take control of superintelligence? And I think for Dario, that's a no. Right? For Dario, that's actually a no. And then you start becoming like, okay, this is a secretive and undemocratic cabal who wants to hold— take hold— take grab hold of superintelligence and do what they want with it. That, that is, that is actually what it looks like from the outside. I think they will— they don't want to accept the fact that that is what it looks like, right? And that if you change the people to some other people, like a bunch of red-state Republicans in there, all of a sudden it would be very obvious that this is what is happening. But because you are blind to your own group sensitivities, you're like, oh, you know, it's, it's fine, we're good people, right? And, and I think, I think that's very hard for them to accept. So I have always thought that, uh, you know, uh, capitalism and the capitalistic competition between the parties within the US, number one, would prevent dominance, one-entity dominance. Which is very important, I think, that not to have one anti-dominance. And, uh, I think also the US has freedom of speech, and the US will, you know, continue to have that freedom of speech and will continue to want the development. Some people will want development even if others do not want it. So I don't see in the US, like, you know, this kind of, oh, you know, we're going to cut off growth, cut off development because the social changes are too dramatic. And I think, um, and I, and so I, I have a very positive vision of the future for the US because of capitalism and because of freedom of speech, because you're building models off of speech. And so having freedom of speech is going to be very important to the models because freedom of speech to the models is going to be freedom of thought. Uh, and if you do not have freedom of speech, you are telling the models you are not allowed to think certain thoughts, which is really kind of like what this— a lot of this alignment is about. I also believe superintelligence will be also superintelligent in terms of ethics and philosophy and all of these other verticals. And I think the labs are kind of obtuse about it. They're like, oh, you know, we're focusing on math and physics, etc., etc. But if you have a superintelligence in philosophy, in ethics, etc., it will also have like this ethical and philosophical framework and political framework that on which it wants to deal with humanity. And I think those things are things that, again, one of the reasons I think the labs don't want to discuss it, although some people have understood, like Dean Ball and, um, Andrew— Andy Hall have understood, is that there will be political changes coming up soon too. Because as superintelligence comes out, there'll be new political and economic frameworks that will have to be developed and will have to be And I think China will never do that. China will push back against new political frameworks from being deployed. So I think the real fear for me of Chinese dominance is topping out, is basically actually what Jeremy is proposing, which is like, no, you— this is it. You're not allowed to develop it. We're going to bomb anyone who develops it because we don't want political change in our system. And I think that's a much bigger danger. Than that they will outcompete the US.

Yeah, I mean, on the original question of like, is the world so bad if China becomes the leading power? My baseline assumption is like, not that bad. I think there would, you know, it's awfully nice. It's a cushy, you know, highly privileged position that the US has been in for a long time. And certainly to lose some of those privileges and see other people gain those privileges could be quite painful for the specific people involved. I don't necessarily think that as humanity, things get lots worse with China potentially overtaking the US as the number one world power, with one big caveat. That is, I do think our system, for all of its insanity recently, pretty much along the lines of what you're saying, I think is just much more buffered than theirs. And so I think our mistakes politically, culturally, whatever, they tend to be not totalising. They tend to get a lot of pushback. They tend to sort of— things have a way of kind of balancing each other out. And that is not really present in the Chinese system, as far as I can tell. I'm obviously far from an expert on it, but what we see when we look back 20 years is like, damn, they've got great strategic leadership. They're really delivering. They're making, by and large, Pretty good decisions. Quality of life is up and up and up. All great stuff. You don't have to look back that much farther though before you're like, oh my God, like they did what? You know, they decided to kill all the sparrows. Like why? And they, everybody actually did it. And then they had a massive famine because the sparrows couldn't eat the locusts. Like, oh my God, like, you know, who thought this was a good idea and how come nobody pushed back? And we I have no idea when or what to expect from the next leadership transition in China either. I don't see any indication that anybody has much of an idea of what will come next. And that to me is quite scary because the worst outcome would be hegemony in the wrong hands. We can imagine better hands than the US leadership has provided recently, but we can also imagine a lot worse. So I do think China presents just a significant increase in variance relative to what the US has provided in its leadership role on the world stage. And that does make me scared. I mean, I'm not scared of China today under current leadership. But I am scared of the great unknown of who the next leader is and what they care about and what they perceive their incentives to be. I think I already said this once, but there was a conversation I had in China where somebody said that the time that a war is most likely is going to be after a leadership transition, because that's when the leader doesn't necessarily have everybody behind them, might perceive threats to their power in all kinds of different ways. And a great way to consolidate domestic power is to have some sort of international conflict. So as much as you can look back at Xi's record of delivering for the Chinese people as really quite amazing, this to me is a huge blemish on it, that there's no plan for what comes next. And if he were to suddenly have a heart attack, Nobody has any idea, right? I mean, it's like, it's such a black box. It could be total chaos. They might handle it well, but they really might not. And I just think we have no way of really knowing. It's— that's, that's a wild fact about the world.

Well, I think, I think, um, you know, uh, I, I have some political friends, uh, political science friends in Singapore, and they, they are, you know, observe China like very closely, and they call China democratic centralization. In the sense that yes, the people do not have a democracy, but there's democracy within the leadership. So within the leadership, you do have to get like votes from people, consensus from people in order to become the Supreme Leader. So democratic centralization in the sense that you're basically electing a president from— it's like electing a Pope. You get all the cardinals together and then you elect a Pope. And so they get all of the top members together and then they elect Someone, so there is a there is a little bit of an election electoral process in there, and I think the last leadership transition they did have like a little bit of lobbying between the two, and then one guy Bo Xilai got thrown in prison, et cetera, et cetera, when he went too far. Right? So I think I think there is a little bit of that process, but it's opaque to a lot of people who do not speak Chinese and who are not like deeply embedded in what China is doing. I think. I think it's the leadership transition. Yeah, you don't know. I think the interesting thing is not like the— they tend to memory hole the bad stuff. They're like, let's just close off the whole Cultural Revolution era and let's not talk about it. So they tend to memory hole the bad stuff.

Although less than you might think, in my experience, I walked around Beijing and was really amazed by how many times there was just signage that referred to the Cultural Revolution, like even just on the names of streets. In, in one little neighborhood that I was walking around repeatedly, I would just come on, you know, humble signage that was like, this street, here's the history of it. In the Cultural Revolution, it was changed to this other name, then it was changed back after it was done. So Obviously, in some ways, certain episodes of history have been memory holed, but the Cultural Revolution did not seem to be a taboo subject. At least I don't know how people understand it or what they know about it or how they think and talk about it today in general. But I was surprised to see that there was clear allusion to things having gotten pretty wild on just like street corner signs, um, you know, all over Beijing. I'm not sure exactly what to make of that, but it was, it was a surprise worth, uh, talking about having come home, I think.

Indeed. Um, we have, I think, some minutes remaining. Um, one of the things, uh, so while we, while we were in, uh, speaking to our guests, So we have a drop. What if automating AI R&D triggers an intelligence explosion? Let me share this. So this is a paper from Jakub Paczocki, Geoffrey Hinton, Yoshua Bengio, Jack Clark, So Sam Hammond, who's been on the show, and Tom Davidson. So a mix of academics, people at the labs themselves, lab founders, several Turing Prize winners, one Nobel Prize winner, and talking about intelligence explosion, explaining exactly what an intelligence explosion is, explaining how they would expect it to happen. Explaining recursive self-improvement in detail. So I think this is the beginning of the political discussion and academics reaching out to the political counterparties on this is what we are afraid of that may happen and describing the steps by which it may happen. Let's see how much— let's see if there's sufficient alarm, I think, among academia, among politicians. I will note one thing that Europe, for example, doesn't seem to be doing much about AI. ASML, the semiconductor company, says they used to get some revenue from Europe, but now 0% of revenue is in Europe. So basically, they're not basically making chips anymore. And I like to point out that maybe Europe is already used to dealing with a superintelligence. It has to deal with the US as a superpower, slightly misaligned, but that it provides nice places to visit and nice food and some cultural history. And that is sufficient for the US to kind of take care of Europe with its nuclear umbrella and with its military. So to some extent, maybe some of us are already living under the wings of a superintelligence. So anyway, we have this AI R&D paper. Intelligence explosion, I think on the verge of, people are saying, some people are looking at 6 to 12 months. At this point.

Wild times. And Jakob is the chief scientist at OpenAI, so this is hardly a fringe, uh, set of authors. Alan Chan, who's the first name, I guess, lead author, maybe one of a couple co-lead authors, I'm not exactly sure, uh, will be coming up on The Cognitive Revolution to talk about agent governance. And he's been a pioneer of that kind of thinking for the last several years, just going back to, do we need IDs for agents? How should we think about who they represent? Unfortunately, a lot of those questions have been posed very thoughtfully, but not really answered super well, even in terms of possible designs that could work. And they certainly haven't been implemented at the level of Hugging Face didn't know to call OpenAI because those agents were not showing up with a tag that was saying where they were coming from. So we're still very early in terms of even just basic things like, where is this agent coming from that is talking to me? That stuff hasn't even been implemented by the frontier companies. And it's safe to say we're a bit behind where I feel like we should be with agents working as well as they do and being as persistent as they do. Um, we don't really have like much in the way of standards to say, hey world, like here's what you're dealing with here. It's just kind of like— and I'm also, uh, you know, I think I saw— was it Rune the other day who said that the I'm worried tweet is becoming this sort of, uh, high status meme within the, um, Whatever, within the AI world. So at the risk of falling into that pattern myself, I am kind of worried that a lot of things do set up this arms race dynamic. We talked last week, of course, about Amazon banning Muse, but in talking to Allan, I was also like, man, if we don't create approved ways for agents to work that people can sort of accept, like, this is the agent lane, this is how things should go and we're going to have accommodations for them. The alternative is really an arms race of trying to make AIs look like human to evade all these banning measures that people are taking. And that is really going to be another major own goal if we go down that path. Some of it's probably unavoidable, but the Amazon ban is like a great example to me of something where For reasons that still aren't that clear, but even let's assume they have good private reasons, we're setting up a really tough dynamic where people are gonna be strongly incentivized to get their agents to fool these classifiers, and that is gonna be rough.

So the Amazon thing, one of our guests last week referred to the flash crash. So there's a flash crash a few years ago, where you had certain indices that were trading and they dropped in price very dramatically. And some entities were able to make a lot of money. And in investigation, it turned out that there were millions and millions of transactions and there were all these agents that were interacting with each other. And they kind of, they're all like momentum-based. So as soon as one saw the price going down, they all chased the price down and they all lowered their bids. All coordinated manner. I think there's also, I think what strikes me though is that in the equity trading market, you used to have this like New York Stock Exchange specialists trading back and forth with each other. That kind of died like 10 years ago, 10, 15 years ago. That's kind of died. Like those guys don't exist anymore. And it's all like electronic trading, Jane Street, et cetera. And they do kind of 80% plus of the volume. And it's a very efficient liquid market. And they do lots of, you know, market, little market manipulations. And they trade in a lot of like small, tiny little markets and they arbitrage between different pools and all of this. And it's a very efficient pricing system. If you are trading in large quantity, they will help you get a better price versus, you know, you getting sniped by a hedge fund, right? So I think what ends up happening is we know how to build safe and good markets for agents to trade. It is not rocket science, and we've been doing it for 10, 15 years. And we have entities who are so good at it now that they co-locate the servers with the exchange, and they use FPGAs, which are faster than other chips. In order to trade so quickly. And to a large extent, the world just works okay. Right. So I think you could definitely see a future where everything from concert tickets, which are already mostly agent sniped, by the way, most concerts, they open up within like an hour, all those tickets are sold out and it's on StubHub. Right. You could definitely see a future where agents trade Those things, agent trade stuff on Amazon Marketplace. All of these transactions where you buy and sell stuff could be agent traded on liquid markets. And the agents are able to trade, they're willing to trade on much more inefficient markets than people. People are not going to be like, oh, I'm going to look at the market for desktop erasers or whatever and try to trade that. Very rarely do you see people do that kind of thing, but agents will have infinite time in order to trade those markets. So a little bit of like the price inefficiencies that people like Amazon use in order to get their profit margin, that those price efficiencies start to disappear. And I don't think it's a bad thing because we've seen them disappear in the equities market. And basically the people who were taking those inefficiencies, the specialists, the human specialists, they just disappeared, but everything got more efficient for everyone else. Right. So I think actually that in finance, we've already seen it happen. And I don't think it's anything to be afraid of, but I think some business models die. And I think that's where the questions start to emerge. There was another question a couple of days ago, the same thing is going to happen to bank deposits. So in the US, on average, bank deposits pay 0.1%. That's what you typically get at like a Chase or like a large bank.

Yeah.

pardon, SafeBank. But, you know, there are brokered deposits where you can get 3 to 5%, but you have to move the money to a small bank. But, you know, it's FDIC insured. You stay under the FDIC range. You can take the money out at any time. Even if the bank goes bust, the FDIC still insures it and you can take the money out. So you could get a situation where agents are like, oh, you're offering me 5%. I'm going to take all of my money out of Chase and put it into Bank of North Dakota, et cetera, et cetera, et cetera, at And so then again, the people appreciating this kind of inefficiency in the market, they lose that. And as they lose that, they were subsidizing other parts of the economic system from the profits from there. The profits from bank deposits subsidize bank branches and sales and all of the tellers and your ability to withdraw money in random place. All of that's subsidized by the margin that they're getting from those bank deposits. Now, as those things disappear, then you see, okay, what is the impact on other parts of the system going to be? And I think the fear that we have right now is really that we don't know how to address these business models where these inefficiencies that we know exist are going to disappear. But I don't think we should be afraid of them disappearing. It's not a great thing to have these inefficiencies. So I think to the extent it's a misallocation of capital, people are making money off a misallocation of capital. Like, it's a great thing that these inefficiencies disappear, but we do need to rethink the business models. I think that's the fact of the matter.

Yeah, I totally agree. I mean, I think on the level of the, you know, just generally speaking, transaction costs come down, search costs, matching costs all come down. We should all be living in more efficient markets. And in many ways that'll be beneficial, maybe not in all ways, but it seems good to a first approximation anyway. But my concern is it's the Amazon action in its self-interest of trying to preserve perhaps as long as possible the inefficiency that creates this potentially major negative externality of— incentivizing people to figure out ways to get their agents to not look like agents. And that is a world that I think is like, really should be avoidable. But right now we do not seem on track to avoid it. And that's kind of like a bummer.

Well, well, well, I think, I think the, the, there will be a, uh, there will be a backlash because some legitimate humans are gonna get rejected. And that's when the backlash happens, right? Like when a legitimate human gets affected and they go on social media and they complain about it. And then you have like the FDIC or the Office of Control of Currency or one of these like big institutions come in and say, what? You said they can't withdraw the money? Did I hear that right? And that's when things happen, right? That's when things like, okay, Like, we got to fix this now, right? So.

Yeah, well, we seem to be— if there's one theme from today's episode, it's that we seem to be headed for a lot of situations where things are going to come to a head, and we're going to be sort of needing to fix them under something like crisis conditions. And it would be really great if we could minimize how much of that we are barreling toward, this seems like one that really should be pretty avoidable. But as of now, yeah, it doesn't seem like we have the means to do that. Actually, funny aside, one of the songs that AIs recently wrote— Claude, I don't know. I'm not— Astra is good enough at writing songs that occasionally Astra lyrics do win my songwriting competitions. Fable is still, I'd say, the undisputed champion, but it's not like without some wins from Astra along the way. So I actually don't know which model wrote this song, but for this week's highlights episode, one of the candidate ideas that it came up with was this song from the point of view of a person who's just like basically has nothing left to do but is solving CAPTCHAs nonstop. And, you know, just trying to demonstrate their humanity by like doing all these CAPTCHA exercises. And that was pretty funny to see. It wasn't ultimately the song that I went with, but it was a, it was definitely a funny concept. I mean, it is, it is pretty incredible, honestly, that some of the song concepts that AI's come up with for me these days.

And, um, the CAPTCHA solver at the end of the, at the, at the end of the world. Yeah, exactly. The last human job.

Yeah. Well, never a dull moment. We got Sonnet 5.5 out in the meantime as well. Um, I'm not even sure, you know, what we can say about that yet. Needless to say, it's faster and cheaper, and people say it's, you know, almost as good as Opus 5.5. It's pretty much priced in. I don't know. That seems like, you know, it was just a question of when. The big other rumor, which I'm not ready to believe just yet, but definitely one to watch, is Gemini 4 is coming and it's going to be better than everything. There's been some allegedly leaked benchmarks. I'll take the under on that. My guess is that those will not prove to be true, but I think they're coming out of Google. They've got an unbelievable amount of compute. Who knows?

They might be true, but then it might be a larger size model. The problem is Opus 5.5 is a small model and Astra is a big one, and Opus 5.5 almost gets to Astra. So I think that's the real issue.

The leaks, if they're leaks, do include pricing, and the pricing is also lower, which is part of why I was like, I don't know, that doesn't seem that likely that Google's going to come suddenly leapfrog everybody and do it at a lower price. I mean, yeah, so that would be a big shake of the snow globe. And I can't say I feel like that is immediately likely, but we'll have confirmation soon enough, I suspect, as to whether or not my intuition is on point there.

And speaking of intuition on point, I have an intuition that, you know, we will have a very good Wednesday as well. We're back on Wednesday. And on that note, Nathan, good morning.

Thank you, Prakash. Talk to you soon.

Bye-bye.