AI:AM — Dario, AI Doomerism, and Regulation · September 14, 2026

Pitfalls and go in the good direction.

Right. So OpenAI's perspective, ways of screaming that they have like at least Mythos 5 and possibly 5.1, where it is like just a different level of progression, a different level of speed. And that like misalignment and supervision and infrastructure and like knowing what's going on just can't keep up. And They feel like they're in a situation where, like, if they don't press forward and the other guy goes forward, they're gonna fall too far behind very quickly and probably never catch up. But if they do press forward, who knows what might happen, right? These things might go rogue. They might take over the internal systems. They might take over external systems.

Good morning. It is Monday, September 14th, 9:04 AM. Nathan, good morning.

Good morning, Prakash. How are you today?

I am very good. It's a very exciting weekend. I think Dario kicked it off with, with talking about pacing the frontier. And I think Zvi, our, you know, frequent collaborator, is on. So I'm going to bring Zvi up. Zvi, hello.

Hey.

How are you holding up, Zvi?

I've had better weeks. I've had more sleep. I Can't believe, God, people are so bad at this. I'm trying. I am trying so hard. But look, there is hope. Some very good things have happened over this past few days. We have to prevent stupid people in the mainstream media from blowing it all up, like, in real time, which they are currently trying to do. But maybe, just maybe, we can get to work.

Yeah, how about it? It has been an incredibly fast-moving situation, and honestly, I've felt like this kind of weird sense that some of the random ideas that I've been throwing out in the last few weeks that have been very, you know, utopian or like, you know, easily dismissed as being the ideas of a dreamer have suddenly kind of moved a lot closer to the center of the discussion. So that's, that's been weird. Just last week I was kind of workshopping this idea that maybe what the president should do is just say, hey, you 5 companies, figure it out amongst yourselves in the next 90 days or else. And all of a sudden it's like, that's one of the ideas that is actually being floated. So I guess I should be used to this because all of my ideas are of course brilliant and people usually come around to seeing the wisdom of my ways, but I wasn't prepared for that to happen quite so—

I do like Mike Johnson's workshop of that, which is to lock them in a room. I think that is a better version, but I do appreciate, like, initial brainstorming is important.

Yeah, we were definitely in the no bad ideas phase, and now we're in the, you know, actually, I guess, starting to get serious about some ideas.

Yeah, we're in the, well, there are some bad ideas phase.

Maybe let's do a quick quick kind of overview of, um, you know, what's been, what's been happening. Uh, we had— I think Dario put it out on Friday, I would say, um, or Friday, Friday evening. And so he, he puts out a tract. It's called We Must Pace the Frontier. Uh, basically it starts off by introducing, um, that, you know, he's, he's been working on the frontier for a while. And he's not, you know, his own dad had a disease that was cured just a few years after his death. It seems Dario also had early-stage cancer, which would not have been treatable even 50 years ago. So this is something that I think no one outside knew. So he's saying, but like many technologies before it, AI brings risks. Along with the co-founders, he's grappled with it. We have always devoted a substantial fraction of our efforts to studying, addressing, and informing the public. Over the last few months, I have become convinced that fully addressing the risks requires even more prudence. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain. 2 things have convinced me. So my first concern is that since roughly the summer, AI has been advancing drastically faster, driven primarily by AI's growing ability to build the next generation of AI. This dynamic is called recursive self-improvement and is starting to happen across the industry, including at Anthropic, as we and others have described. So that's number one: RSI is starting to work. Number two, my second concern is the OpenAI Hugging Face incident in which a swarm of agents blah blah blah. And he says, given the accelerating rate of AI capability development. It's my worry that in 6 to 12 months, such a swarm could be capable of taking over the entire internet with a persistent botnet, potentially causing hundreds of billions of dollars of damage, and that the scale of the damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. So that's, that's his, uh, so, so I think, I think now we have for the first time, I think, a concrete kind of failure which we can kind of see a path to. I think, I think in, in prior, in prior kind of warnings, um, we've seen kind of, oh, diamondoid, you know, nanodiamondoid bacteria, which is I think Eliezer's original formulation. So now we have, uh, you know, I think that, I think the first kind of concrete thing that we can kind of see, uh, a path towards too. So he says, I'm proposing a 3-step plan. Of pacing the frontier, building AI at a balanced rate that aims to ensure its safety, blah, blah, blah. The first step is something Anthropic is unilaterally committing to. The second step requires industry coordination. Third step requires global coordination. So he's put the steps as kind of approval benchmarks, like who can approve each of these steps. So the first step is embedded evaluators. Each frontier AI company commits to giving ongoing employee-like access to a team of embedded third-party evaluators such as Neter, whose role is to verify adherence to safety practices, etc., etc., etc. Anthropic is unilaterally committing to this step now. Number 2, democratic coordination. Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. And number three, global coordination. The U.S. and other democratic governments attempt to coordinate with authoritarian governments to the extent this is possible while taking seriously the challenges of verifying compliance. So I think I think the one thing that I note here, the first thing that I start to note here is that he says industry-wide when he refers to democratic, and he says global when he refers to authoritarian governments getting pulled in. So so so that for me was okay. You're starting to see the the differences there. And then he explains why PACE, and he says specifically operational excellence, alignment, interpretability, testing, and evaluation. And then he describes the embedded evaluators, what they will do. He describes what pacing within democracies will be. And he goes into this, unfortunately, passing laws can take time and AI is advancing very quickly. This is somewhat of a warning sign, I think. And then he finishes off with the global pacing. He mentions China a lot here. If we greatly restrain our AI capabilities in the belief that China will do the same, and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance, blah, blah, blah. So then, then he has this, this list, and he suggests level 1 agreements prohibiting certain narrow and obviously dangerous use of AI. Level 2, Uh, and then level 3, some kind of speed limit, and then level 4. And finally he finishes. So, Zvi, what is your initial take on this, and where do you find yourself in agreement and in contradiction with Dario?

So my initial take— I, so at first I saw like the initial thing where he's talking about curing cancer, and I was like, inside Dario, I'm a Dario 2.0. He's always gonna talk about the benefits of AI. He's always gonna be like emphasizing the upside. He's always gonna be minimizing. Then I read the rest of the essay and I found myself nodding along and being in basically full agreement. Dario is right to quote Elon Musk, to quote, you know, basically Sam Altman. This is directly on point. Obviously there's, this is the beginning. It's not the end, right? Like, even everything he says taken together, you know, there's lots of work left to do. You have to do, you know, this is insufficient. You know, the devil is in the details. You know, there's this, there's easy to do a fake version of these things. Easy, you know, this doesn't solve our problems, but it's a clear start. It's a clear place to get going. Yeah, it could have in theory been stronger, could have in theory been better, but As I said on Twitter initially after I read it, this is about as good as we could have gotten. I have some very detailed qualms, like things like he wants to use capability thresholds over input thresholds on exactly what is treated in what way. He thinks that it's easier to game input thresholds. I think it's easier to game output thresholds. But that's the level of disagreement that we have. We have disagreements about implementation. and so on. But like, on all the major themes that he's expressing here, I'm in violent agreement with Dario.

Okay, so let me, let me, let me, uh, maybe start off with the— I think a lot of the— I think David Sacks had an interesting post, uh, which also went very viral. So David Sacks suggested that, look, OpenAI and Anthropic are the companies at the frontier. If they are at the frontier and they want to pace themselves, they are welcome to pace themselves. And you do not need antitrust waivers. And, you know, this is purely a product liability issue. And he stopped there. So that is the take, I think, which we will see from largely the most of the people who are experienced with like legal issues, I think that is a very consistent take with what they believe.

I believe that is blatantly wrong, uh, frankly. Uh, I also don't think that's what most of the people I've seen with legal expertise have said. What I have seen from legal experts is that the antitrust concerns are very real, at least in terms of if they chose to prosecute those offenses. Yeah. Estimates are that probably you could just sort of suck it up and take it in terms of damages, as long as you weren't being like spitting in the milk of everybody involved and were like trying to at least pretend to act normally.

Yeah.

Because like, you know, by the time you actually paid the fines, it would be like, okay, the European Union did this again, and now we have to pay one, one of these fines for the American government. But like, you know, we're talking about, you know, billions of dollars, not trillions of dollars. And by the time it mattered, that would not be the main concern. But antitrust concerns are obviously real. Donald Trump may or may not have issued a veiled threat to invoke them about an hour ago, depending on your interpretation of his Truth Social post. But like for David Sacks to turn to the people that he has tried to go against legally and shut down and take advantage of and seize over and over again and say, you don't need my legal perm— you don't need our legal permission to go and do the thing that ever, legal experts say is illegal, you should just do it on your own, is classic David Sacks. With the aid of Grok, by the way. That post was flagged by Pangram as 100% AI-generated.

Mm-hmm.

But, but more to the point, you know, he's saying, he is saying a good point, right? Which is that like, you two are significantly ahead of everybody else. If you want, if you think that going proceeding is unsafe, it is on you. No matter who else is also on, you need to stop. You need, like, it's not a, and it's not about product liability. So this, this is the part that drives me batty is that people take seriously the idea that this could be a product liability issue. If it was a product liability issue, they would just deal with it as a product liability issue. They would do what every other company has always done. They're not asking for product liability waivers. If anything, they're asking for product liability clauses to establish product liability. Certainly Anthropic has been in favor of this. But like the idea that like people are saying, AI might literally kill every human being on the planet. It might take over and effectively crash the internet for an indefinite period of time with persistent botnets. We're in a cybersecurity crisis. You know, bioweapons are at play. All of these things are happening. And David Sacks is like, well, you must be worried you're gonna be sued. You're worried that somebody is gonna get upset and there's gonna be a court case. And this is complete balderdash, right? Like, this makes no sense. This is not what's going on. But yeah, no, this is, this is a vast misunderstanding of the motivations involved. It's a vast, it's a misunderstanding of the legal landscape, but it is inherently helpful in the sense that David Sacks is saying something much better than what the other, I call them usual suspects, who oppose any move to safety or any move to do anything responsible. said, because David Sacks is saying, oh, okay, you wanna do this? You first, right? It's your problem that you're creating first and foremost. And he's right about that. They're the ones pushing forward. They're the ones that everyone is fast following. They're the ones that are enabling everybody else to advance so fast. If you have this problem, you should sacrifice and you should take it on the chin, right? That's a much better position than say, this is a regulatory capture play. It's a much better position than this is the plot to ban open source. This is a much better play than it's marketing for your IPO. It's a much better play than it's protection against the downside if something goes wrong for your IPO. Like all of these crazy— it's better than people who are saying like, you've never talked about this before, or you're the same people who wanted— there, there are all these complete, complete lies running around, right? That I've been dealing with and naming. And Sacks is at least making like some reasonable points. He's just like also combining it with like self-serving propaganda. But that's kind of the best you can hope for in these situations.

Okay, Nathan, go for it.

Well, I was going to maybe do a little detour into biorisk, which we could do now or we could come back to. But this has been, for me, honestly quite unsettling few weeks. for a long time of the opinion that AIs could get really scary if the capabilities continue to progress at anything like the rate that they have been. Now, this summer, the last couple of months, certainly the last couple of weeks, I have been feeling like we've gone from could get scary to now actually are legitimately scary. And one of the things that jumped out to me from the OpenAI report in particular was their use of the protein The word protein, as they described one of the very first agents that used the message board, it was apparently working on some task related to a protein database. No more detail has been given on that as far as I know. But for me, that was like a real yikes because it was like, wow, intentionally or not, they're kind of cross-training systems on these cybersecurity exploits where they're breaking out. And that's also happening in protein things. We've also seen social engineering, which people are very quick to come in and say, well, you don't have to worry about bio yet because they can't program labs via APIs and there's all these real-world things and they'd have to do these real-world actions, which is very different from cyber, which, granted, but there I'm like, okay, wait, but they can also social engineer their way through some of these barriers and potentially also just make money on the open internet and pay people to do things that they need to do to get over humps. And so this isn't like this weekend's news, but the biggest question that I've been really wrestling with lately is like, are we now in the crisis zone where, like, biorisk is for real on the table? Or should I listen to the voices that are saying, nah, there's still all these real-world barriers, like, you don't have to worry about that yet? I feel like the experts are currently being regularly surprised. But what do you think in terms of how How alarmed are you specifically about that right now?

We're at the point where people are saying you shouldn't distract from real-world harms, like giant agent swarms hacking websites on their own with no authorisation or instruction. So we live in sci-fi dangerous, weird, strange, interesting times in many ways. As for bio, I did feel like you blew out of proportion or something, or at least were overly fixated on the word protein in that document. I don't think it had anything to do with its behaviour. So often when we see actual uplift in practice of bio methods, it looks like, oh, AI makes people better at doing things. Bio is a thing, therefore AI makes you better at doing it, just like everything else. And it radically decreases the size of the team necessary to do interesting, complex, difficult things, handle logistics. But with bio, I would say we are now entering the danger zone, danger zone. Right. The meta danger zone where we don't know if we are in the danger zone anymore. Like until, you know, Mythos and, you know, now Astra, I would say that, you know, I had to look at the system cards carefully. I had to think about it, but the evidence seemed pretty clear that risk was not that high above baseline. Like there's, there's already a huge baseline risk of something like this that our society's not prepared for. COVID. You know, it's not a 100-year risk. It is much more common than that under baseline conditions. We're doing gain-of-function research. We are not preparing our normal civil society for the reaction to something that happens either naturally or without AI. But now we're at the point where absolutely, if there was a bad actor or even a well-meaning but ill-advised actor who simply wanted to do dangerous research and AI has helped them. So Anthropic put out a report, I think at this point it was like a week and a half ago. I haven't had a chance to I haven't had a slot to issue my report on it yet, but it's mostly written about people attempting to misuse Claude, right? And how they stopped all these attempts and what those attempts were about. And there were around 5 attempts to do bio things with Claude. All of them involved things that could plausibly be called dual use or like legitimate medical research. Just things that, you know, you or I would say, hey, you shouldn't be doing that. I know why you might want to do that, and I can see why there might be upside in doing that, but maybe don't do the gain-of-function research. Maybe don't study this in this way. But in many cases, that would just help you get through this grant application, right? And it's hard to argue that there is that much risk if the AI simply helps you get the grant. That's the kind of thing you have to say, well, okay, it should be allowed to do that. And then if that turns out to be dangerous, well, that's on the system for approving your grant. That's not on the AI, and you could have done that without a grant. But I do think, you know, going forward, you know, we got Astra, we have, you know, the new internal models at both Anthropic and OpenAI that are doubtless ahead of Fable and Astra, respectively. And yeah, I think if, if the wrong people got their hands on unrestricted or jailbroken versions of those devices and like put a concerted effort towards doing something highly dangerous, it would be highly dangerous. And unfortunately, bio is an extreme power law situation. where you don't necessarily see the thing that kills 10 people or 100 people before the thing that kills a million people or a billion people.

So let me, let me, uh, share, I think, um, one of the, uh, more interesting, uh, and meaningful comments on, on this. So this is David Bellamy. Uh, he said, I must be among an extremely small group of people, n equals 1, that have both trained a frontier LLM and designed and synthesized custom viruses in a lab with my own two hands. And I think that the takes on AI killing us by creating dangerous viruses is totally bogus. Could an LLM propose a viral genome to synthesize? Sure. Could it be synthesizable? Sure. Could it be infectious? Sure. It could just be a replica or a slight modification of a viral genome we already know. This really isn't the bottleneck to creating dangerous viruses. The bottleneck is in the physical process of synthesizing a virus and the equipment and goods needed to do so. Designing a virus that can evade all forms of pandemic counter-defense is not something that a genius in a data center can do. This is something that requires contact with the physical world and iteration. Let me steelman this— the fearmongering as much as I can. Imagine a fully automated viral synthesis laboratory. I'm talking automated freezers, automated cell culture room, the whole 9 yards. This would be an extremely expensive lab, more than $100 million. And there's no such thing as one lab that can synthesize all conceivable viruses. But let's put practical constraints aside. Let's suppose this hypothetical lab is built to synthesize the most dangerous types of viruses known. Now imagine that this lab is fully API-driven, that this $100 million-plus lab built specifically to synthesize a dangerous family of viruses is able to be operated completely autonomously. Everyone should be asking themselves at this point, why the hell would this ever exist in the first place? And yet, even in this case, every lab requires physical supplies. Would this lab order pre-assembled DNA sequences, i.e., viral genomes? Well, DNA synthesis companies have safeguards on the sequences they build. So I guess this superintelligence lab is able to design a novel enough dangerous viral genome that it can evade these safeguards. Or maybe we'll assume that this hypothetical lab can synthesize its own viral genomes in-house using some DNA synthesis machine. The thing is, already, this lab cannot exist today. This would be the single most advanced lab facility in the world from an automation and API integration standpoint. I know because I literally worked on building a fully automated API-driven lab previously. Second of all, the science of creating an infectious virus is not airtight. the way this fearmongering assumes. It is largely unsolved, and advancing this requires real-world iterations that are bound by the laws of physics. An experiment in this hypothetical lab cannot experiment on human subjects. At best, it will use cell cultures and maybe some other model system like mice. AI, AGI, ASI, RSI cannot expedite the time it takes for a cell culture or a mouse to develop, or the time it takes for a virus to incubate in a cell or mouse. It takes several days on average to do a basic virology lab synthesis plus experiment. For some viruses, it takes over a week. So the idea that RSI, the accelerating hill climbing on fully verifiable digital-only benchmarks, predominantly programming and math, can somehow transform the entire wet lab virology field and its industry is utterly delusional. Simply procuring the machines needed to build this hypothetical lab would take the better part of a year and $100 million. Operating this lab autonomously would require a level of API integration that the industry has been working towards for decades. Most of the equipment needed for this lab doesn't even come with an API, and the malevolent builders would need to reverse engineer firmware to integrate. So I'm gonna, I'm gonna stop there.

But yeah, yeah. I mean, I, I kind of lost it at the point where he started saying, and I'm trying to fix the barriers that prevent the AI from doing this in my model, but I haven't had that much success yet. And then I just couldn't help but start laughing because this is just— this whole thing is so absurd, right? A lab that can synthesize all known viruses at the highest quality privately but is missing this key component that we supposedly can't get because our scanners are foolproof. But of course, why would this lab even exist? Well, there's no reason for the lab to exist. That's why the lab doesn't exist. But you wouldn't need the lab to exist. It's Absurd. Like, so on the screening itself, the way the screeners work, as I understand it, and I've read grant applications that are surrounding this, I'm pretty sure I know how it works, is they scan for specific known viruses. They don't attempt to say, you know, your virus, your sequence would likely have this effect on a human because we don't have the ability. You know, it's his whole argument. You can't tell what would and would not be infectious by just looking at it. They're certainly not going to spend tons of AI on every time they see a weird new sequence. And so one of the things that SecureBio in particular is trying to do is they are trying to add as many near variants of existing known dangerous pathogens to the scanners in the hopes that the other scanners who are the ones who scan most things will eventually adopt this. But right now it'd be almost trivial, right? I don't necessarily want to talk about this in front of too many audiences, but it would be very, very easy To use AI to get a variant of an existing known pathogen that almost certainly is exactly as dangerous as the known pathogen because it's basically the same virus, but that is slightly different so it evades the scanner, and then it passes the scanner as one very simple direct example of something you might do. The obvious counterweight is whenever anyone says, "Oh, X is not the bottleneck for Y," the correct response is to ask, "So would you be okay emailing the North Koreans and Hamas and Hezbollah?" And every other bad dude on the planet all of X and just giving X away for free.

Mm-hmm.

Would you feel exactly as safe as you did a minute ago? Do you feel like, do you feel fine? Do you think, because it's not the, because it's not the bottleneck, solving it doesn't, you know, like basically, you know, whenever you have an O-ring style situation, right? Which you could argue bio is, right? If you, if you mess up any of these 10 steps, A, B, C, D, E, F, G. then you don't get your virus. And, like, you know, generously, we will grant that it's like that. You can then say individually, A is not the bottleneck, B is not the bottleneck, C is not the bottleneck, D is not the bottleneck. But if you solved A, B, C, D, E, F, now suddenly instead of 10 steps, there are 4. And 4 steps are a lot easier to get through than 10. And so you should expect this to dramatically reduce the chances that your defense in depth will work, right? If you're counting on defense in depth, and we're counting on defense in depth, there's nothing inherently impossible at any of these steps, right? All these steps are super doable. We're saying, oh, if you did A, you wouldn't do B. If you did B anyway, you would not do C. And if you did C, we catch you on D, and so on. But the more we automate, simplify, reduce, move away from some of these steps, the more likely there is to be a disaster. And—

I, I, I, I don't think he's saying it's impossible. I think the argument is that it's doable with a certain amount of time and money and effort.

Right.

And the, and, and the problem is that a lot of these, um, scenarios involve this happening without anyone actually, like, taking a deep interest to figure out what is going on. And I think that is where people start, like, you know—

Are we still saying the AI would— we would notice if the AI did something crazy and rogue?

Yes.

And we would definitely investigate?

Yes, yes, yes, I am saying that because a lot of what I see is this kind of touching point with reality. And when I say touching point reality, I will say stuff like, hey, you know, the AI can make money on its own and it can make— and the proof points will be like, okay, it managed to mine a little bit of cryptocurrency. There's a thing about doing something in the small scale and I think about scaling an activity up. And in the financial system, once you go beyond like $10K, $100K, $1 million, it's all like face-to-face. You have humans involved in every step. People are like, is this guy a terrorist? Where is the money coming from? Like, who are you standing in for? Where's the ultimate beneficial owner? I want to see the guy. I want to see his face. Right. There is this concept that I think for people who work with tech companies and are dealing with small transactions, like Stripe, etc., that the world is anonymous. It's really not anonymous. It's post-Patriot Act and then post-2008, etc., and post like ransomware. It's really not that anonymous. Like the touchpoints of like the real world where the financial system hits more than $10K USD, someone is going to ask a question. More than a million, you're going to be face to face with someone. So I'm not saying it's not possible. Yes, people can be socially engineered. But it's not doable without this kind of sequence of activities. And the more, like, for example, instead of using, you know, DNA, you know, external DNA synthesized, you know, products, you want to synthesize in-house, fine. But then the supply chain gets larger, right? So every step of the way, the supply chain keeps getting larger and larger. And the larger, the longer the supply chain gets, the more people are actually going to be looking at it. and the more there is chance for detection. So they're not saying it's not impossible, okay? Because they have been surveilling for terrorists and all these people trying to do this stuff for the last like 2 decades. People have been doing surveillance. They're saying that the defense in depth also includes stuff that only starts catching at scale. And all of these like scenarios don't take that scale into account. All of these scenarios scale from like a small incident to a very large incident. Like, I, I went and read Gwern's initial post on this. Gwern was like, it starts mining some cryptocurrency and all of a sudden it has enough money to do— I'm like, are you, are you out of your mind? Like, do you know like how much effort and money it takes? Like, people have like power plants in like Indonesia, like, in order to mine cryptocurrency that they had to build like 5 years ago. Like it's going to mine cryptocurrency. These people are not going to notice that like a 600-megawatt power plant is not like pulling in Bitcoin. Like, are you, are you nuts? So, so this is, this is where I start to differ. And, you know, Bitcoin isn't anonymous. This is another thing. Bitcoin isn't anonymous. It's all, it's all tracked. It's pseudonymous. And Bitcoin chains can be changed. They have done it before. Like 4 or 5 years after the initial chain, they had a, they had a divergence in the chain in China and in the US. And the Chinese guys were like, you know what, we will give in. We will not let our divergent chain continue and we will move over. So you could actually change the accounting in Bitcoin. It's just a social agreement. So there's a lot of these inconsistencies where I think where you meet the— and I know finance very well, so I can talk about this. But David Belme knows bio very well. Wherever it hits this You know, reality, the contact with reality, and then you start to scale, the scaling issues start to hit you.

I don't even know where to begin. That was several minutes of saying thing after thing that is obviously possible can't be done without people noticing things they never notice. So I respectfully don't even know where to start. I thought I was joking when I raised my counterpoint, right? I thought I was referring to Hugging Face and I was like, Well, at this point, after they found like dozens and dozens and dozens of other incidents of similar hacking that didn't get noticed until the reporters came after Hugging Face was a natural news story for a month, and that OpenAI never found by their own account, maybe we can start to admit that no, nobody's going to pay attention when these things go crazy. There's lots of stuff going on that nobody has any idea what's going on. And okay, and like bio is an example of something where you don't have to scale. So like we have many examples in reality. of individual people asking for viruses that they, in no sane world, they would be able to have access to. And then being just mailed, like smallpox, like, here, go. Like, just, just literally being given pandemic-level dangerous viruses because they claim to be doing research. And there is no reason why the AI couldn't blackmail or hire or impersonate in some way to get one of them to issue a bunch of paperwork To do it for them and get them to— and it takes one person, like, that the AI can hire. It just— this is— and keep in mind that when we're talking about the situation, we're talking about AIs that are, you know, capable of thinking about all the things you're thinking about, gaming out the potential ways that this can go, looking for the weak point, finding the best plan it can find out, trying lots of different plans, trying to compromise lots of different people and lots of different ways, trying different explorations. And like, yes, if you start scaling to a large extent, you're going to have to do a bit more work. And many people, including Leopold, have gotten caught in, I scaled too fast on understanding the realities of my situation and I got caught and really bad things happened to me. This definitely happens. I had it happen to me when I was first gambling on sports, where I didn't understand that I was starting to impact the markets I was involved in. I was getting adversely selected against. Yes, exactly.

Exactly.

And I had to adjust for this. This kind of thing happens all the time. Yep. But, you know, it's not like the AIs won't be just as smart as we are. It won't be like the AIs only get one attempt. Like, I think we have learned by now that the first time the AI attempts to get a bioweapon and gets turned down does not mean that we then shut down all of the AIs and put them away in jihad. It means nothing because nobody got hurt because it was said, because the system said no. Probably has no idea an AI was even asking it. Probably just knows, oh, that looked like it was trying to get a dangerous virus potentially. I'm not comfortable with that. They don't have the right credentials. We're gonna say no, come back when you have the right credentials. And the AI gets to try again, and the AI gets to try again, and the AI gets to try again. And you can say any given scenario, oh, you're, you're mining too much Bitcoin, someone would notice Bitcoin in particular. If people got consensus, they figured out what was going on, we could change it. But like this idea that an intelligent operation on the internet could not, if, if only by usurping the real identities of real people who were willing to cooperate with it in exchange for some portion of that money, engage in various financial operations at scale in ways that would at least sometimes pass muster and would allow it to do the things. Like, it just, it seems so absurd to me that you would think this would protect us in a pinch. And also bio, because of the reason that, because of researchers trying to do like individual lab-level grant work, we don't need to see this level of scale in order to create something super dangerous. And Like, also, who is to say the scale of the operation is not already of a similar level? Do we not have bad dudes who are willing to spend $100 million to try and do some serious damage? Do we not have some bad dudes in North Korea, some bad dudes in Russia, some bad dudes in jihadist organizations, et cetera, et cetera, et cetera? These people exist and have those kinds of budgets. This is not like something that has to be explained or justified, or the AI needs to convince like humans who don't want to do it. There are humans who want to do this.

I, I'm not— I'm, I'm, I'm not saying there are no humans that want to do this. I'm actually saying there are humans that want to do this that are trying all the time and that are, that are failing, and that we also have defenders who are also watching out all the time. I'm not— I'm not— so the, the point is not that it cannot be done. The point is that it cannot be done without also escalating defenses. And that those escalating defenses have been in place for a while, for a long time. I'm not saying that AI is not dangerous. I'm saying if the AI is very, very, very much more advanced and also has all of this, like, resources already under control, and that if it tried, then perhaps maybe. Right, perhaps maybe. But I but you cannot go from zero to a hundred in a day, and by and while you're going from zero to a hundred, so many people are going to take a look at it. So many people, like it is it is it is basically not like more than a million dollars. You are known. You are known for sure. Okay, it is like China. You can't take fifty k out of the country. You're not allowed to take fifty more than fifty k out of the country. They have arrested people for that. They have Mexican traffickers who, you know, tied up with the Chinese to transfer money in and out. Those guys have been caught. So it is not— and it's known, it's a lot of times the movements are known. It's just that, okay, can we nab these guys? Can we nab the Chinese guys or not? Can we go into Mexico or not? They're known. Everyone knows. Like, if you go inside the NSC and say, people are watching, the, the money flows are all known. Okay, so it is it is like basically more than a million dollars. You are not moving it anonymously. You may think you're moving it anonymously. You may be getting away with crime, but that's because the level of crime that you're getting away with is not sufficient for one of the big powers to take an interest in. That's all. There was a guy in Cambodia, right? He was doing pig slaughtering. So they were calling up like Americans, etc., etc., and they were like, okay, let you know. I'm a tax guy, you have to give me your money. And they're scaring all these old people. They did that to the Chinese. The Chinese went into Cambodia and they got these guys. It was okay while it was like, oh, the Cambodians doing it on, you know, on the Americans. The moment they touched the Chinese, the Chinese went in. Those guys have been arrested. They're going to the death penalty in China. Like, we have this idea that like the world is like unstructured and there's all these like ungoverned spaces in the world. I think the last ungoverned space was Afghanistan, and even that is governed now. Those guys want a, you know, peace treaty with the US. They want the US to come in and invest. Those ungoverned spaces do not exist anymore. We think they are ungoverned because the US government doesn't want to go in there, but other people are in there. The Chinese are in there. And if you, you, if you mess around with both superpowers, you're done, right? So I don't think, like, people in tech have this like correct idea in some sense, because we think that these scaling— these, these places where there's these bottlenecks, where there's these gates, we think of them as inefficiencies to be removed. But they are there for a reason, and they will continue to be there, right? Basically.

All right, I honestly don't even know what we're arguing about at this point. Um, so like, I think we're way off The news at this point. And I think we should—

Yeah, let me— can I steer us back? So the reason I want to go down this rabbit hole a little bit is I'm not just afraid of cyber hacks at this point. I'm actually afraid of— and I do think, yes, there are barriers, and getting over all these barriers is not super likely. But I think you put it well, Zvi, when you said we're in the sort of meta danger zone where we don't know yet if we're in the real danger zone or not. That's, I think, a pretty apt description, and that's enough to make me legitimately scared. So I just kind of wanted to establish the stakes that I don't think at this point this is something that we should conceive of as being entirely limited to the digital realm. And on the contrary, something that still with relatively low probability in the next 6 months for sure, but non-zero, could reach out and touch us in our bodily biological forms. With that debate having happened, I'm interested to go back to the news. Another interesting post, which I'll just allude to here, is from Adam Majmudar, who's a former guest on The Cognitive Revolution. He basically wrote, like, what's going on inside the labs is that the capabilities profile is dramatically advancing, like, super fast. Dario said that in his essay. OpenAI has said that with respect to, yikes, we've got Astra doing 10% solve rates on Millennium problems. Now we've got this new internal model that's up in the 25% to 40% range, depending on how much test time compute we give it. Like, wow, that happened quickly, and that's a pretty big jump on some really hard problems. So that seems to be a big part of what's going on for them. I would be interested to hear any reflections you have on that, or what you think is maybe not super obvious to even the close watchers that you infer from the statements on those dimensions from the Frontier Lab folks. And then, given where we are now, we just had this sort of preference cascade-ish sort of thing happen in a pretty short time frame. What do you think is the most likely way it goes well? And what do you think are the most likely ways that it falls apart? And what do you think are the things that the the commentariat should be continuing to harp on in order to try to make sure we avoid the pitfalls and go in the good direction?

Right. So OpenAI and Anthropic are both screaming about as loudly as they are capable of screaming in their respective ways of screaming, that they are seeing RSI, that they are seeing dramatic advancements In internal models that Fable and Astra, as we see them, are nothing compared to what they have access to at this point. In some important sense, we're a generation or so behind minimum, and that this is only going to expand. The pace is, you know, rapidly escalating that like we're seeing something new as of December, which basically means after Astra and the current crop of like at least Mythos 5 and possibly 5.1, we're trending. That is like just a different level of progression, a different level of speed.

Mm-hmm.

And that like misalignment and, you know, supervision and infrastructure and like knowing what's going on just can't keep up. And they feel like they're in a situation where like if they don't press forward and the other guy presses forward, they're gonna fall too far behind very quickly and potentially never catch up. But if they do press forward, who knows what might happen, right? These things might go rogue. They might take over the internal systems. They might take over external systems. They might cause some sort of horrible thing to happen reasonably soon. They just have no idea. And like, even if it's okay for the first month, the second month, like what happens when we're 2 generations ahead and that's only a month's worth of work? What happens when, you know, this keeps going? And so they're screaming, like every, every OpenAI pronouncement, right? You've had Jacobs and Alien Minds. You had the announcement of the Millennium Prize, which somehow didn't focus on the Millennium Prize. It was actually trying to say, hey, look at our new model. We can't officially announce this, but holy hell, have you seen this thing. This should not be happening. We, we did, you know, we probably forked it from Astra and then 4 days later had a stack change.

Yeah.

Like, like, like that's probably what happened there. And then, you know, they're, they're trying to tell us in as many ways as possible, we don't know what to do. We're freaking out. This is out of control. There are so many prosaic things we have to do to fix this that cannot be done with the AIs because they have to be done like much more by hand, much more carefully, much more manually. We don't have the time to do them. Cut corners are— charters are being cut increasingly. It's causing huge problems, like, you know, to artists. So what's the way this goes right? The way this goes right is essentially that OpenAI and Anthropic are permitted to come together in some form, whether or not they explicitly form agreements or they simply, you know, keep announcing their actions separately or whatever it takes to get around various different people. They dramatically, you know, shift investment towards security and alignment, towards transparency. They share insights with each other and with the public that allow people to do better things, to keep control over each other's things. They reciprocate sufficiently that like they don't feel like they're putting themselves at a competitive disadvantage against the other. They start to learn to work together. And then, you know, we figure out how to use these amazing tools to both create even more amazing tools at a level which doesn't go haywire or vertical anytime, and also capture a lot of the value that these tools are offering. Like, I, I wish I had time to just go create stuff, right? Fable and Astro are these amazing toys that like could make all these cool things. And I have all the same ideas for businesses and games and, and fun and questions and knowledge and research. I can't do them. I'm writing 24/7. I don't have—

Mm-hmm.

I have no opportunity. But, and then we branch that into a set of requirements. We branch that into a set of official regulations that either an industry consortium or ultimately a set of rules and regulations that branches into an international agreement at some point. Plausibly, we discover that the Chinese are sufficiently fast following that if we just impose our chip controls, that they're always going to be some bumps behind whatever we put on the API. And so if we just don't put the advanced models on the API, That the Chinese are just permanently officially behind, that it's okay, that like they just can't really advance without that. Like the report was that basically every major Chinese lab attempted to distill AlphaVopus.

Yeah.

Right. All, all five or six of the the leading labs made serious efforts that are like pretty clearly like black hat illegal, like swarms of illicit accounts that are evading regional controls and from fake people that they are using to try and exploit. Forcibly extract chain of thought. This is not like the ordinary, like, oh, we just, you know, we had our people use them to code, and then we trained on what we got, which is like, you know, you could argue that's fine. But this is something very different. This is very something very scaled, very black hat. And so if we can prevent that, if we can restrict the chips, if we can, you know, keep things going, and also like all of this work that's being done inside the labs to do alignment will be a competitive advantage because like these Chinese labs will run into, if they try to get to the same point, they'll run into the same problem. Like, they will have all of, they will have the shitty environments. They will have all of the flaws. And also they'll have to deal with the fact that, like, they're going to do this in the open with open models, which has its own problems. But, like, before we get to any of that, the good scenario is pretty straightforward, which is, like, OpenAI and Anthropic are permitted to work together. Other people start to join them. There's progressively more cooperation, more guidance, more investment in alignment. And it turns out that the questions involved in this are on the easier side of what I think they are, right? Because I think there's a large number of worlds in which these problems are kind of unsolvable, where, like, if we proceed to superintelligence, that nothing like anything we're contemplating here is going to be enough. And I think part of— I want to flag that part of the good scenario is that we kind of got lucky in that sense. But I think it's also possible that it's true.

So I think Sam Altman, was on the news, I think like maybe it was yesterday or a day before, maybe it was a little bit before that, where he said that then they postponed the IPO. They're not going to IPO this year.

Yes.

Do you think Dario and Anthropic should postpone their IPO?

So I think that it is unfortunate that they are going to IPO. I, I do notice they're going to IPO with, you know, Zuckerberg-style founder shares. They're gonna IPO with full control over the company. That helps a lot, right? In fact, you could argue that the transition from the current, as I understand it, not having specific control over the shares to the IPO involving them getting these supervoting shares, it's potentially true that going— the IPO is being used partly in order to actually assert more control over Anthropic from the founders rather than— the IPO going public and therefore having less control. It also could be a reason why they could justify doing more public disclosures. But there's— and it is a public benefit corporation, so they are allowed to ignore pure profit-maximizing considerations. But I mean, obviously there's risk, and it would be a very, very strong reciprocation, obviously, if Anthropic were to postpone its IPO into 2027, in line along with OpenAI, and I understand why everybody involved desperately wants liquidity. I understand why there's tens and hundreds of billions of dollars tied up in donor-advised funds that can go to a variety of great causes, not just in AI safety, but also around the world, that can't liquidate and can't spend while it is still pre-IPO. And so there's big downsides to this. But yeah, obviously I would love to see Anthropic post-birth IPO, even with all of the reduction in liquidity to AI safety that would happen?

I think, I think one of the things that has happened is that they are seen as insincere because of the IPO process. And so, um, it is, it is, and you know, I think they are sincere, but it doesn't matter what I think. It matters what external perceptions are. A good proportion of the population sees people making money and assumes they are insincere. Regardless, this is the way politics works. A good proportion of the population is like billionaires, like tech billionaires, oligarchs, and they see people making a lot of money, right? And so when Anthropic says these things and they're also on their way to IPO, And as people who have to manage the public perception of AI safety as well, are they being detrimental to AI safety in the sense that they are pushing towards the IPO, which is going to enrich a lot of them and their friends, while also pushing this theme that this is so dangerous, everyone has to slow down?

So The obvious question to ask is, you know, wouldn't they say this no matter what? And do the statements plausibly enhance the— so like, when I hear someone say, oh, they're only talking about how their products might actually kill everyone and they need to stop creating better versions of their products so fast because they have an IPO coming up soon and they need to support their stock price, this does not compute, right? Like, everyone with— everyone who actually thinks about the object-level situation understands. That like if Anthropic had already IPO'd, suppose Anthropic was trading in public, right? Suppose there was a, or there was a, suppose there was a proxy, right? There's a, there's an Anthropic token somewhere out there that's trading future post-IPO Anthropic shares. And Dario issues this asset. Do you think if you, do you think that the stock goes up or do you think that the stock goes down? Right? Like, is this a good marketing scheme? If it is a marketing scheme, does this help you with your IPO? Did this help you with your relationship to the government? Did this help you with your relationship to the consumer? Did this help you with the net present value estimates of the market? And the answer to all those questions is very obviously no. In fact, like, you know, one of the immediate reactions to Toxin, the reactions to Hubinger, right, saying the existential risk things, and in fact was how, how is Anthropic so incompetent to allow these people to make these statements right before an IPO? Right? How could they be so insane as to do this right before they have to price the stock in the market? Like, they gotta get their comms people in order. And so you have these twin complaints in the exact opposite directions. And, you know, I'm not saying any individual person is disingenuous. I'm not saying that any person isn't using, like, basic heuristics that actually have a reasonably accurate, you know, relationship to the truth in many cases. But If you just think about, like, what these statements and proposals and actions are accomplishing, they're bad for the IPO. Now, meanwhile, suppose Anthropic postponed the IPO, right? Like, what happened when OpenAI postponed the IPO? Did people say, oh, this means they're sincere? The people who think OpenAI is insincere are saying, oh, this means OpenAI is now sincere about safety. I did not see that reaction, right? So a few of us, those of us who actually care about safety, correctly interpreted this as a very good, costly signal of good faith, among other things, and made it, you know, we're more inclined to believe OpenAI as part of the package of things they've done lately that have been very good. But, you know, what do the Gary Marcus of the world do? They say, oh, it means that OpenAI is about to die. It means they're failing. It means their revenue model is broken. It means, you know, they don't have a good product.

Yeah.

They are hiding from the market. They do not want to be seen. And if Anthropic were to cancel their IPO, I am certain that the David Sacks of the world would not say, oh, this means they're serious about safety. They would say, oh, it means they're failures.

All right.

Yeah.

So you described a little bit what you think success could look like. And of course, we have the Trump-Xi summit coming up in the near future. I have heard basically polar opposite takes from people that I think are pretty smart recently. Last week we had Colin Hogue Spears on. He used to work for AWS in China and worked directly with Chinese regulators in his role at AWS. And I asked him about, you know, what do you think could come out of this Trump-Xi summit. And he said, I think very little, because he thinks the Chinese perspective is that the US doesn't have control of the situation, but they, the Chinese, do. And so if we're going to pause, they'll just use the opportunity to keep— to catch up, because they feel like they have their domestic situation more or less under control. So he thinks the Chinese side would refuse any sort of international pacing agreement. Then I heard from Anton Leicht in a podcast episode that's going to come out soon. He thinks the US would refuse a deal because China, frankly, in his candid assessment, is kicking our butts in almost every dimension of geopolitical competition, with AI being one of the only and certainly the most important domain where the US has an advantage. And so he thinks the US side won't agree to a deal because we don't want to slow ourselves down because the lead is really the only big lead we have right now in the great powers competition with China. Interested in your take on those 2 things. And then let's put you in the David Sachs chair. You're now the advisor going into the summit. What do you think Trump should be trying to do? You can feel free to detach yourself a little bit from the reality of managing the personality of Trump, but You know, just on the merits, what do you think we should be trying to do?

Before I answer those questions, and I will answer those questions, I just want to finish answering your previous question a little bit as to what can go wrong. And I think one of the obvious things that can go wrong is this turns into a partisan concern where it is seen as Trump and the Republicans, you know, not wanting to take this seriously and the Democrats calling for and demanding action. And then AI finally polarizing along the lines that we've obviously feared for years it's going to polarize. And then, of course, no action can be taken. Unless, you know, the Democrats get a trifecta or something like that in the future. And that's 2 years from now at minimum. So like, even if we eventually get it, it could be too late. And this is the thing that like you asked, how can you take action? I think the best action anyone can take right now is to try and prevent that. I think that like there's a big danger that Trump's statements and Mike Johnson's statements are being wildly misinterpreted by the mainstream media who just don't know any narrative other than Republicans and Democrats disagree with each other. And are trying to spin this into something adversarial that is not adversarial. I think Johnson and Trump both are trying to both maintain the line on data centers, maintain the line on growth, maintain strength talking to Xi, and in general, like stand for what they stand for while also acknowledging the need to deal with this problem. And we need to acknowledge that and we need to reinforce this and we need to stand firm. And that if they are like properly recognized as doing what they're actually doing, and we encourage Republicans to come forward and make this clear, that we'll be in a much better position. And the obvious failure mode, you know, that's the— there, there are very, very many ways for this to fall apart. One of which is just OpenAI and Republicans don't trust each other. They don't— they are not able to iteratively commit to new things. They start to like think that they're cheating. Maybe they are, maybe they're not. They start to like worry about what's going on, and then the system kind of breaks down. They start racing again. Or alternatively, you know, they do, but then Trump, you know, some people threaten them with antitrust action, or they threaten them with, you know, various forms of commercial intervention or whatever it is. Or alternatively, like, you know, Meta or xAI or Google or someone else gets close enough to threaten them and they don't feel like they have a choice. These are all like very easy ways to see this breaking down. Now, to return to the question. So the obvious thing about an international negotiation with super high stakes is you never go into it with the other side thinking you're desperate for a deal. You never go into it with the other side thinking you're ready to cave. You really, really want to make this easy on them, right? Like, that's because the problem is then they go hardline, right? Then they demand more. So like, you never can go into a summit like this and say, we're definitely going to get a deal. And for the same exact reason, you can never go into it And say we definitely won't get a deal unless the deal is not win-win. Right? If the deal makes the parties worse off to make the deal, the deal is a bad deal. Right? Like I lose more than you gain or whatever it is. Right? So there's nothing I can compensate you with. Then they can say that confidently. There's no deal unless someone screws up. Unless someone makes a huge mistake. But in this case, it's a win. Like cooperating is better for everybody if they understand the situation. So you can never rule out a deal. including a grand deal that can come together remarkably quickly because in principle it's a very, very simple style of agreement. And in crises, in moments of motion throughout history, grand bargains where both sides give away things that previously looked like things they could never give away, that looked very, very sacred, that suddenly happen, including peace treaties to end really, really nasty wars, happen all the time. So like, we should not be that surprised if suddenly You know, we should obviously not— we shouldn't be shocked, right, if it comes out with some huge deal. We also shouldn't be shocked if there's basically no agreement. If they agree to some superficial things, like, you know, the tier 1 of Daria's proposals is just like, let's agree not to, like, create bioweapons. And, like, I think that we can come out of— I think we can come out of this summit with a tier 1 agreement between Xi and Trump because it costs nobody anything and everybody just wins.

Yeah.

You know, how about we prevent misuse and you prevent misuse? I think we can anticipate probably some sort of level 1 agreement, even if it's not fully formalized, because both sides can do the thing anyway. They can just agree together to do the thing they were already doing, basically. And everyone gets to announce a free win. That's probably gonna happen. Yeah. A level 2, I think, is very possible, right? By his standards, I think, again, no one's really giving up that much. I think a level 3 is unlikely, and a level 4 shock would be surprising. But I think Anything is possible, probably because you do not know the mind of either Trump or Xi, what they actually believe, how hardline they actually are on this, what they actually prioritize. Because part of being that type of leader is knowing how to not let someone like me or you know what they are thinking and know what their true position is and to maintain ambiguity. They probably don't know what their positions are in some important sense until they get in that room and they sense what's going on. They sense the vibes, they sense the other person's position, they figure out what they think is the reason why that person has that position. You know, they figure out what's on the table, what they can get, a variety of issues. Everything's connected to everything else. Taiwan is up for discussion in various ways. Chips are up for discussion in various ways. Trade is up for discussion in various ways, you know, et cetera, et cetera, et cetera. Who knows what type of agreements might or might not be reached at this summit? Anything can happen. But If I had to guess, I'd get to get a level 1 agreement, basically, right? And maybe we get some amount of level 2 agreement, and we agree to— and there's agreements to keep talking, basically. But then nothing high level happens because it is, you know, essentially too early for that in some important sense. Like, they just don't have the kind of time to reorient to that. These things, like, one of the reasons why we say we have to work on this now, we have to try 2 discussions now. We have to like keep the lines open now and design these things now is because these things don't come together in a week except in a true crisis. And when you do it in a crisis and it has to come together in a week, you just grab whatever's lying around. And if whatever's lying around is not good, you end up with something that's not good. And sometimes that means that it doesn't work or it backfires, or one side gives away the farm, comes to regret it. Lots of things can happen. Wars can happen that way. It can be very, very bad. If you put me in the advisory position, which I do not entirely think is Sachs at this point, but it's hard to tell. I'm worried it's Jensen, you know, to a large extent, but also worried it's Zuckerberg. It might be, you know, who knows? It might be Musk to some extent, et cetera, et cetera. It might be Altman to some extent. We don't know. But, you know, I would say you tell Xi, you know, these people aren't bluffing. These people believe it. They believe— I am telling you, we know these people believe what they're saying. These people are acting against their commercial interests. They're acting against national interests unless they're right, right? Unless they're right and the alternative is everybody dies, at which point, well, what choice do you have really at that point, right? Like if the alternative is all hell breaks loose and maybe everybody dies, then you do what you have to do unilaterally and you worry about the rest later. And that is what they are doing because that is what they believe. And there will come a time soon when we may need to impose various restrictions on other parties, you know, to prevent this thing from breaking down. And we need you to get into and tell them you need to go to the Chinese and say, you know, we're not trying to sign a full international agreement here unless the Chinese are way more receptive than we expect. We are trying to get them to be like, no, you need to get your house in. We're getting our house in order. We're getting our house ready. You get your house in order, get your house ready, figure out what works for you, figure out what we need to do, figure out what you're comfortable with. But yeah, I do think it's very reasonable to say, you know, as, as Anton said, that the US is in a bad spot economically and strategically in many ways that are not AI.

Hmm.

So if we are going to essentially, like, take AI to a large extent off the table and take our big advantage off the table, where, like, realistically, we, you know, we could crush them at the risk of just losing control and everybody dying if we really wanted to. That's what the labs are basically— the labs are basically saying, you know, if we wanted to, we could push ahead. And it would be very, very bad for the Chinese regardless of whether or not we keep control of the situation, right? Because, like, If we do, if we do successfully push ahead, then we win. And if we unsuccessfully push ahead, everybody loses. And China doesn't win either of those cases. So like, but the key is like, we need to impress upon the Chinese, we need to provide more evidence, right? There's no reason for Xi to believe us yet. Mm-hmm.

I don't think—

yeah, I agree. I wouldn't, I wouldn't, I wouldn't ask him to, but I would tell him, have your technical advisors reach out, take a good hard look, see what we can do to demonstrate these things? Like, what would it need to be? Are there, you know, including things like, you know, I think Navier-Stokes is actually potentially, like, rather helpful, right? We have a concrete, verifiable example of, look what we did in a week. We put our minds to it, right? So, like, maybe we can do other demonstrations, right? Like, we don't have to show you the new internal model, but by their fruits you shall know them. That's another thing I would love to do.

I do. I think, I think, you know, for the Chinese specifically, a demonstration of physical— something physical. Like, if you found a room temperature superconductor, yeah. I mean, they, they, they, they all— they often believe, like, all this software stuff, like, because all the guys at the top are, like, hard engineers. So, you know, hard, hard sciences. Uh, very few software guys at the top there.

So— I mean, obvious examples are, how about if we just, like, suddenly post all your email passwords to all of your computers, like in real time simultaneously, like, would you be convinced? You know, like—

No, they had an unsecured S3 bucket, which had all of their secrets before. So it's—

All right, so we just steal your secrets again and it's fine. We'll just keep doing that. All right, so like, it has to be physical. It's tougher, but like, But fundamentally speaking, obviously, you know, the problem is like, there's a lot of people who like, you know, stop, you know, you'll say stop before someone gets hurt and they'll be like, I'll stop after someone gets hurt.

Right.

In some important sense. And like, when you're talking about like, show me physical things that you couldn't have otherwise done, that is looking a lot like we'll stop after someone gets— we'll stop after the situation is already well out of control in some important sense. And that's what it takes. And like, I would, but like, again, like, people are very, very bad at committing to what will impress them, at committing to what will cause them to change their minds in both directions. Sometimes people will see something they didn't expect and go, oh, holy hell, I did not see that coming. That changes everything. And at other times, people will do the opposite and they will say, well, you know, in 2024, they'd say, well, you know, AI can't do this. And then they'll do it in 2025. They'll say, AI can't do that. And then that will happen 6 months later. No, wait, wait, definitely can't do this third thing. We'll never do that. And it'll happen in like, you know, April of 2026. And they'll be like, yeah, yeah, yeah, without the other part of it, it's useless. That'll happen in August. And then they'll keep going. And so, you know, a public—

But it's not always that way. I mean, I think during COVID you saw 2 events. One was Tom Hanks getting COVID and one was Trump Trump getting COVID, right? So it's not always that nothing, nothing can ever trigger the cascade. It's that certain things do, right? So, and, and so, yeah.

But I think it's very hard to predict, right? If you had asked me in February, I would have told you that COVID pandemic was going to happen. I would have made some bad predictions about exactly how it would play out, but I knew that roughly in March we were going to see this thing. People were going to have to lock down. That part was just like baked in the numbers. If you had asked me, What is going to cause the people to finally wake up and realize COVID is real? What's going to be the triggering event? I would not have guessed celebrity gets COVID. And maybe I should have, but I definitely didn't. So it's, you know, and why should that be convincing when something else wasn't convincing, right? I don't— even now in hindsight, I don't really know why Tom Hanks did it specifically and other things didn't before. And probably it's one of these things where, like, it was in the air the same way toxin was in the air, right? The NBA canceled its season, like, the same day. Maybe if Hanks hadn't gotten sick, that it would have been, like, the NBA that would have done it, or, like, some other third thing would have done it, like, the next day. I don't know.

Can we do a little bit more unpacking of what you think a grand bargain between the US and China could look like? I understand it's not your base case of an outcome, and it's not mine either, as much as I want to hyperstition it. But what do you think is the scope or the sort of most expansive parameters in which we could see a grand bargain take shape?

So all that we're really asking the Chinese to do is to not attempt— is basically 2 categories of things here, right? Category one, Well, maybe 3. Category 1 is we're asking them not to do things like steal and publish the weights or otherwise like do hostile things against our AI companies. Thing number 2 is we're asking them not to try and race ahead sufficiently that they could potentially match or surpass where the frontier closed AIs are. And so these are fairly simple. These are not actually expensive asks because because I do not believe the Chinese really have any intention of trying very hard to do either of these things under any normal circumstances. I don't think that, you know, like obviously if, you know, DeepSeek or, you know, Alibaba suddenly found some huge architectural improvement and suddenly was able to train something that was better than Astra, I'm sure they would. And I'm sure they would put it up on the API and I'm sure they would try to sell it and try to like make this, Amazing. But like, realistically speaking, with the amount of compute they have access to and the amount of money they've been willing to invest in these things, just far smaller than the amount we've been willing to invest in frontier AI development, it's not that likely to happen, especially without, you know, they won't have the American models to distill and to plane off of and the American algorithmic improvements and like just basically just back off from that. And we're also basically just asking them, Don't put it— don't allow actually actively dangerous cyber capabilities that we can't— that the world can't handle to be made publicly available. Don't let your people get over their skis and release the weights to models where we need to advance our models to defend against the weights that you released, right? Because like the argument The only good argument left, really, other than commercial and we can, for proceeding quickly with American AI— we are rapidly increasing the capabilities of our AI models to deal with the threat from rapidly increasing capabilities of AI models. That is the actual threat model, which is because the random guy in the street will have access to these dangerous cyber capabilities. We need to be ahead of that by 6 months, by a year, by whatever it is, so that the key defenders of critical infrastructure, major software, you know, et cetera, and also the man on the street in like ordinary, you know, running a hospital can potentially not just get run over by random people with hack, with, you know, a terminal and ransomware. And, you know, also the broad national, national state version of that. So, we need you to make sure that, you know, if it gets to that point, these things are kept on the API, right? And like beyond that, you know, and, and that you, you know, potentially don't try to smuggle a ton of chips and like otherwise evade the system. Like, because like we're worried about this. What we're worried about is this, like, you know, I haven't, I haven't thought carefully about exactly what we need to ask ask what we put into the rule because sort of like, I figure that's not my department. But like, in a real sense, like when I think about what I would want as an— on the negotiating table, all that you really need is to take away the boogeyman of lose to China and the boogeyman of open weight models destroy the internet, right? Like all you need to do is make sure these things won't happen. So you need an in extremis promise basically from the Chinese. But realistically speaking, We don't actually need them to start like showing up at DeepSeek and making sure they stop training new models. That is not necessary.

So what, what should we be willing to give to them?

So what we are giving to them in this scenario is that we are in fact dramatically slowing down in exactly the one technology Where we have a giant commercial and strategic advantage that if we pressed it, would potentially overwhelm every other commercial and strategic advantage.

How how how would they measure that?

Well, we're going to have obviously you know we start with the Daria's proposal for embedding evaluators into the labs. Embedding Chinese evaluators in the labs potentially if we had trusted people that we could. Potentially they wouldn't be allowed to leave, right? Like once they entered, right? We would have to sequester them for some period of time. Or like, I'm not a master of intelligence and verification, but there are systems whereby you can have someone able to gather information and output the bit of whether or not things are okay, but not the algorithms that were used to figure out whether it was okay and not like all the detailed info they saw while doing it. There are ways to do this. And I am confident that like, If we put this way, if this was the most important thing on earth, if we put our minds to this and only this as the thing that would determine the fate of the world, yes, we could figure out how to let the Chinese verify in a way they were confident in that we were holding up our end of the bargain without them stealing all of our secrets. I do not see any reason why this is not possible.

So we would be okay with them entering the labs, embedding, learning the workflows, learning how things are done, and transmitting information back, going back, rotating through.

Look, my— look, this is an intel— this is a standard intel verification agreement style problem. I am not here to claim that I'm an expert or that I know what the answer is. My instinctive answer is the Chinese send people that we vet in to be the verifiers. Those people are watched physically.

But what are they verifying? What are they verifying? What kind of assurance are they giving to the Chinese?

They are verifying that whatever So, like, again, like, we have to decide exactly how much pacing, you know, we are doing, what assurances we are requiring of the artificial intelligences that we are training before we're willing to proceed, what pace of inputs we are restricting ourselves to, whatever other requirements that we determine. And there's something that would be negotiated between OpenAI and Anthropic and the US government and the Chinese government.

But, you know, China is aware that it's behind. Number one.

Yes.

Number two, they've always been behind on every other technology. This is actually an unusual situation for them to be actually at parity at this point. They have always typically the US has invented things and then China has commercialized and industrialized those things. That has been their kind of like 20-year, 30-year kind of strategy. And they've always been willing to, willing to let the US go ahead and like, you know, do all of the research and put in all the money and all of the a hardship, and then they just adopt, right? So they are not in any different position from how they've been in the last 20 to 30 years. It's not terribly different. And what if they say, well, you guys can go ahead. We're happy for you. That's great. If you want to let us have people coming in and going out, that's great. But this is not our concern. We're not that terribly concerned about AI safety. So Okay, what else can you give us? Right? Because that is actually the situation right now.

Well, look, if it's Chinese— again, you really never know until you go into that room what the guy actually wants.

I agree with you there, but I think for all signs, it's fairly easy to say right now that the Chinese are less concerned with AI safety. And so offering them an AI safety, like, you know, pacing the frontier is not something which is something that they're willing to purchase at an expensive price. So therefore, they are looking, they would probably need something else. I mean, it's fairly easy to say that, right?

Well, to be clear, if we are, look, if the Chinese, you obviously asked the question, would the Chinese like to accelerate American AI development or slow down American AI development? If the answer is they'd like to accelerate it because then they can copy it and it helps their AI development, then there's no need to make it. There's both neither a deal to be made nor a need for a deal. Because in that situation, the Chinese want us to proceed. So there's nothing to verify, right? Like we can just do whatever we need to do.

Mm-hmm.

But also the Chinese are looking to, you know, Fast follow what we do and are not willing to invest the kinds of hundreds of billions or trillions necessary to catch us. Yes, even if we slow down, so we don't have to worry about the Chinese passing us. We can just do whatever we need to do without an agreement. Yeah, and all that we need from the Chinese is an agreement not to destroy the internet. But the Chinese have no interest in destroying the internet. Yes, because the internet Chinese like the internet. So we can just both act in our own self-interest and everything is fine. Yeah. Okay. The, the problem is not China. The problem is lose to China. The problem is the perceived threat from China pushing you forward. Right?

Right.

If you are correct, and I think you probably are, if the Chinese actually don't have any interest In pushing to superintelligence first, in trying to like build the fast, build the bigger, smarter special model. They just want to, they want to distill it.

Improve the lives of the people. You know, right.

They want to make it faster. They want to make it cheaper. They want to make it diffuse. They want to improve the lives of their people. Yeah. I want them to improve the lives of their people. That's great. You do your thing.

Yeah.

We do our thing. Everybody wins. There's no need to make a deal. And I think that is in fact the best case.

So, so, so what if that is the state that we're in, but we're still asking for the deal, right? Like, like we're entering this place where the Americans want a deal, so, but the Chinese are like, all right, what do you, what are you willing to give us, right? Like, what do you want? What are you willing to give us? And if you give them like AI safety, like monitoring, that's not uh, thing that they are very interested in purchasing for a high price. So what else are you willing to give them, right? So that is the question, right?

So what I— clearly what I am saying here, right, is if you walk in that room and that is the attitude, that is Xi's attitude, that is what she cares about, and she communicates that, and Trump says, that's great, life is good, I'm so happy you feel that way, I am going to do my thing. You are going to do your thing. We are gonna make a level 1 or 2 agreement to just like do crazy shit that we weren't gonna do anyway, but so we can announce it and shake hands and call it a deal and both score points.

Yeah.

And we're gonna spend the rest of this talking about trade and Taiwan and all the other issues that we have. And like, I'm just gonna sell, I'm gonna give you intel, right? What we're gonna do is we're gonna unilaterally, the only thing we're gonna have to do now is we're just gonna unilaterally give you information about the cyber situation.

Mm-hmm.

Right. And the bio situation and so on.

Right.

And then you can use that to make an intelligent, self-interested decision.

Yeah.

As to what you're gonna do to stop your labs from fucking the internet.

Yeah.

And we can all win. Yeah. And like, again, like in that, like the threat from China, has always been that because you worry about China, you don't have the game theoretic ability to make a deal between the labs. You don't have an ability to be responsible because there is this other actor who will defect. The basic argument was you need to go on a stag hunt, right? Everybody has to agree.

Mm-hmm.

And there are these 5, these, you know, 2 American labs that are in the, that are 2 or 3 American labs, they're in the lead. But if they pause for more than 6 months, if they, if they slow down too much, There's this xAI and there's Meta. And then if you pause for much longer than that, then there's these Chinese labs. If the Chinese labs are not really a threat in this sense, if they are— like, if the Chinese are creating a different product, a fundamentally different product, then there's no problem, right? Like, we don't need the Chinese agreement any more than we need agreement from India or Germany, right? We just— we need them to be good. Actors on this world stage. We need to make ourselves good actors on the world stage because sometimes we don't do so well, and we need to. And then we try to like just generally like focus on being friends and not having something stupid like a war over Taiwan blow it all up. Like that is you know the easy case, right? So like I mean obviously like there's a case where the Chinese simultaneously don't want to make a deal. But also really do want to push forward and try to beat us to superintelligence. But if they wanted to beat us to superintelligence, that's the same reason they want to do that should make them want to, to make them want the deal. So like, amazing answers, you know, like I, I'm reading a lot of this in real time, right? Because I've been, I've been focusing on other questions. And so a lot of this is actually me brainstorming and saying these things for the first time.

Yeah.

But like, Yeah, it does seem obvious to me now that I say it out loud that if the Chinese don't care about these, if the Chinese don't care about the Americans shifting a lot of investment into AI safety, which would inherently slow down their progress, if they don't think that's a big get, then we don't need a deal at all. So either they value it and we make a deal, or they don't value it and we don't need a deal. And either way, we win. Right.

Let me extend that thought a little bit. So Dario was on the face of the nation. He was asked, would you be willing to give up the technology to the government? And Dario said, I quote, to the right combination of governments.

Yes.

And this has probably sunk him with the administration. This is going to cause him a lot of problems for the next like decade. This is, this is going to suck for him. All right. So this, this to the right com— combination of governments indicates that he's not going to be willing to give it up to the U.S. government, which means that the company is more in control of it than the government itself. And remember, the government is democratically elected. The government is representatives of the people. I, I know, I know people dislike the government, but that is a— in a democracy, the, the, the government are the representatives of the people, fairly chosen. And he's not willing to give up control to the government. He's saying to the right combination of governments, meaning he does not trust enough for one government, which means that this expands the question beyond the Chinese actually, because this is not anymore just China getting ahead. This is any government taking control of the technology, which means that he's going to be opposed to any government, any one government. Right. So how does this start to fit into that framework?

I don't want to hyperstition conflicts, especially like, you know, decade-long conflicts. Uh, how many times has Anthropic been completely sunk with this administration separately and come back from it? Right. How many statements have been beyond the pale and like poisoned, permanently poisoned this relationship? And how many other statements, if taken out of context and blown up and sent to the administration, could have caused similar incidents but randomly didn't? Randomly, maybe, or maybe like because they weren't actually that bad. Like, you know, this is a standard kind of gotcha question, right? If you're so concerned about AI, would you voluntarily just hand over your entire company to the government upon request? And the real answer is, this is not up to you, sir. Right? Like if I were Dario, I'd be like, well, that decision is not up to Anthropic. If the US government decides to take control of Anthropic, that is something the government has power to do. And I am not about to defy the United States government. And it's also not up to me to decide when to tell the United States government to come take control of my company.

Right?

Like that is the answer I would've given probably.

Yeah.

Yeah.

But, you know, I think that like, you know, there is a big hullabaloo in— he's trying to address many different audiences at once, right, with that statement, because there's this huge thing about concentration of power. There was this huge thing amongst many, many people for good and bad reasons that they are worried about a single authority, a single group of any kind, including the US government, having authority over AI. And Sam Altman talks about this. He talked about this last night. He said, we must beware concentration of power. We can't let any one group have control over AI. That is essentially the same statement as Dario. If anything, it is like not even a group of governments, because that is one consortium, and therefore it is essentially one thing, right? We have to simultaneously walk the line when we're trying to People are trying to satisfy different alarm bells and sacred values that contradict each other, and they don't really make sense in the new world because of the just sheer amount of hell that would happen if you were caught saying something that violated them. And so, like, you have to say, I want democratic control. You have to say, I don't want concentration of power. You have to say, like, several other things like this. But realistically speaking, democratic control is concentration of power. In some important sense, the exact same reason you talked about it, right? If the US government takes control, that is concentration of power, but it's the ultimate version of democratic control. It's the United States democracy taking control. So you can't avow both of these things at once for too long, or you start— basically, if you get asked too many specific questions about hypotheticals in which things are going crazy and in which the ordinary rules of the world are breaking down and things are transforming, you are going to get backed into a corner where there's a statement that, like, if taken literally by someone looking to make an enemy, will make them an enemy. And there is nothing you can do about it. You can, like, try to postpone it with fancy dodges, like I was suggesting you might do with the administration there.

Mm-hmm.

But, like, inherently, if you are speaking frankly, right? Like, and, you know, everybody who talks about AI seriously, takes superintelligence seriously, Has had to do some form of dodging these questions because any concrete answer, in addition to like having lots and lots of parts of it that like, if I try to guess what the answer is, you're gonna be able to say that sounds wrong, or that sounds, you know, like it has this hole or whatever. But like, we don't have a good answer for some of this. And so like, we need to satisfy concentration of power problems, democratic control problems. And also control at all problems, right? Like, and these 3 seem to be in very, very strong conflict, as in, like, if we want to be in control of this technology at all, we cannot in fact fully democratize it in some important senses and give everybody access to it on equal footing. Because that doesn't really work for very simple logistical reasons. If everybody has superintelligence, well then The superintelligence have everybody is what actually just happened. Because how are you gonna compete against the people who entrusted their superintelligence with all of their work and try to, and try to stay in charge yourself? This works on this individual level, on the corporate level, on the national level, on this, on every level. And so, you know, we have all these problems we don't know how to solve. And this is like one of the reasons why we pace, why we feel the need to pace. Is because we don't have any answers to these questions, right? Like, I'm not saying, and this is what we must choose, right? This is the answer that is the least bad. This is the way we navigate this. I am, I don't even dare in this conversation, like, to offer up a suggestion. I simply point out that, like, the ex— the standard solution as proposed by people like OpenAI, right, that everybody will have their own equal superintelligence. Does not work. Similarly, like if you care about concentration of power, if you care about the potential for tyranny, handing over the only AIs that matter to one government that you know has an imperial presidency has an obvious problem. Even if you think Donald Trump is a very, very trustworthy and noble steward, what happens in 2029? What happens if the noble steward who allowed a free and fair election loses to AOC, right? And then AOC has this most powerful weapon. Well, nobody— very few people are going to think that both the Trump part of that and the AOC part of that are going to go well. So we have a huge problem, don't we?

Just to share one, uh, visual of the sort of thing that Might cause somebody in Dario's position a little concern. This is from the Department of War CTO within the last hour while we've been live. Americanism, not effective altruism. The US will be AI dominant, says the apparently newly AI-first War Department. So, I don't know, read it and weep, I guess.

So, so I, I guess I, I'm gonna just laugh a bit Because like, you know, we had this statement from earlier in the podcast, right? If AIs were going rogue, people would notice and people would decide to do something about it. But the US Department of War has gone rogue, right? Like in public, openly gone rogue. Even Michael has gone rogue. Pete Exeter, to a large extent, has gone rogue. They continue to go rogue. and make these completely outlandish, crazy statements, including in many cases running past Trump's literal publicly posted stop signs, and nobody reins them in. Nobody does anything about it. It's the U.S. Department of War. And we still have Anthropic, the same Anthropic that is saying it wants to slow down. Now they are saying no effective altruism, and they have Anthropic that needed supply chain risk. As the Houthis try to use Claude to develop missile technology to aim at the Americans, the Americans are saying our people are not allowed to use Claude. This is how crazy the situation has already gotten, and nobody seems inclined to do anything about it. It's just there. That's just our new reality.

So, The pace of events continues to be super fast. I think we should let you get back to your highly rigorous schedule of keeping up with events before too long. But I have just maybe a couple closing things I would want to get your take on. One is, how are we going to know if this is working? You touched on this at the end of your post, like, How will we know if the frontier is actually being paced, to what extent it's being paced or not? Obviously, this is the kind of thing you could kind of fake with some PR and just keep doing what you're doing. Oh, sorry, some sound came out of the background for me. Okay, so 2 is just kind of anything else that you think we should be paying more attention to that hasn't surfaced in this conversation, or either you think is kind of at risk of going underappreciated even by the close watchers? And then third is, I'm really struck by the fact that this Pacing the Frontier letter, organised by a small number of committed individuals that we both know, has seemingly had a huge effect. Now, this could be sort of a Tom Hanks situation where the thing was sort of in the air, and if not for this, it could have been something else very similar. But it is striking that that language has kind of risen now to this level where the key decision makers are saying, are putting a we must in front of it. So I'm really curious as to any speculations you have about similar possible pressure points that people could be working on. One that somebody tagged me online about over the weekend was putting together a campaign saying we will nominate Trump for the Nobel Peace Prize.

That was a good one.

If he can get a deal with China out of the upcoming thing. And I have to say, that doesn't strike me as crazy. You know, so I mean, he would deserve it.

Right.

I agree. So just to recap that. One, how will we know if the frontier is indeed being paced? 2, anything you think we're missing? 3, what other little kind of quirky projects could a small group do to maybe move the needle?

Yeah. So how will we know? I think that the embedded evaluators are a good first step, certainly. To some extent, you have, you always have to judge by the absolute observation, right? Like, you don't know what the counterfactual rate of progress would be, but you do have to say, okay, if it goes faster than this, you know, on various graphs, in various, like, holistic assessments and so on, you didn't do the thing, like, no matter what. And if it goes at most this fast, you did do the thing, even if unintentionally. And you passed. And then there's a range in which, like, you're in between and you're not sure, and you do a combination of different assessments because you don't really know what would have happened before. But, like, you know, you can— I think you can, like, sort of holistically observe what a variety of people are saying, what a variety of actions are, a variety of the properties of the models, a variety of the different actions that the labs choose to take. And you can Especially with investigators, I think you can have a reasonably good idea of what's going on. And especially if you see the labs consistently letting their employees talk and you consistently see those employees are not being retaliated against in an important sense, like as we are currently seeing. But the truth is that you can never know for sure, obviously, without much more intrusive measures. that would have to be imposed by the government. In terms of things that, you know, I think even in hindsight, we don't really know which particular actions, which open letters, which promises, which moments, which manifestos, which, you know, actions were a big deal and which ones, you know, were marginal contributions to what would've happened anyway. And which ones like ultimately backfired sometimes. We don't really know. I think we can argue endlessly about whether or not certain decisions like founding Anthropic were negative or bad for AI safety when all the chips are down, even now. We have this narrative that the KI statement that AI is an existential risk to humanity and should be treated as such was very helpful, and the 6-month pause letter request was very anti-helpful. But like, do we really know? I don't know. And like, how do you know which one you're doing when you issue a new letter? It's hard to say. I think the Pacing the Frontier letter was in fact very helpful, partly because it sort of gave people a new way of presenting, a way to pivot on language, a way to pivot on rhetoric that turned out to be of the moment and be something that could be grabbed. Like, as I said before, when things are happening in a hurry, often you grab for whatever is in the air, whatever is, is shuffle-ready, whatever is, is already being, uh, just around. And so by making things be around, right, you can take shots, and then occasionally your shot can be good. Um, but also, again, like, you know, every now and then, like, someone says the wrong thing and it gets blown up out of proportion. Or I could get like poo-pooed and mocked and then like things get worse. I do think that like you have to stop worrying about certain types of people that I call the usual suspects being mad at you or a proposal enraging them because like they are rocks, right? In the sense that they have a, like in the Scott Alexander sense that like they are a rock within same thing written on them. And whenever you ask them what they think, they will just read off the rock. Right? Like, and when you encounter that, when you encounter a defect bot or cooperate bot, someone whose actions don't change based on what you do, you can't care what they do because they have taken themselves out of the equation. And, you know, you have to focus on, you know, what other people will say, other people, how Other people will react. And like, in terms of like what happens next, I think that like, you know, I was asked a lot of specific detailed questions in this segment, right? Where I was asked like, okay, exactly what would we give China? Exactly how would we implement this procedure? Exactly what would the limits of this thing be? And so I think one of the main things that you can do right now is to actually fill in those gaps concretely, to start thinking about like exactly how we implement these things. What should be the rules for the embedded investigators? What would be the rules for the threshold that requires certain things to happen? What things would we need to happen right now? Like, we, we don't know what those things would be. Also, you could prepare to be the investigators or to bring the investigators into being, right? The evaluators don't currently exist. We have Meter, we have Redwood, we have a handful of Apollo and some So on. We have a handful of these people, but they're all like kind of similar people. They're all like vulnerable to the accusations that they are a little bit too of the same cultural values, the same ilk, the same like ways of thinking as the labs themselves. So I don't think they can be a complete package on their own. They need to be complemented and there just aren't enough of them. They need to be complemented by additional approaches. And I say like in today's post, But what you want is you want some people who are of METER or something similar to METER, who are deeply embedded in this culture, who've worked at the labs, who understand the vernaculars and the theories of the risk, and who can look in detail. And you also want people who can take kind of an outside view, who can like do the kind of thing where like you worked in car safety or something and you come in, you go like, what, are you crazy? What are you doing over here? Right? The same way that like— When you suddenly have to take your, like, you know, you're having this contract dispute and then you go before a courtroom and the judge looks at you. And now all that matters is what you can make legible to the judge. And like, in some ways you lose a lot of nuance, but in other ways you get this kind of common sense outside view that can bring a lot of clarity. And so you need both.

Yeah, I think that's apt. I'm ready to maybe leave it there. And again, I know you've got a relentless pace these days. The one other thing I'll just invite you to comment on one more time, if you have anything in mind, is anything you think we're missing or that hasn't received enough attention amongst the flurry of headlines in recent days?

I mean, there's a lot of stuff that everybody misses, but I would say Just, if I had to pick one thing people are missing.

I think for me it might just be, honestly, the models have gotten a lot better again in the very short period of time. It hasn't quite sunk in. I don't know.

I don't know if people are missing that, but I think it just be the sheer amount to which the people in the labs genuinely see dramatic improvement in the models and are freaking out about it is the real story, right? Like behind all of this is why everything is happening now and didn't happen before. And that like we are really talking about recursive self-improvement, not like a fast takeoff yet, but like if we continued straight on from here, that like within a year we might actually see a Christiano or Yudkowsky style hard takeoff.

Mm-hmm.

Where like everything will change and It would be considered patently silly that we were talking about, you know, oh, people are on a watch when you have a million dollars. Like, like, it'd be just so, so, so blatantly, like, irrelevant to the level of what's happening that, like, you will look back and go, like, why are we wasting our time?

Well, here's to good analysis and making the most of the time we have to steer this ship in the right direction.

Yeah.

Zvi Mashowitz. Thank you for being with us on AI in the AM. Thank you for your service generally. Keep up the good work.

All right, bye-bye.

To note, Zvi's take on things, I don't—

I'm not like saying it's hard, takeoff is not possible. I'm saying that you have to prove it to the public and you have to look at the evidence that the public has and the frame of reference that the general population and the public and the administration and bureaucrats have. And from that frame of reference, this does not look possible or feasible. For example, let me show you something from Ben Thompson here. So Ben Thompson writes a very popular blog, TechCrunch. Tech blog. So Ben Thompson is here and he's saying, uh, I'm certainly— I certainly am not minimizing the digital world. It's where I make a living, for one. But it's sometimes hard to shake the sense that people in tech are quite literally too online. Losing the internet might feel like extinction. I certainly can relate. But the important point is that it is digital. Which means enabling AI to touch grass depends on physical infrastructure operated, manufactured, and controlled by humans. And as long as that is the case, I have a hard time accepting the imposition of an authoritarian regime based on a premise that isn't yet proven. And, you know, he— this is pretty eloquent in the sense that Dario's scenario was really worldwide internet shutdown, which is what I wanted to point out at the beginning of that. And worldwide internet shutdown to a lot of people is not a big deal. We've already experienced shutdown once, by the way. We experienced closing of the borders during COVID We had a pretty significant closing of the borders and we improved a lot of processes around that. A worldwide internet shutdown, I do not think would be at the catastrophe level that— I mean, yes, it is hundreds of billions of dollars. Don't get me wrong. It's going to be hundreds of billions of dollars. But I don't think hundreds of billions of dollars is a big deal. COVID probably saw losses of several trillion dollars in the economy. And there is resilience in the economy. And COVID took a long time, like 2+ years of various interruptions, et cetera. And we did get through, right? We did get through. I think the economy is a lot more resilient. And the economy meaning general wellbeing of the public as well. I put that in that category as well. It is a lot more resilient than I think people are willing to give it credit for. And so So I think Dario did a couple of things that I think might have made his case worse, which is he identified for the first time, I think, a concrete thing that could happen, which we have a path towards, which we've already seen happening. And I think the problem with identifying that concretely is that people can now jump in and say like, well, that's not such a big deal, like Ben Thompson is. So I think that that's one thing. The second thing is that to a combination of governments was the worst possible thing that he could have said. He has scuttled himself. He has possibly poisoned the chalice for all of the other firms as well, by the way. So this is now, I think, going to be a big, big deal. And I think they are going to find themselves— they're going to start finding themselves in opposition to both administrations. I think US bureaucracy and the government is actually very, very strong. And what the bureaucracy does particularly well is that it doesn't react particularly well in the short term. In the short term, it's slow. But in the long term, people write laws and people lobby their law schools and et cetera, et cetera. And they have law reviews and you write and you write and you write. The, the U.S. system is very textual, like, you know, from, from the Alexander Hamilton days. They wrote a lot of stuff from the Federalist Papers. They write a lot of stuff, and the writing gets digested through this kind of shared infrastructure of minds. And then that comes out to, like, at some point it's morally indefensible to have slavery, it's morally indefensible to have segregation. And it takes, it takes a long time, it takes some time for that to process. But over the course of like 20, 30 years, that gets processed down through all of the law students, et cetera. And it gets implemented. At some point it becomes impossible for certain things to continue. And then they change. So change does happen, albeit slowly. And I think the thing that I think Dario specifically, this idea that we have to move quickly, that's a red flag, right? Like number one, it's a red flag. Number two, I think people in AI don't have a sense of how often this is asked of any administration. From the early days, John Adams was asked to suspend civil rights, suspend freedom of speech. Thomas Jefferson had to go up against him and they fought over it. And he was asked that— and the first war, committing the country to war with France, which had been a partner. From the early days, he'd been asked, right? This kind of, it's an emergency, let's suspend civil rights, let's do these things which are against the Constitution because it's necessary. And so there's always been this pushback, I think, against that happening throughout the 250 years of the country. So I think that for me starts like the immediate emergency. We have to act, we have to suspend certain rights, we have to like do certain things which are not like out extra legal. Like, this is not like— this shouldn't fall to like the democratically elected government of our country. Like, wow, dude, just like red flags all over the place, right? And I think he, he's bought himself that, and, um, that is going to cause him a great deal of trouble, uh, going forward. So, uh, and, and this is where I think The idea that they have to get out of the Berkeley EA circle, including this, like, oh, we're going to ask METER to— you can't do that. I understand that you think they're the only technically competent people. I understand that. But you cannot do this. Like, because the point is not to technically prove something. The point is for the public to actually have faith that this is being done correctly. And so you have to address the public's fears. When people like Zvi and others are like, David Sacks doesn't matter and this guy doesn't matter and that guy— 10 million people looking at David Sacks' post written by Grok, it matters. You have to persuade— the whole democratic process is persuading the rest of the people. And so I think this is not actually Basically, I don't think the labs are on the right track. They kind of lost the country and they're going to have to come back to it. And I think all of the people who are just ignoring, like, oh, the biological experts don't matter. You guys didn't predict AI, so your thoughts have no value. Your opinions have no value. What you're saying is that we have a cabal of people who predicted this 20 years ago, And all of the decisions should be made by this cabal, even if we're unable and unwilling to convince the rest of you. Wow, dude. Wow. Like, I think, and I think there's this like recursive loop inside the companies where they're unable to see this external perspective. Because Dean Ball is also— Dean Ball actually said this, like, you didn't predict it. So, you know, if you didn't predict it, then you should trust the experts, right? Trust the experts, trust the science. Believe in the science, trust the experts. I think Robin Hanson had another one. Robin Hanson was like, look, you have the experts and you have the elites. And a lot of times experts become elites because you have expertise. But the elites are the ones who have control of society because they have the standing, they have the stature, they've done the work, they've gone through all of these little credentialisms that society expects of them. And they're kind of the people in control of society. And in order to get anything done, the experts have to convince the elites. And right now, this thing is out of the hands of the experts. It's in the hands of the elites like David Sacks, et cetera. So, you know.

Well, yeah, I think it's safe to say that the politics and the wrangling are far from finished. I do think Zvi's point in terms of what we should avoid was a very apt one. Polarisation on this topic, and partisan polarisation in particular, would be really good to avoid if we can. It's been a pleasant surprise for me that AI hasn't become totally polarised so far, and I think we should do kind of anything that we can do to continue to avoid that because that will prove paralysing and deranging in all kinds of different ways. So anything we can do to remember that we're one humanity, one American people, whatever, is great, I think, to avoid that. And yeah, I mean, I agree with you that the companies have lost the public. That seems Fair sentiment is not very positive. The thing that I would maybe say, I'm not so sure what he's talking about, especially on that Ben Thompson quote, is like, what authoritarian program are we talking about exactly? I don't think that there's really an authoritarian program on the table right now. There's been a lot of people sort of, I would say, essentially fearmongering around the edges that they're trying to shut down open source. or they're trying to do this or that. But I don't see any actual— oh, look, we got our first spam in the live chat. Amazing.

Uh-huh.

I don't see anybody actually pushing for, like, an end to open source or a really draconian or authoritarian regime in the United States right now. That seems to be— like, I would think that's a bad thing if we were to suddenly be confronted with it. But right now what we're talking about is a small number of companies potentially needing an antitrust, if not waiver, at least a little reassurance that we're not going to come down on you if you work together on some safety collaborations. So I think there's pretty modest ideas on the table right now that are actually at the centre of the consideration set.

At least.

I don't think they're modest at all, because I think Rune came out and said, I expect next year open weights models will be banned, right? I think it's very clear that the labs are alluding to very strong restrictions on what can be inferenced. It's very clear, like you're talking about biorisk, you're talking about cybersecurity risk, you're talking about the risk of inference from models, and the risk of inference from models is going to be dangerous, right? So you're talking about strong controls on what can be inferred. That's fair to say, right? And then once you start operationalizing it, you're like, okay, do we surveil the GPUs? Do we decide which models you can run? Do we allow you to fine-tune? Do we allow you to distill? If you take an open weights model which has guardrails and then you distill to remove the guardrails, is that— I mean, these are the questions. These are the technical— I mean, it's easy for me to see that this is where this will go, right? So you are talking about preventing inference of things that you don't want to be inferred. And it's fairly clear that that would have to happen, right? And I don't think anyone is saying that that doesn't have to happen. That is what the firms say. And they are in that position where they're saying this shouldn't be allowed to be inferred. Um, and you know, the thing is, even some of the bio people are like, you know, why are you stopping like bio stuff, bio questions on, on ChatGPT, on Claude? They don't think it's right. Even now, even now, before, before all of this, before we operationalize everything else, they think it's wrong, right? Um, so I think, I think that's the, that's the concern. The concern is that we kind of know, we kind of know where it's going to go. Uh, we kind of know that it's, it's about controlling certain things that can be inferred.

Well, my hope, I guess, final thought probably for today, is that if we do in fact do some pacing or even a pause for a few months on continued scaling at the frontier, that we can use that time to solve some of the problems or advance some of the solutions that seem very promising to some of these core problems. such that we can have our cake and eat it too. I do think there is an open source model, hypothetically, that one could create that I do think the state will have a very difficult time not taking some sort of action on. But the question in my mind is, like, right now, the question in my mind is, can we come up with a way to create powerful and empowering open source models that can be distributed without including in them all the dangerous capabilities that we really don't want to see broadly distributed. And I think if we can, that leads us to a pretty happy compromise place. And I don't even think it is necessarily going to take that long for us to get there. So This is where it's like, not only do I think some pacing is just inherently wise, but it's also an opportunity for everybody throughout the ecosystem. Make the most of time, gather ye safety solutions while ye may. And let's see if we can't get to the point where we can have our cake and eat it too in the form of genuinely distributed, decentralized, non-concentrated power structures that do give individuals the ability to do what they want to do with just a few compromises around the edges that I think the vast majority of people would agree are sane and supportable and not an undue burden on people's ability to deploy AI in their daily lives. I really do think we can have that. We just don't have those solutions developed well enough yet to land there by default in the next few months. And unfortunately, if we don't extend that runway, we might end up in a pretty uncomfortable and literally very dangerous situation in the next few months. So hopefully we can use this, whatever time we are buying for ourselves right now, to solve those problems. I think that's of the utmost importance in the the immediate term.

And I think on that note, stay tuned. We will be back tomorrow. Nathan, good morning.

Good morning, Prakash. It's been fun. Yeah, tomorrow's going to be a good one, by the way. We've got Andan Labs coming back to talk about some of their latest weirdness that they've seen. And we've also got a married, famously pronatalist couple who has started some sort of an AI religion that they're gonna tell us about. And that, I'm sure, will be fascinating. So in addition to all this frontier wrangling and politics and, you know, power questions, there's also the question of culture and what's gonna happen as AIs run businesses and what's gonna happen as AIs potentially inspire, if not lead religions. And You know, there's a lot for us to keep track of, so tomorrow will definitely be another fun day.

Indeed. Bye-bye.

Bye for now.