Anthropic's Dario Amodei refuses to back open AI models, arguing the real threat is closed, powerful AI reaching authoritarian regimes like China's military. He proposes chip export controls and mandatory safety testing instead of debating open versus closed, citing risks from cyber-biological attacks and attacker-defender asymmetry in biology.
Anthropic refused to sign the $18 trillion market cap open weights letter, and Dario just explained why, and he's positioning that it's not protectionism. Here's what happened and what matters for anybody who is looking on the outside around the open versus closed and US versus China fight and debate. Some source facts: on July 27th, CEO of Anthropic published Anthropic's position. Anthropic has never advocated for a ban on open weights Models. Open models without dangerous capabilities are a public good. They don't cost anything besides the compute needed to run them. So why didn't they sign the coalition letter signed by $18 trillion worth of market cap of tech companies who believe they should, including SpaceX and Meta and Nvidia, et cetera? They agree that open weights expand access and competition, but disagree that open weights necessarily make it easier to develop safeguards, citing attacker-defender asymmetry in biology. As an example, Dario talks about 2 nightmare scenarios. One, authoritarian governments building more powerful AI than the US, regardless of whether these models are open or closed. They see that as the primary threat. Number 2, misuse for cyber and bio attacks, where open models do pose higher risk than closed models because you can't really stop it once the weights and stuff are out there. You can't like retrieve that and say, hey, cut off bad guys. And that, that's part of their argument there. So what Anthropic positions in this paper, at least from my read of it, is instead of a ban, look at chip export controls, which are the most efficient and direct way to block that threat, number one. And then cracking down on distillation as a technique which can bring the Chinese frontier to within a few months of the US frontier, and mandatory safety testing for all sufficiently capable models, open and closed. The surprise is, given that they're a closed safety champion, they explicitly reject the accusation that they want bans to protect The most dangerous scenario is a secret model handed only to the PLA, i.e., a closed model in the wrong hands, not an open one. This is a very sharp take and position. Dario basically explains why they didn't sign. The coalition says open is good, don't regulate. Anthropic is basically saying the open versus closed access or debate is kind of a distraction. We really need to win the war. The China versus US thing is one for freedom. And so the real levers are chips and distillation. So anyway, uh, that distillation point comes right on the heels of KIMI sparking all of these tech companies and operators with real token budgets being like, well, why don't I move my workloads over to KIMI? Jonathan, from your CTO perspective, thoughts on Dario's position on open weight models coming out of Anthropic here?
You know, I'm just reading the letter right now and I think it's well articulated. I think it talks about the right points. The risks he talks about are not negligible, right? And I think that there's kind of like, um, maybe different perspectives about whether or not having open models is better for security or worse for security. But then there's also like the bio angle and the bio risk. And I think he's pointing to like, you know, in warfare, for instance, like there is a combined arms dynamic that happens at the, on the battle lines where like one side will figure something out and then the other side has to adapt. And like during World War I, this is like really pronounced where, you know, they, they brought these machine guns on the battlefield, hundreds of thousands of soldiers are dying and like both sides figured out, okay, let's not do that anymore. And then they come up with tanks, they come up with strategies that minimize casualties, and you see warfare evolve very quickly. But then there's like nuclear weapons, right? Where one side can potentially just end the whole game with asymmetric capabilities. The main deterrence on nuclear weapons has been mutually assured destruction. Like, that's been the thing that holds the balance. And I'm not sure what the balance is when it comes to like using LLMs or other technology to create cyber or biological weapons. That might be something that gets into the domain, this is something we can't contain. There is the ability for capture and asymmetric positions to hold. So again, I think like on cybersecurity, you want this back and forth of like, you know, escalating capabilities on all sides. I think open tools are like a good way to create that dynamic. But I don't know that anybody has a real serious answer for these things where there could be something like a biological weapon that, you know, is not stoppable. That might be fictitious or it might be real. And I think we don't know. And I think that's the place where, you know, our security agencies need to do a lot of research and be like really ahead of the curve on this.
