In this discussion, Robert Ta explains why Anthropic declined to sign the $18 trillion‑market‑cap Open Weights coalition letter, summarizing Dario Amodei’s public position. Anthropic does not oppose open models per se, viewing non‑dangerous open weights as a public good, but argues that openness can exacerbate attacker‑defender asymmetries and increase the risk of cyber‑ and bio‑weapon misuse. Two “nightmare scenarios” are highlighted: (1) authoritarian states developing AI superiority regardless of openness, and (2) malicious actors leveraging openly released weights to create unstoppable cyber or biological attacks. Instead of a ban, Anthropic proposes concrete policy levers—chip export controls to curb the first threat, restrictions on model distillation to slow frontier convergence, and mandatory safety testing for all sufficiently capable models, whether open or closed. Speaker B expands on the security trade‑offs, drawing historical parallels to arms‑race dynamics (World I machine guns, nuclear deterrence) and emphasizing the uncertainty surrounding AI‑enabled bio‑weapon threats, calling for proactive security research and agency preparedness.
Chapters
Anthropic refused to sign the $18 trillion market cap Open Weights letter, and Dario just explained why. On July 27th, CEO of Anthropic published Anthropic's position. Anthropic has never advocated for a ban on Open Weights models. Open models without dangerous capabilities are a public good. They don't cost anything besides the compute needed to run them. So why didn't they sign the coalition letter signed by $18 trillion worth of market cap of tech companies who believe they should? They agree that Open Weights expand access and competition, but disagree agree that open weights necessarily make it easier to develop safeguards, citing attacker-defender asymmetry and biology as an example. Dario talks about 2 nightmare scenarios. One, authoritarian governments building more powerful AI than the US, regardless of whether these models are open or closed. They see that as the primary threat. Number 2, misuse for cyber and bio attacks, where open models do pose higher risk than closed models because you can't really stop it once the weights and stuff are out there. So what Anthropic positions in this paper is instead of a ban, look at chip export controls, which are the most efficient and direct way to block that threat number one. And then cracking down on distillation as a technique, which can bring the Chinese frontier to within a few months of the U.S. frontier and mandatory safety testing for all sufficiently capable models. open and closed. Dario basically explains why they didn't sign.
The risks he talked about are not negligible, right? And I think that there's kind of like, um, maybe different perspectives about whether or not having open models is better for security or worse for security. But then there's also like the bio angle and the bio risk. And I think he's pointing to warfare, for instance, like there is a combined arms dynamic that happens on the battle lines where like one side will figure something out and then the other side has to adapt. And like during World War I, this was like really pronounced where You know, they brought these machine guns on the battlefield. Hundreds of thousands of soldiers are dying and like both sides figured out, okay, let's not do that anymore. And then they come up with tanks. They come up with strategies that minimize casualties. But then there's like nuclear weapons where one side can potentially just end the whole game with asymmetric capabilities. The main deterrence on nuclear weapons has been mutually assured destruction. Like that's been the thing that holds the balance. And I'm not sure what the balance is when it comes to like using LLMs or other technology to create cyber or biological weapons. That might be something that gets into the domain. This is something we can't contain. There is the ability for capture and asymmetric positions to hold. So again, I think like on cybersecurity, you want this back and forth of escalating capabilities on all sides. I think open tools are like a good way to create that dynamic, but I don't know that anybody has a real serious answer for these things where there could be something like a biological weapon that, you know, is not stoppable. That might be fictitious or it might be real. And I think we don't know. And I think that's the place where, you know, our security agencies need to do a lot of research and be like really ahead of the curve on this.
