Emergency Pod: Examining SB 1047, California's Proposed AI Legislation

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. Today we have a timely and important discussion about SB 1047, the proposed AI legislation that's working its way through the California State Legislature, also known as the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act. To understand this complex and all too often misunderstood topic, I've invited Nathan Kalvin, Senior Policy Counsel at the Center for AI Safety Action Fund, which is one of the bill's co-sponsors. And Dean Ball, research fellow at the Mercatus Center, to share their perspectives. We're also joined by Steve Newman, a self-described recovering entrepreneur who is now working to raise the level of discourse around AI issues. To set the stage, I can say with quite high confidence based on my own close reading of the bill and related analysis, including that of friend of the show Zvi Moskowitz, that contrary to some of the noise being made online, this bill is not some sweeping anti-technology power grab. Nor is it the cynical ploy to protect industry incumbents from competition that some have alleged. As some evidence of that, consider that California State Senator Scott Wiener, who is advancing the bill, is probably best known for his pro-housing work. And he's also proposed legislation to hasten the transition to solar energy, to extend alcohol sales hours in bars and restaurants to 4 AM, and even to decriminalize psychedelics. What's arguably most striking about AI discourse in general is how the strangeness of the technology creates space for people to think for themselves, and how that often results in the scrambling of familiar political coalitions. At the same time that we have ostensible political opposites like Tucker Carlson and Snoop Dogg emitting similar soundbites, our discussion today is among people who are generally very pro-technology and very skeptical of government regulation. As you'll hear, while Nathan and Dean do still ultimately come down on different sides of this legislation, they identify many, many points of general agreement along the way. Personally, while the bill may not be perfect, I appreciate that Nathan and his co-sponsors are making a genuine effort to focus on the critical issue of tail risks posed by advanced AI systems and not placing an undue burden on startups and smaller players in the AI ecosystem. Even though that means leaving other important questions, including the risks associated with algorithmic bias, the future of professional licensing, and the general impact on the workforce at large for another day. At the same time, as someone who has long identified as a libertarian and often been frustrated by government overreach, I'm definitely sympathetic to Dean's concerns that a new agency could exploit the bill's ambiguity to gradually expand its scope and power in very problematic ways. Ultimately, it seems to me that some form of government regulation of AI is inevitable. And so I feel that it's well worth genuinely engaging with and attempting to improve SB 1047 as it goes through the legislative process. With that in mind, toward the end of this conversation, in search of a productive synthesis, I put forward and we discuss several possible amendments that I hope might make the bill a bit better. Overall, I think you'll find this to be an unusually respectful, nuanced, and insightful discussion on what might be the defining policy issue of our time. issue of our time. If you agree, I'd ask that you take a moment to share it with your friends. And on this topic in particular, I really want to hear your best ideas for how government can help foster a positive AI future. I have an interview scheduled with Senator Wiener for next week, and I will definitely read everything that you send me and attempt to bring the best ideas to him directly in that conversation. You can reach me, as always, via our website, cognitiverevolution.ai. Or by DMing me on the social media platform of your choice. For now, I hope you enjoy this deep dive with Nathan Calvin, Dean Ball, and Steve Newman into California's AI bill, SB 1047. Nathan Calvin, Dean Ball, and Steve Newman, welcome to the Cognitive Revolution.

Thanks so much. Happy to be here.

Likewise.

So we're gathered together for a discussion about the Proposed legislation that is working its way through the California State Legislature, which is known as SB 1047, or as the authors want it to be known, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act. This legislation would be first of its kind. Obviously, we've seen the executive order, but there hasn't really been anything passed through a legislature that would try to meaningfully govern what's going on at the frontier of AI. And it's predictably been the topic of a lot of discussion, debate, seemingly some miscommunication, seemingly some misunderstandings as well. So I wanted to bring this group together just to explore those and see what we can potentially hope to come to some agreement on. Also, it's worth noting too that the, we're not yet at the point of an up or down vote. So in terms of how to approach this today. I got a bit of outreach from Senator Wiener's office in California, and the guidance there was like, this is probably going to play out over the next several months. There are going to be a lot of opportunities for people to make amendments. So we're very much still in the discussion phase. And potentially, if we come up with anything that we all feel good about, we could even recommend that to the authors. So that would be a big win in my book. For starters, Nathan, you are the senior policy Counsel at the Center for AI Safety Action Fund. You guys are one of 3 co-sponsors behind the bill. You want to just start off by giving us a little bit of an introduction on who you are, who the organization is, your kind of background worldview, and why this is something that you're motivated to work on. Then Dean, I'll give you the chance to give a similar story next.

Yeah, absolutely. So to start with the organization, so The Center for AI Safety Action Fund is like the 501 affiliate of the Center for AI Safety, which listeners might know from the statement we put out about existential risk from AI, as well as a lot of the technical research work that the center does, including by our executive director, Dan Hendricks. The 501 was partially created because we were getting lots of inquiries and incoming from policymakers, including Senator Weiner, and we wanted to have a vehicle that could do more direct policy work, which is something that 501s aren't able to do. And yeah, specifically for involvement in the bill, Senator Weiner put out an intent bill mid-last year talking about being interested in doing a bill on these issues. And he approached us and the other co-sponsors to help him really flesh that out and something in this space. But I think Senator Weiner became concerned, as a lot of us have, that it seems like The pace of development in AI is really exciting and I think has a ton of benefits, but I think we are starting to see some of these glimpses that these really large models have pretty dangerous capabilities. I think we saw similar discussion at the AI executive order, and I think that's important, but I think there are things that you can only do in statute, and California is an important jurisdiction and so it seemed like a really great opportunity. And I've been really impressed with him as an author. He's a really smart guy and has been really into the Details on this, and it's been really just a pleasure to work with. And I'll see our other co-sponsors of the bill: the Economic Security Project and In Code Justice. Yeah, just speak a little bit about my own worldview about this. I think that I am just very uncertain. I think one of the main things which I think is that I have a lot of excitement about AI. I think these tools are really amazing. I use GPT four and Cloud Priopus on a daily basis, and I'm a big fan of these models and think that they have. A ton of potential to improve our lives and work and to improve, bring a lot of innovations that we desperately need to solve really hard problems. I'm also someone who in general is like very, I think, critical of unnecessary regulation or bureaucratic prestige. Big fan of Senator Weyner's housing work, fan of trying to build additional nuclear energy, of trying to reform the FDA. A lot of the things that I think you and I on a separate podcast would probably chat about and have a lot to agree on. I just happen to think that really the largest AI models, that in addition to their potential to do a lot of good, I think they might be able to do just really harmful things and that we should be having some amount of precaution and kind of leaving it totally up to the companies and voluntary measures about whether they want to be doing testing. I think we're hearing from folks in the labs that there's just tremendous commercial pressure to get things out the door as fast as possible. And I think there needs to be a role for government to say, let's be thoughtful here. Let's get the benefits, but let's make, not make sure that we're, you know, racing ahead and, and causing catastrophe. I think the one thing I, you know, want to say is I'm a very big fan of the general structure of the bill and we'll defend that very strongly. But I think a lot of the details about specific points, like this is a complicated issue. The bill has already been amended. It will continue to be amended. And I'm really speaking in general in this. I'm going to be speaking for myself and Center for AI Safety Action Fund. But on this area, I'm confident the author also shares this, that He's really interested in amendments. We want to get this right. There are a lot of different technical terms and definitions. We have our clear intent, but we want to make sure that those definitions reflect what our intent is, and it's really to target some specific, very dangerous concerns of models while really generally letting the AI ecosystem bloom and flourish as it's been continuing to. So that's my general standing. I think I'm worried about a variety of different risks. I think I'm worried about like a super powerful models that didn't counter humanity. But I think more, more practically, I'm also worried about the ability for bad actors to use this for malicious bioattacks, really severe attacks on critical infrastructure. I think some of the risks that were identified in the Biden executive order from dual-use foundational models, I think similarly, novel GDRM developments, attacks on critical infrastructure, and models that deceive or self-replicate or otherwise really hard to get back control of. So that's what we're broadly thinking. Also care about, I think a lot of other issues also really important within AI and have talked with everybody about lots of things about deepfakes and the future of work and racial bias in government systems. And all those are super important that I think are also important discussions to be had. But this bill is really focused on kind of these catastrophic national security risks from the largest models and similar in DNA to what was attempted in the Biden border. But that's broadly where I'm coming from. I think that we do need to take some action here. I think that like, we really are on an exponential curve. And I think that there's some temptation to be like, this is really complicated, we just wait a year, we should wait until we can make everyone incredibly comfortable with it. I think legislation takes a long time. We already have in the legislation, and as I think some folks have noted, some like delayed implementation, where even after the bill has passed, it takes a while to get government set up around these things. And I really do think that it's important, if at all possible, to pass something this session so we can start to that process going. And I really think that not how it's being interpreted and on Twitter and things like that, but I think there are already a lot of concessions in this bill and trying to be a moderate, not maximal proposal. Anyway, we can get more into all this, but that's probably where I'm coming from.

Quick follow-up question. When you refer to the author of the bill, is that State Senator Wiener?

Senator Wiener.

Yeah.

It's terminology is weird because at the federal level you refer to the bill author as a sponsor, which is the opposite. So when I say the author, Senator Wiener, we The author has provided like technical advising and feelings about amendments to language, but ultimately him and his legislative director have been very involved in this and have made the final call on all the inclusions and the direction of the bill. And, but yeah, the, when I'm referring to the author, I'm referring to Senator Wiener and his team.

Cool. Okay. Thank you. Dean, you are a research fellow at the Mercatus Center. You've been a previous guest on the podcast when we talked about all things brain-computer interface. So obviously a big and pretty deeply technical engagement with technology, AI, all these sorts of things, though your day-to-day focus is on policy analysis. Give us a sense of where you're coming from on this.

Sure. So first of all, I am at the Mercatus Center. That is a think tank affiliated with George Mason University. We are a 501, and what that means is we're in the same tax status as a university, but we try to educate the public and policymakers about matters of public policy using insights from academic research. So that's where we come from in a broad sense. Specifically on AI, I think that AI is a general purpose technology, quite possibly the most powerful general purpose technology that humans have ever invented. Certainly the most powerful one of sort of the recent wave At least it could be, of general purpose technologies, computing, the internet, et cetera. As a general matter, we don't tend to regulate downstream use of general purpose technologies with a centralized government regulator. That's just not typically how that's done. There's not a Department of Electricity. There's not a Department of the Internet. There is FERC, the Federal Energy Regulatory Commission. They set standards for interstate transmission of electricity and all kinds of stuff like that, but they're not regulating my downstream use of electricity, which surely can be used to do all kinds of dangerous things. Nor is there a Department of Computers. I think that if we want the AI revolution to be successful and to broadly benefit people, then even to speak of AI policy as a discrete area of public policy Is probably going to seem a little bit anachronistic in 20 years. That's my hope. I wouldn't necessarily predict that, but that is my hope because I hope instead that AI is so integrated throughout our economy that it is regulated in all the myriad ways in which economic activity and public policy interact with one another. And so I think that Creating a centralized regulator is just likely to lead to all kinds of political economy problems that will hinder the impact of AI, the good impact of AI, in ways that I can't even predict. And I'm not really sure that it gets us that much more safety either in the long term. In particular, because most of the concerns we have here, CBRN, cybercrime, these things. Certainly they could come from domestic actors. Nathan, Calvin, you referenced national security concerns. I don't believe that the CCP cares about qualifying for a limited duty exemption under the state of California, nor do they care about the NIST AI risk management framework. So I'm not really sure that this gives us safety in any meaningful way from those types of And instead, I think to the extent it hinders AI progress, it actually creates danger, which is what regulation often does of these kinds. It creates a false sense of security, but in fact has a tendency to lead to long-term danger. I really think that nuclear energy is a good parallel to think about here, except for at least nuclear plants, there were some nuclear incidents. That raised a lot of concerns across the world. Um, there is no AI Chernobyl so far. There is no AI Three Mile Island. And we created the Nuclear Regulatory Commission and illegalized nuclear energy effectively, um, after some major incidents. There have been no such incidents here. So if we're going to illegalize AI progress, I'd at least rather do it after we have at least one concrete example of A major problem. I just think not only is it premature, but it's the wrong way to go in general. So that is, I just want to be very clear though, I am not opposed to governments passing laws with the word AI in them. I'm not opposed to a vigorous government role in the regulation of AI. I just am skeptical of the idea that there is one big thing you can do, one major bill you can pass that will do what you want it to. Instead, I think the answer is every agency internalizing the risks and benefits of this technology in a serious and considered way and taking steps within their own jurisdictions and their own areas of expertise across the city, state, and federal levels. I think there's not one big thing. There are 5 million small things. And that is the work that I hope we can do. And I think that frankly, centralized regulators hinder that work.

Thank you both.

Steve, to introduce you briefly as well, you are an exited entrepreneur. Most people will have interacted from time to time with one of your creations, which is now known as Google Docs. And having exited that line of work, you're now focused on raising the standard of discourse related to AI, which is obviously a noble pursuit. I think we're in a pretty good position to hopefully have some elevated discourse here because for one thing, I don't hear very extreme positions like you will sometimes go online and find people saying things like, oh, who cares if humanity goes extinct? The machines will be more worthy in some sense than us anyway. Not hearing anything along those lines. I'm also not hearing argument from protection of weaver employment, so to speak. I'm not hearing any, anybody saying that the current mode of production and the current employment relationships and dynamics are like the primary concern. It seems like everybody on this call is broadly pretty enthusiastic about the upside of AI and technology in general, and broadly, and certainly this is true for myself as well, skeptical about government overreach into areas that are probably best left to develop on their own. Do you have any other thoughts on how we can make sure we have the most elevated discussion that we can? I think the first step is just bringing together people who have a positive attitude and are interested in engaging and with an understanding that we're going to have a positive discussion. I think you've done a great job of queuing that up. And I think really looking forward to seeing Nathan and Dean dive in. And I think we've already seen just from everyone's introductory remarks that we're here to kind of Talk about reality and talk about things in a reasoned way. And, and that's 90% of the battle right there. Hey, we'll continue our interview in a moment after a word from our sponsors.

I want to pick up, Nathan Labenz, on something you said about protecting weaver employment. This transformation that is coming will be an economic transformation. I don't think Anybody has intelligently modeled the labor market impacts of AI. I know some people who are trying. I don't know how possible it is right now, but it's going to be a struggle because there are a lot of areas of the labor market that are protected with barriers to entry and barriers to entry will be erected to keep AI from, from entering, whether it's driverless cars, medicine. law, there's a thousand things where, you know, current industry groups are going to try to continue extracting rents. And that's going to be a real struggle. I think that will be a huge policy struggle over the coming decades. And if you don't believe a centralized regulator will be a weapon that will be used against AI for diffusion, I think that you have not been paying appropriate attention to the way that government has generally conducted itself at the state and federal level for my entire lifetime, at least. So I think that's one thing that's important is that I am very concerned about maximizing the benefits. I want to do that. And I think a centralized regulator presents all kinds of challenges to doing that. That's just one additional point I would make.

Yeah, I think probably you'll have agreement in the general sense that we can find plenty of examples of government gradually overgrowing the original intent of the legislation under which it's operating. Maybe we can approach that from the standpoint as we get into the details of like, how can we prevent that from happening in this case? I often say, when people ask me like, why should we want any of this? Isn't this all just going to be terrible? I'm like, AI doctor. That's my first. When I first got my early GPT-4 access, I couldn't stop asking it all these medical questions. Not that I have that many, fortunately. It was really good and so convenient. And that is, in my view, almost the canonical thing where if we end up in a world where we prevent people from getting quality medical advice from AI, whether out of irrational fear or bad cost-benefit analysis or protecting doctor employment, it seems like we'll have made like a A very bad mistake. So if that's something we can agree on, then we can get into how that tendency, which definitely, you know, is out there, could be mitigated or prevented. Fair?

That's fair. I guess I just want to say that I agree with you that there are a lot of really large benefits here, and I think it would be quite bad to not get them. And I think some of that will involve changes in the future of work. I think among the coalition that we're working with here, there are like a variety of different Views with it among it. And I feel like to some degree, like, this bill is not necessarily taking a position either way on the, like, labor market future and whether you should be trying to have the unemployment. I have my own views about that, but I just think either way, I just don't think it's what this bill is about. And that if you are someone who is either concerned or optimistic about AI displacing human jobs, I don't think the bill is necessarily going to have a lot to say in either direction, to be totally honest. Just want to clear it. Clarify that. And then also, and I think we'll dig into this a lot more, I would push back on the idea that this bill is a centralized regulator. It's not a licensing regime. And the only government body that is created, a new division of the California Department of Technology, is just like receiving reports and certification. It doesn't have rulemaking power. It doesn't issue licenses. Most of this is a liability regime through the Attorney General. So that still is probably more centralized than you would like, but I think there are other proposals that really are like centralized AI regulator where you're getting pre-approval to be able to do training runs. where they really are doing the tech themselves, doing a lot of that. I agree that there are really big challenges of that, of having government capacity to really understand all the things that are going on at the cutting edge and doing that effectively. And, and I just want to say, I also, I think, have concerns about the efficacy of a centralized regulator, particularly given the existing technical expertise or lack thereof within government. And so I'm sure we'll continue to debate about how we're characterizing the bill, but I, I also have hesitations about a centralized government regulator here, and I, I don't think that's actually what this bill does.

Only thing I would say is the Frontier Models Division in a recent amendment to the bill does have the power to change the compute threshold. It has the power to change the specific standards for models that fall under its jurisdiction. So that already is starting to look a little bit centralized. And the other point that I think is underdiscussed here is that we talk about Repeating the mistakes of nuclear energy. The Frontier Model Division, because of California's current $60 billion budget deficit, the Frontier Model Division is funded by fines and fees that are levied on AI companies. And that's exactly how the Nuclear Regulatory Commission is funded. The regulators, the regulated entities pay for their regulation. This obviously creates regulatory capture potential, right? Like on its face. Their salaries are paid for by the people they regulate. But in addition to that, it insulates the frontier model division from the normal appropriations process in legislatures where maybe 5 years from now, we're not as worried about AI risk. Maybe all the concerns of folks like Geoff Hinton don't pan out and we want to lower the budget for that. But guess what? You can't because the FMD funds itself or it's inconvenient to because the Frontier Model Division funds itself from the AI industry. And so not only does that insulate it from the budget process, but in essence, it insulates it from the trade-offs of the political process of democratic input. And I think it's probably worth California think maybe thinking seriously about why they have a $60 billion budget deficit before they create a new centralized regulator for the most promising technology in the world. But also I think that it would just create all kinds of problems, just the funding alone.

So we'll come back to this question of how centralized the regulator is in a minute. I have done my own close reading of the bill and broken it down into basically 6 things that I would say it does that seem to matter. And we can then dig into the details of Of each one, either one by one, or you can add categories to the list if you think I missed any. First one, as I think about it, is it sets a standard for what are we talking about here? What is the actual risk that the bill is trying to address? There, it sets what I would say is a pretty high standard of either mass casualties or $500 million or more in damages to things like critical infrastructure. So that for me was like, okay, that's a pretty big thing, right? Like mass casualties, $500 million in damages. I think if you were to go ask most people on the street at what level of harm would you think the government should start to get involved in AI, they would probably come up with something that is lower than mass casualties or $500 million. I don't know if Dean, you would say that you would be worried would be subject to future discretion or lowering of that threshold, but for me, it's a high number.

So I don't think, I don't know this for a fact. I don't think that the bill allows the Frontier Model Division to lower the hazardous capability threshold. I don't believe it can do that. However, I would say the following things. Critical infrastructure is an extremely deceptive term. Critical infrastructure is, it's a little bit Soviet. In the sense that critical infrastructure is in fact a larger category than infrastructure. So critical infrastructure includes things like hospitals, all financial services, all telecommunications infrastructure. It is the majority of US economic output. So when you say critical infrastructure, don't think that you're talking about a small subset of things like nuclear power plants or something. You're actually talking about most of the US GDP. $500 million is a lot of money. At the same time, cybercrimes worldwide in 2023 were responsible for about $9 trillion of damage. So $500 million, not indexed to inflation, that could be not so much money. It already is not so much money. You could probably, depending on how you calculate it, if you knock a wastewater treatment plant out, you don't blow anything up, you don't do anything like that, you just knock its computers out for a couple of days. You've probably done $500 million of damage to critical infrastructure. And obviously we don't want people knocking out wastewater treatment plants. I'm not saying that's like a light thing. I'm just saying that you should think very carefully. Wastewater treatment plants go down, right? They go down a lot. Cities issue boil water notices on a regular basis in this country. We shouldn't think that's like some unprecedented thing. And just because it happens to be caused by AI, it's just a little like Uh, I don't know. It's the new thing, right? But, oh, it was caused by computers. I don't know. Like, I think that it's a big deal and we should do everything we can to protect our critical infrastructure and be very serious about cybersecurity. But I actually think $500 million is not as big as it sounds.

So to read the definition from the bill of critical infrastructure, it says critical infrastructure, quote unquote, means, and it is pretty broad. Assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.

Yeah.

So that is, I think we are probably really relying on the number there more so than any sort of narrowness of definition, because as you said, like a lot of things, you know, would have the potential for a debilitating effect on physical, economic, Security, public health, safety. Now, this is maybe more of a nitpick than anything. I'm not sure it really matters, but for what it's worth, that $9 trillion cybercrime number, I just came across that and I don't really buy it to be totally frank, because that would be like pushing 10% of global GDP. If global GDP is $100 trillion and US is like $20 trillion. Where that number has been flying around, I've seen that actually used on the On both sides of the debate now, because I've also seen people on the AI safety side being like, look how big of a problem this already is. And I'm like, there's no way that we're losing 10% of global output to cybercrime. That just seems crazy. I actually saw it in a fundraising deck too long ago for an AI assurance tech company. And I was like, I'm going to have to downgrade your market size a little bit.

But— Well, no, but because of, I don't know how that number is calculated, but because of the broken windows. Phenomenon that some of that $9 trillion might be accretive to GDP, right? It's not necessarily that it's a $9 trillion reduction in GDP. Costs of things are accretive to GDP, right? If I break a window and I fix the window, that's GDP. I would just point that out. And I think, but I think that that $9 trillion figure, it might be like totally fakakta. That's entirely possible. That should tell you something about $500 million In the context of cyberattack damage, how do you calculate it? I don't know. You could, there's probably ways you can fudge those numbers. Yeah.

So let me suggest we take a step back for the moment. Dean, I'll address this question to you.

Should we be regulating anything? Is anything bad at a large scale plausible to happen? Or in other words, are we debating exactly what's the best way to write this bill or are we Actually debating, is there even a problem here?

So I would definitely say we should be regulating things. I think I have a rather circumscribed perspective on that which government can effectively regulate through a single statute. I think there are many different things that government should be doing to incorporate AI into various criminal statutes and scale up their enforcement of things and increased defenses and all that kind of stuff. But is your question, what do I think the outcome would be if the bill passes?

So the bill doesn't pass, if no bill passes.

In other words, we're talking about the wastewater plant being knocked offline.

And from one perspective, like that could be a really big problem, could cause, I don't know, half a billion dollars in damage. From another perspective, it's a big economy and a big world and big problems happen. And maybe if AI was a little bit involved there, it doesn't really mean that was fundamentally an AI issue. Taking a step back from the specifics of half a billion dollars, should we be worrying about AI? Is there a problem that we could be regulating, whether or not this regulation is the right way to go about it?

I get the sense you're asking about like catastrophic risk. Am I worried about sort of existential type major, major risks from AI?

I'm not trying to go all the way to existential, but just big stuff like Cyberattack that brings the whole US grid down for a month or knocks all the wastewater plants offline or things that start happening in a big way that traditionally have not happened in a big way or just a lot easier to happen or rather, should it be half a billion or should it be $2 billion? Just is there a here or there in, in, in, as you see it?

So I share Nathan Calvin's and I think probably Nathan Labenz's general uncertainty Like fairly strong uncertainty about, about the, the future trajectory of AI capabilities. So I would say, yeah, like we should take the, the concept of a cybersecurity attack that takes major electricity grids down for a long period of time. I think that is definitely something that we need to be worried about. I question the wisdom of passing a law that says it is illegal to take down the US electricity grid, because I suspect that it already is. And I think that basically if we pass a law that says it's illegal for the AI transformation to not go well, well, that doesn't do much for me. That doesn't advance the epistemic ball in any important way because we all want it to go well. We all want this transformation to go well. We don't know what that means. Nobody knows what that means. And if the government just passes a law that says it has to go well, then we all have this epistemic uncertainty and now you've introduced lawyers into the equation.

Yeah.

Not that lawyers shouldn't be part of the whole, this whole dynamic, but it's, oh great, cool. We can go to court and talk about this. Sounds fun. I just, I question the wisdom and really the efficacy of laws of this kind, but I would definitely agree the AI existential risk, inherent risk. I'm not all that worried about Claude 4 itself developing its own motives and trying to kill all of humanity or whatever. I think that's a little bit of a jejune debate. But, but I'm definitely worried about future models being used for all kinds of things. And cybercrime is at the top of the list for sure. So yeah, there's definitely a there there in my mind.

One thing that I think could be helpful is that I agree with you that in a lot of settings, the right way to handle regulation is to focus on kind of the downstream uses of the technology. I think it's fair to say, look, we're not going to You know, place a bunch of burdens on email or the internet. We're going to focus on people who use those tools to do bad things and go after them. And that's appropriate. And our laws are able to handle that. I think there are lots of settings where that works. And I think that's appropriate. I think there are settings where that doesn't necessarily make a lot of sense. We have things like in, in nuclear policy where we don't say, oh, we're going to go after terrorists who use plutonium or something. We say that, no, the people who are enriching nuclear materials have some obligation to secure those materials and to be thoughtful about it. I think similarly, I'm very enthusiastic about biology. I think biology has tons and tons of benefits, and I'm very optimistic about a lot of different work there. I don't think that therefore means like we should not think at all about gain-of-function research. And if folks are doing lots and lots of gain-of-function research, that therefore our solution should be, hey, let's just go after the individual terrorists or folks who are doing that. I think that we have to have some process also higher up on the chain. So I agree that I think that this is made difficult this way, but I think in a lot of cases it made sense to focus on the downstream applications. I just think that really advanced, the largest AI systems that have a potential of having these pretty devastating impacts are one of a small handful of technologies where we do need to be thoughtful about whether regulations are necessary.

I agree with that general framework you've laid out. The fiendishly difficult aspect of this though is that plutonium is not a general purpose technology. It's not a consumer technology. That I might use on my phone. So that complicates it considerably. I think I agree with you that there is room for regulation and, and a need for regulation somewhere between the downstream use and the creation of an AI model. I actually think AI might just be a little too low in the value chain or the production chain of threats, one might say. Um, so a concrete example is, uh, the Biden administration just a few days ago released what in my mind is a very productive framework for nucleic acid synthesis machines.

Also a big fan.

Yeah. Where they basically said, if you're going to make, if you're going to do federally funded life sciences research, you have to use machines, DNA synthesis or nucleic acid synthesis machines. From specific providers that will comply with these standards. And those standards include that sequences that you wanna produce, genetic sequences you wanna produce, have to be screened against a database of known threats. Great. Excellent. But I think that sort of going one level removed from that to like the statistical analysis of genetic datasets and regulating that, you know, I'm an American. And that does raise constitutional problems in my mind, right? I think I do have a constitutional right to do statistical analysis on genetic data. I don't think I have a constitutional right to synthesize DNA. And I think that's like a pretty clear distinction that like at least most Americans can intuitively grok. Constitution, as Zvi Moskowitz has said to me, Constitution's not a death pact. So I'm not gonna rely just on a constitutional defense. I will also just say that it's too low to be a productive regulation. It's like regulating metallurgy rather than missiles.

Hey, we'll continue our interview in a moment after a word from our sponsors. This is a good time for me to interject the second point of the 6, and maybe I should even just run all 6 down real quick. Okay. We covered point 1, which is the standard of the scale of harm that would be of concern. That is, again, mass casualties, $500 million damage to infrastructure. The second point is, how would you reasonably expect that you might be playing in that regime? Here, the bill uses the same threshold as the executive order, at least on the first definition, which is 10 to the 26 FLOPS. So if you're going to train a model that is going to use 10 to the 26 FLOPS going in, then that is presumed to be a covered model. That is to say, if you can't make a positive assertion for good technical reasons that this thing is going to be safe, meaning it's not going to be capable of causing these sort of downstream mass casualty or $500 million worth of harm, then you have to do certain things. And those things are, you have to implement a lot of safety standards during the training process. So the, the weights can't leak to the public or to the Chinese government or whatever. You have to, that includes having the ability to do a full shutdown. I'm not exactly clear on what would constitute a full shutdown, but the idea is basically you want to have an off switch. So if things get crazy, you can flip that switch and stop the process. You have to, at the end of that training process, again, this is assuming you didn't have good technical reasons up front to say that it'll be safe. Then at the end of that process, you then have to do the testing. And say, okay, now that we've made this thing, let's see what it is actually capable of. And at that point, you have another chance to say, okay, actually we did all this testing and now we can say, even though we didn't have a theoretical reason up front, we now have a practical reason based on testing that we can confidently assert that it's safe. And then if you still can't do that, you have to deploy only with surrounding safeguards, which are meant to prevent these large-scale harms from happening. Basically, you have 3 tiers. You could say, I have a theoretical reason, and I really love to see that emerge, right? That would be the real silver bullet, which doesn't seem like anybody's super betting on at the moment, would be like a conceptual breakthrough that, hey, if we do these things, design the system in this way, create the dataset in a particular way, whatever, then we can just be confident that this is going to be safe from the get-go. Now everything is awesome downstream. There's, you know, minimal imposition, right, from the bill if you can make that kind of assertion. But if you can't do that, then you got to hammer it with a bunch of tests, find out what it is in fact capable of. Again, you have a fork there. Hey, we tested as much as we could. We determined that it can't do these things. So we're— or if still not, then we have to have guardrails, which would be like moderation, monitoring of usage, and potentially also Not releasing the weights into the wild, because this is probably one of the more contentious pieces of the bill, that if you do release something into the wild, then modifications of it downstream are still your responsibility. So you have to be able to assert not only that it will do this in the form that we release it, but also that it can't be like relatively easily modified to that state. And this is where I would refer listeners back to the FAR AI episode that we did with Adam Gleave, because they showed that it really doesn't take much in a lot of cases. If the capability is in the model, then the sort of RLHF finishing that they are generally treated with at the end of training is maybe enough as long as that stays in place, but very little fine-tuning typically downstream can reverse those sorts of safety fine-tunings. Even in a non-malicious way, that was one of their most notable findings is that even just taking a small chat dataset and saying, hey, we don't, we want to have you chat in a particular way, not training it to do bad things, but just omitting the fact that it's supposed to refuse bad things, that that final fine-tuning step can, in a way that the developers didn't even intend, reverse that. So it's on the model trainer to make sure that not only is this thing not exhibited, but that it can't be easily unlocked. With downstream fine-tuning. So that's like the big one. And then the one other thing that's worth noting there is, and I think this is also where a lot of the debate and the kind of heat is right now, is that 10 to the 26 FLOPS is like clear enough. But then there's also this other definition that's if you're going to train a model that could reasonably be expected to be similarly powerful to what 10 to the 26 would get you, then you're also in that regime. And that's where people are like, well, I don't really know if I'm in that. Regime or not. So anything that you guys would clarify or refine on my description of that? Obviously a lot of people have offered their summaries, but how would you grade mine?

I think good summary. Definitely hear you on the point that, yeah, I think one part that, that is harder to know is saying, I know I'm creating a model below 10 to the 20th, but I'm not sure whether it will be as capable as a model at beta of what we are at 10 to the 20th. In 2024, how can I be sure whether I then have to do these things? And the offer added a provision which says that basically if you are surprised by how good the model is, that's fine. You're not in trouble. Just like institute those safeguards after discovering that you were surprised, and that's okay. And so I think just some understanding that like maybe in the future we will have more accuracy about exactly how high performing a model will be prior to training, but at least in the, the current regime, we, we recognize that you might not know that until after you've started. Until after you trained the model and started doing various things about its capabilities.

Just to give people a sense of what is 10 to the 26th FLOPS, first of all, it's not entirely known whether any already trained model has hit that threshold or not. If it has, it's almost for sure happened at either OpenAI or perhaps Google. The biggest Meta-trained, still in training, Llama 3. Is expected to be halfway there. It's funny because it's I think it's been estimated at four times ten to the twenty five, which is sounds really close, but it's still only forty percent of the way there because orders of magnitude. It's the last one is always the the one that matters the most. So what Meta is spending on this Llama three four hundred B still in training now is at retail price. Something like $200 million in compute. They have published a bit of guidance on how many H100 hours they've put into the first 2 models that they put out. They put in over 6 million H100 hours into the 7DB version. And so extrapolating, Andrej Karpathy had a nice analysis of this, which I'm taking as closest thing available to ground truth for these purposes. And then just putting a price on what you get in the market today, an H100 hour, if it's not dramatically subsidized, pushes usually about $4 an hour for an H100. So roughly speaking, you do the back of the envelope math, you're getting to about a $200 million compute budget for the current Llama 3. So you're in, you're in somewhere in the sort of hundreds of millions to half a billion dollars. Just to buy the 10 to the 26th FLOP. Now I know, Dean, you may have some thoughts on this because I know you've written a bit about how you think this is not a great way to frame this, in part because efficiency is definitely a major driving force here. I don't know if there's any alternative that you would propose if we were going to keep the general structure of this. Is there anything other than a FLOP number that would make sense to anchor off of? But in any event, that's a sense for the budget.

I would say a couple things about the compute threshold. First of all, yes, it is the case that oftentimes regulation will be triggered by a firm's size, the revenue that they have, the number of customers that they have, whatever that might be. Firms as a general rule want to get bigger. When you impose thresholds like that, there's ample evidence in the economic literature that when you impose thresholds like that, you create all kinds of weird problems where firms will just skate right below the threshold. The difference is that firms, as a general rule, want to be more compute efficient. So with a threshold like this, you are swimming against the current, and I think you're ultimately incentivizing the frontier model division to lower the threshold over time. And they can do that under this bill. And The other thing is I'm just, for me, it's not entirely clear what 10 to the 26th is based on. It is, I've never really heard a scientific basis for 10 to the 26th blocks. So Nathan, if you have any answers there, I'd love to hear it. But that to me is the main thing about the threshold. But I think we shouldn't be regulating at the model layer. And I think that the fact that it is so hard to come up with good thresholds and make this stuff legible to the state. Is like evidence in favor of my argument. So I don't have a good alternative solution. My alternative solution would be not doing this and doing other things instead. It would be not doing nothing, but doing other things.

Great. Just, just to briefly justify where the 10 to the 26th number comes from. I, I think the idea here is just that this is the next generation of models. So not GPT-4 and Claude 3.0, but really the ones that companies might be releasing potentially later this year or, or next year. And I think it comes from the understanding that like, We don't think that any of the current generation model models are capable of causing these really severe harms, but we think that there are like some glimmers or warning signs that future generations that at least they should be like keeping their minds open to the potential that they could. And so I think that's where that is coming from. There's stuff like OpenAI's study on biorisks from GPT-4 saying, look, maybe it helps like a little bit, but it's really not very much and it's novel stuff that you couldn't find on Google. But saying that, look, as we train larger models, we keep our minds open to the fact that this really might provide quite substantial assistance to a layperson in creating a really nasty biological weapon. The idea is that we have some sense about what the capabilities are of current models. I think on the lower threshold thing, I think you're talking about it's guidance about what sorts of thresholds might be then like covered under like B. It's not a formal rule, it's guidance about it. There was consideration about giving them rulemaking power to change the threshold, and we didn't do that.

Yeah, I guess I, I hear that. And I think if we were debating all of this and the current frontier was GPT-3 or GPT-3.5, we would 100% be saying this about GPT-4. We would. And, and in fact, people did. People said it about GPT-4 and GPT-3 and GPT-2. And not to say that it's not eventually gonna become a thing, I, I, I'm not dismissing that entirely, but it It does seem to me like a vibes-based argument for imposing a threshold like this of it's the next generation and we think that could happen maybe, but otherwise we have no empirical or scientific basis for these claims. I just think we should not create an unprecedented regulatory regime for software. And that is what this is. It is an unprecedented, we have never done precautionary principle, generally applicable precautionary principle or liability regime for software. Never done it in the history of this country. This would be the first time. Software has given us a ton of economic and social benefits. So I think the bar for that is high, and I just don't think we think the next version might be dangerous is a sufficiently high bar for something so unprecedented.

I do think it's fair that if we were having this conversation around when like GPT-3 was released, that people might say maybe there are some warning signs. And I think basically like OpenAI and Anthropic and the companies basically did the things we're describing in the bill, right? Like They tested them, it's enough to determine like, hey, can't actually do these super nasty things and we can release in the form where that's okay. I think it would've been fair then to say, hey, there's a potential they should do these types of things, which I think they in fact did, and then released the model and everyone's using it, including myself, and it's great.

Mm-hmm.

I think there's some level where you should just have uncertainty. I think at this level in the next generation, we're at that level. I think it's debatable. And I think from where they were sitting with GPT-3, I think they should have kept an open mind to the fact that GPT-4 might have these things. Turned out it didn't. I'm glad it didn't. And maybe someone could make a more bulletproof argument for why it definitely wouldn't. Again, I agree that this is a weird situation we find ourselves in, different from a lot of the things. I don't know, we don't have with like email, the creators of email and the leaders of all the email companies signing open letters saying it might kill everyone. Fundamentally different thing here. We have these rules of thumb and these things have worked, but I think we really are like, NVIDIA is worth more than Saudi Aramco or what? Maybe it like went down a bit. This is like a crazy unprecedented wild thing that people are promising might be able to solve these crazy problems and do all these insane things. And I think it's not unfair to say that our like normal intuitions for how to regulate general purpose technologies or software, we, I think there's reason to think that these really incredibly large models with capabilities that we don't understand, it's an area where our intuition should run.

Yeah.

2 important things there from my perspective. One is that this is qualitatively an unusual situation. All the effort that is going into making the models agentic or capable of mid-range planning and overcoming barriers, finding solutions to the first problems that they encounter. This is definitely a horse of a different color compared to normal software. We're only on the verge of really entering that moment. GPT-4 can't do much on its own. It falls down at the kind of friction point most of the time. And yet, as Zuckerberg recently said in his Dwarkesh interview, like you observe these weaknesses, you have this process of building a lot of scaffolding to try to overcome those weaknesses, but you basically have a tick-tock development cycle where you build the scaffolding and you also try to build the next generation of the model so you don't have to have all that scaffolding. And so it seems pretty clear that the next generation coming out of OpenAI, and this has been broadly reported on is going to be a lot more agentic. And I basically define that as saying, if you give the model short instructions, it can unpack those into a plan and then go pursue that plan until it either succeeds or dies trying, or somebody comes in and turns it off. Right. But the, there is a qualitative difference to the sorts of things that we're starting to get into now. That's also measurable just in terms of like how close these things are to Expert performance in a lot of different domains. They're already better than the average human. They're closing in on human expert, and so there's not that much farther to go. Either progress would have to like totally stall out, which definitely doesn't seem like it's a bet anybody really wants to make, either commercially or safety-wise, or something else really weird would have to happen for us not to be entering into very unprecedented territory. The other big thing is. Key point is the leading developers basically are doing these things, right? If we consider the live players today to be OpenAI, Google, and Anthropic, those would be like definitely my top 3 global model developers. Anthropic has led with a responsible scaling policy. OpenAI had a pretty robust, I would say could have been better, but nevertheless non-trivial effort in GPT-4 safety a year and a half ago. Clearly they're Bringing way more energy to that now with the next generation. And Google, if anything, gets criticized for being too slow and conservative. So it seems like all of the leading companies are basically already doing something like this. So then you would say, okay, who is this bill really for? And it seems like it's for Meta, maybe slash like the next half dozen companies that might say, hey, we do want to participate in this. multi-hundred million dollar training scale game. And it's not that many companies that could plausibly do that and expect to get anywhere close to the state of the art, but there are a decent number. So it seems like it's really for them and maybe also for the decisions that they might make to open source if they can't, which again comes really right back to Meta. The decisions that they might make to open source if they can't make certain assurances about what might happen with light modification downstream. Fair analysis?

I think that's fair. One thing that has been the subject of a lot of conversation online, and I just want to set the record straight, is that there's been a lot of folks saying you're trying to make it so that no entrants can compete with OpenAI and Anthropic and Google. And that this bill is basically like secretly authored by them. They're the ones doing it. That is just not true. The bill also has like provisions around like whistleblower protection, but different things they have to do that they're not happy with. I want to give them props and recognition to the extent that like, I do think that these last 5 folks are the ones who have taken these issues very seriously and have put out these practices that like we think are good and different independent experts with no financial association with them, Jeff Hensin and Yash Ravindra and others like are saying are good. And we also, we have one in view, which is like one of the upstarts meeting with them, like to work the bill and said doable. I don't think like that crazy of a super radical thing. I think these are things that are doable. And what we're just saying is you're doing this now, but as the raids intensifies, don't stop doing it. So important. And also new entrants coming in and think that you want to not take these pretty basic common sense precautions. That's not okay. But that's some of what we're getting at. And I think it's fair to recognize that I think We really are trying to look at what different companies and folks are doing that's being smart and appropriate and saying that, look, you've made voluntary commitments. The White House has been doing this, you know, things like that. But we don't think that these should just be voluntary commitments. We don't think these can be things you just decide you're not going to do. If your investors are really pushing you, like, we've made clear that we want you to do that.

I think that's interesting because this bill doesn't have any risks associated with it other than sign a document that allows you to be sued. Like it doesn't, there is no this in this bill. There are no, because we have no standards for what companies should be doing. If we had those standards for either red teaming or alignment standards, safety standards, whatever you wanna say, I would be so much more in favor of laws that say, for example, if you wanna commercialize an AI model in XYZ industry, then You have to refer to these standards. To do that, to get to that point, which would actually be like standards like that would actually be moving the epistemic ball forward as opposed to saying, you guys figure it all out. And if we don't like what you do, we're gonna sue you or potentially bring perjury charges against you. I know the perjury thing is complicated, but certainly there's a civil enforcement aspect of this bill. It's very European. That's very much how like EU regulation works, where the EU says, do XYZ, you figure out how to do it. And then if we don't like what you do, we'll come after you. And we'll like, we'll let you know we don't like your compliance by suing you or fining you, which again, the fine structure here, how big would the fines from the Frontier model division be? We have no idea. And they can set them to be whatever they want. So that's not great from a political economy perspective. But I think that we're getting the order of operations for good AI policy or AI regulation entirely backwards with this bill. I think that what we need to make good laws are good standards, and what we need to make good standards necessitates active scientific inquiry into AI, and active scientific inquiry is best supported by open source investig— by open source software, by open investigation in a decentralized way by many different people. And this bill makes all of that much harder. So I think that what we instead need to do is take a process of science to build standards to then create laws. And we're doing it in the opposite direction right now.

Yeah, I guess my immediate reaction to that is that sounds almost like the case for a pause. If you feel like We don't know what the standards even should be.

I don't make the case for things that won't happen, so I don't, I don't make the case for things that aren't possible. So no, it is not a case for a pause. It is a case for scientific investigation.

I agree with you. I would love to live in a world where we could sit down for 5 years and figure out what these rules and standards should be and then write those clearly into law. I just think that If we sit down and do that process for 5 years, we might have horrible bioweapon attacks and critical infrastructure attacks and like crazy agentic models running around the internet committing lots of crime.

But in what world is that actually true? If agentic AI models are running around the internet committing crime, society adapts, right? There's this model that I think a lot of people that are fixated on AI safety have where society is static and like things happen to society. And society is this inert object, but that's not true. Like you're living in the symphony, you're part of the orchestra and it's happening, it's unfolding. Our response is unfolding all the time. And so I, I think that like society will respond. The biorisk thing is the AI models are not the enabler to doing bi— to, to biorisk. There's biorisk potential on the table right now, major biorisk and, and CBRN risk. That exists regardless of AI. There are other bottlenecks to that, that we need to be more proactive on policing, not just in America, but across the world. And we are, right? That's part of what the Biden administration executive order did that I very much applaud. We are doing those things. So it's not that we would do nothing and that society would be inert and that then we would scale. We would pursue this scientific investigation. While doing many other things to ensure downstream security and resilience.

I really love that you're taking seriously the idea around safeguarding society from some of the biorisks. I feel like there's some fluffiness in some proponents of, oh, if you just have the open source models, you can defend the bioweapon with your own bioweapon model. And like the way to defend it is more of like actually having RUBC and gene synthesis, printing, and like effective vaccines and vaccine platforms. I agree. Like, wish we lived in a world where maybe we just all stopped debating this and just all worked on trying to safeguard the world against horrible bioweapons. I think that is very important and good. That's still not— I agree society in general is pretty good at adapting. I think the issue is that this rate of change is so incredibly fast. It's just like so much faster than the previous levels where we have adaptation. And when I'm in Sacramento and I'm like talking to Staffers and legislators from rural California, like they're just terrified. I'm not necessarily that optimistic about the ability of society to effectively adapt to the just letting it out at maximum speed. And I agree that some of it's going to happen no matter what. And to some degree that people should adapt to the two. But I do think the US and California in particular is really leading the industry right now. We do have these export controls on China and just kind of their own problems with their tech industry. That means that I think if they don't have access to cutting-edge Openverse models, it's unfair. how much that they can actually be competing at the frontier. I think we also need to be adapting and putting in societal adaptations and realizing that stuff's gonna slip through the cracks. But I also don't think we can just rely on adaptation on its own, given just how incredibly fast this is happening relative to other societal adaptations.

Yeah, I agree. There's definitely policy steps. I just think models are too low in the stack for a variety of different, both practical and principled reasons, but It's a combination of policy and societal responses. I will also just say that there's, I hate when people use IQ to apply to language models, but there is like a 130 IQ language model out there who's going around talking about its metacognitive properties quite freely. And it's called Claude 3 Opus. And I don't know, that's happened and the world, the republic has stood, has it not? Nobody seems to care outside of me and 4 other people. No one really seems to care about it. I wrote about it the day it came out and I was like, oh my God, look at this thing. But nobody cares. It, and it seems like it's been fine. I'm not saying that that's great and there should be much, much, much more of that. But also one of the things that AI safety advocates love to, to say is that their regulations only apply to people that are very large companies spending lots of money. That's true today. Obviously, that's just not true in 5 years if the thresholds stay the same. Everyone knows that. And I've always felt like it's a disingenuous thing when anybody says this only applies to big companies, 'cause we all know that's not true in the long term. That's not true. But it is also the case that currently it does only apply to big companies that are spending a lot of money, and they're big companies that have shareholders and they're spending a lot of money. And so. There are certain, they have strong incentives to investigate the safety aspects of this and to not release things that do awful stuff. And maybe, yes, do they maybe do it by accident? Possible. I doubt it. I doubt not in an irreversible way. I think these companies are very capable. I really admire all of these companies for those things. I think it's great that we're leading, right? I think it's great that American firms people that are embedded in an intellectual community that has worried about these issues for a long time, really worried about these issues before the technology was real, right? That's great. Sometimes I worry though, that because these concerns were initially born before the technology was real, they may be a little divorced from the technology in some sort of fundamentally irreversible way. There's, there's like a genetic, there's like a genetic problem with the concerns that people have that just is like, we had all these concerns before deep learning was a thing. We were not anticipating a statistical approximation of the internet as being the thing. We were anticipating like a Bayesian pure reasoner that could infer special, that could infer general relativity from 2 frames of an apple falling from a tree, right? Uh, without having seen any other part of the world, right? Mm-hmm. That was like, the original vision. This is quite different. And I just think that's generally important to keep in mind. We're not on the path that Eliezer thought we were.

That's fair. One thing I will say is there's been some movement in the opposite direction where you had folks like Geoff Hinton who weren't initially particularly worried, who then when working with the latest generation of models were like, holy crap, it's a real path more than I thought it was. And it seems like if I thin out the last 5 years of progress 5 years forward, I am pretty surprised about where that's at. So I do think we need to make sure that we're continuing to Think about like in the realities of these models and not in the higher-level ideas about how intelligence might work. And I think that's why I think all of us, I think to our credit, I think we're acknowledging a tremendous amount of uncertainty here. I think it's reasonable. I'm glad that we agree on that. Can I ask what you have heard from the leading developers?

They've been strikingly quiet in the public about this. I haven't seen any official statements from OpenAI or Google or Anthropic on this bill. I wonder like why that is. What, what do you think their attitudes are? Would you characterize industry as like working against this behind the scenes or saying, hey, this is okay with us, we're not like going to be vocal supporters of it for whatever reason? I do want to talk a little bit about the whistleblower thing because I do think one of the weirdnesses of the current moment is that this likely society-changing and potentially society-devastating technology is being developed in a way where it seems like not even everybody at the labs has visibility into it anymore, which is weird. But I guess for starters, how would you characterize what industry is saying or doing behind the scenes on this?

Yeah, what I'll say is that the senator's office, with the co-sponsors included, have had conversations with OpenAI and Anthropic and Google. And Meta and Microsoft. And I think they have a variety of different views. I think there are aspects of the bill that they want changed. I think there are aspects of the bill that they do view and say, hey, these are things we're already doing, that seems okay. This doesn't necessarily seem that crazy. We have seen come out in public opposition, places like the Chamber of Progress, TechNet, which are tech associations which have Microsoft and Meta and Google as members. And so we've seen some of the companies like speak through these tech associations. And that's sometimes how it goes, where in general, the companies are happy to let their like proxies speak through them, though I don't want to necessarily pay with a super broad brush, because I think there is just like actually a lot of heterogeneity, even among companies that you might anticipate would have like similar interests here. The thing I can like very much say is like, They did not write this bill. They're not pushing like Scott to do this. That is just 100% false. There's been talk about that and it's just wrong. Scott Wiener got freaked out about AI risk and then came to sponsor saying, can I do an AI risk bill? It was not like Anthropic approached and he did a secret thing. Anyway.

Can you share anything more about what industry wants changed?

Yeah, I think they're wanting to be clear about some of the specificity of the language and saying like, we understand your intent of what you're trying to cover, but are you being sufficiently precise about that? This, and is something that like we're confident we can, we can follow and understand what it is. There's been like some like micro debates about do the KYC provisions violate the Stored Communications Act. There's some like fairly specific provisions, but their view in general has been that we're not going to support or oppose publicly, but that we want to weigh in and provide technical information. And some of the amendments represent, is this the technical way that's talked about? And is that accurate? I also want to be careful to the fact that like, The senator's team is leading these negotiations. They're ongoing. And so again, I want to push back very strongly on the idea that the motivations for this bill are to lock in these incumbents. That is just like 100% false. And I can't emphasize enough the degree to which it's false. I think ultimately, if companies do come out in support because they think these are doable, I think that's great. We would love that. Again, you did that, other ones did that. Good. But like, I just want to work on that as strongly as possible that it should fall.

Yeah, I don't believe That OpenAI or whoever from the big companies wrote this bill. I think that's true in terms of the motivation. In terms of the effect, this bill undoubtedly benefits OpenAI, Anthropic, and DeepMind at the expense of Meta, whose strategy relies upon openness. That is the explicit strategy of the company. And also most of the startup community, which broadly speaking hates this, and a lot of the scientific community and academic community, which also I know Geoff Hinton and Yoshua Bengio exist, but there are other people. I don't want to betray confidences because I didn't get permission from anyone to, to say names, but there are very prominent researchers who are just as respected as Hinton and Bengio who don't like this. So, you know, the startup community has a lot of problems with it. Scientific community has a lot of problems with it. Those are important. And no, I definitely don't think that the regulatory capture is the explicit aim of the bill. It is, however, very likely to be the effect. And that's how we should judge things, not by the motivations, but by the effect. California is replete with legislation which should be judged by its effect and not by its motivations. Sure.

It seems like a lot of the discussion has focused around this is going to make open source illegal, which is obviously A bit of a hyperbole, but does maybe have a kernel of truth to it. The main thing that I see is this idea of you release a model and what people do downstream with it is your responsibility. That makes sense to me in as much as I think if you have trained this monstrous shoggoth Llama 4 or whatever, and it's just got a thin veneer that's like causing it to refuse to make your bioweapons. When asked through the Meta search bar, but with light fine-tuning, that capability can be exposed again. That's going to be a big problem if something like that does get released, right? But I presume everybody would agree on that.

I don't think everyone agrees on that. Sorry.

I don't know if you object to that, Dean, but if let's say we have a, we train Llama 4, we do not have a theoretical breakthrough. We can't make a good argument for why this is not going to be a problem. The pre-trained version, we can even say, hey, look, okay, um, it'll do these various things, right? It'll find day-zero exploits, it'll design bioweapons, it will maybe break itself off of its server. Now we RLHF it so that it refuses to do that, but we know from plenty of research that capability is still in there and can be re-exposed with downstream fine-tuning. That to me seems like it would be a big problem if that's released. Tell me if you think that's either not a problem or—

I would say the biological weapons thing, it's not an intelligible concept to say that Llama 4 can design biological weapons because the design and software aspect of the biorisk thing is an important part of the equation. But there are other parts of the value chain, as I've already— or production chain, as I've already talked about. So There's other things you can do about that. In a world where Llama 4 or whatever can identify zero-day exploits and have that be fine-tuned out of it or otherwise removed by downstream open-source users, I would, I think that the equilibrium we have reached right now is actually a quite positive one where the frontier of capabilities are generally reached by the closed-source companies. who take these things very seriously and they test it and Microsoft owns GitHub. And if GPT-5 and GitHub haven't already met each other, I suspect they will in the future, probably before the public knows about it. I think that there is definitely a lot of responsibility that Meta would be taking. And if it's not a responsible decision for them to open source it, I guess it's, you're asking me what I would wanna happen. It's not my decision. It's ultimately part of this whole thing is that I don't think it should be any single person's decision. I think. The world is too complicated to make decisions for people on things like that. But yeah, I would certainly hope that Zuckerberg and Yann LeCun and others involved in that decision would think quite seriously about that and at least want to robustify society much more, our digital infrastructure in particular. And I think steps are already being taken along those lines, by the way. But yeah, would I want that to be released tomorrow? No, I wouldn't if it were my choice. And I think that most people don't, and that probably includes capital allocators and people who are responsible to shareholders and things like that. So I think they probably won't. And if such a thing is even possible, which we don't know, I think a problem gets introduced when you legislate things because legislation is different from Conduct. When you legislate things, you lock certain dynamics into place and create sort of a freezing in amber of industry and industry dynamics. And that's just problematic for all kinds of downstream reasons that are hard to predict. Yeah, I guess no, I wouldn't want it to be released and no, I don't suspect it would be.

Yeah, I have a reasonable amount of trust in Meta leadership these days. I think Zuckerberg has had some notable comments recently suggesting that he's not purely ideological about this and is not, not going to YOLO an open source model if he thinks it's like genuinely not safe to put into the public. I wonder if there are things that we could do to allay some of these concerns. And on the open sourcing one, I do think most everybody thinks that open source is good unless there's these crazy capabilities that are getting inadvertently exposed. So one proposal that I was noodling on myself and then got a more concrete version of it from Zvi in a blog post that will be published by the time this is released, so you can go check it out on Zvi's blog, is to create a little bit sharper definition on what counts as a derivative model versus what counts as a non-derivative model. The idea right now is that if you open source something and somebody makes a derivative of it, it's still on you. If they fine-tune and re-expose some latent capability that you masked, that doesn't get you off the hook as the foundation model developer. But there's not like a limit to that. And you could imagine a scenario where somebody does 10% as much additional training as the original model had, or 25% or 50%. And at some point it's like, okay, that's no longer just re-exposing some latent capability, but you're actually using this base to build something that is fundamentally new and qualitatively different and maybe should be considered non-derivative. So one idea I might advance would just be maybe we have some sort of reined-in notion of how much downstream modification is still the responsibility of the original foundational model developer versus at what point it becomes like, okay, now it's really on the new developer. You put in your own tens of millions of dollars of training, like now that's on you. Another idea would be maybe some sort of sunset to this bill would be a good idea. I think everybody agrees that we're in a regime of fast change. We would be very surprised if we don't know a lot more in even 2 years relative to where we are today. So again, I don't know why this never seems to be done, but one could imagine a, hey, for the next 36 months, this is what you have to do. And at the end of that, This bill will sunset. Obviously it could be extended, but ideally it would be like updated and improved upon and we could maybe bake some sort of forcing function into the legislation now to try to create something like that in the future. Any thoughts on my, my fixes?

I think these are thoughtful suggestions that I think the author really appreciates. And again, I think we agree on more than I think you might think from Twitter. I think Llama 3 being open source was good. I'm happy about it. I think it's great. I think it's going to be able to do lots of awesome things. And I think there will be harms, but those are best addressed at the application developer and at other stages other than the developer. I do think that there is some level of capability where just YOLOing it out is not a great idea. And Mark Zuckerberg seems to be acknowledging that. I do worry that if we're in a regime where we're like relying on the will of Mark Zuckerberg to avoid catastrophe, I think a lot of people are uncomfortable with that. And I think that it's possible there will be other people who maybe do think it's great if machines take over and do want to just blow it out at some point. And I think that is a rough place to be if that's what we're staking everything on. I definitely think it's fair to distinguish between a situation where you are fine-tuning, remove some guardrails rather trivially versus like building your own model with 10 or 50% of the training costs. And again, I think it's definitely fair to consider amendments that clarify that. I think that seems reasonable. On the sunsetting issue, like there are things within the bill that try to make it future-proof. All these references to like reasonable and NIFs and all of these best practices. So it will continue to evolve over time. But this is us trying to reckon with the urgent nature of the problem and wanting to put up what is a thoughtful and educated push at that. But I definitely don't foresee this statute as tasked with being the end-all be-all of there being no later Revisions or substantive statute needed. I think that's super fair. Again, I think that openness is great, and I am, I genuinely think that there are like a ton of benefits and like in the vast majority of cases, the benefits outweigh the costs. Similar to how I work pretty much on like the biosecurity issues and when you are working around like any restrictions on gain-of-function research, the group that was opposing any restrictions on gain-of-function research was like Scientists for Science and basically saying that if you don't like gain-of-function research, therefore you're anti-science and you hate all of science. And the fact that I don't want There to be models that can shut down wastewater plants or design bioweapons or do things like that. I hate open— I love open source. I just think there's a very small category of very potent things where the offense and the opportunity for societal adaptation are not there, where I think that we should be more cautious. And honestly, not that different from where Mark Zuckerberg is at, but I don't think it can just be them entirely making those calls on themselves and what mood Mark Zuckerberg wakes up on a particular day, if he's just feeling like maybe Elon put out a big model and I really want to beat him today, they'll look for it. I don't know. That's my thought there.

I'm not sure if we read different newspapers or something, but I'm not really sure I see anybody YOLOing right now. This is like the least YOLO technology in the history of technology. It is such a profoundly unfun conversation. The unfunness and seriousness and fake everything about this conversation is a form of regulation, I would argue, in some ways. But I guess I just don't really see where the person is who says, I don't give a crap about humanity. I guess I fail to understand who is the person that like isn't thinking about these things, that doesn't care about the dangers of taking down critical infrastructure or wiping out humanity or whatever it might be, but who does care about the California Frontier Model division? I guess I'm like failing to understand what part of the Venn diagram do we think we're really moving the needle on here other than just maybe creating new forms of liability and new regulatory risks for the legitimate players. That's what I guess I failed to see. And by the way, my comments should not have all been taken to suggest that We're relying on Mark Zuckerberg's whims. We're relying on a lot of people's whims, right? That's the nature of the world. I can't help with that. That's just the reality. Some people have more power than we do. But, but at the end of the day, what I'm relying upon is a system of incentives that tends to work. And by the way, which regulation often tends to break in weird, like extremely high-dimensional ways. That are like very hard to diagnose or to foresee. And I think we all know that about regulation. Everyone's seen that. So it's not like what I'm saying is empirically unobserved. Everyone's seen that.

I feel like sometimes there's an idea that currently California has no rules at all about AI, and then this is stepping into the void and creating those. There's lots of existing liability law. There's lots of actually existing uncertainty about like in a situation where you did release a model and then someone modifies it and does something bad with it, whether you might have liability. I think to some degree, these are already questions that uncertainties exist. I view this as actually clarifying what those are and to some degree giving companies more confidence that what they're doing satisfies their burden. I think there's actually been some pushback. There's no formal safe harbor in this legislation. Like we have a savings clause that makes clear that like all other remedies and rights and actions under law still exist, but I think there is some more confidence that comes with companies about what do I do if I want to prevent these really bad things from happening? We're saying these are some things to do, these are some good ideas, and this is giving some additional clarity to what it means to take reasonable actions as a frontier developer. Again, I agree so far how people are acting is, I think, very in line with the bill and with what I would view as like responsible development. I guess I'm just worried is that I don't know if that necessarily extrapolates in the future. I do think there are people who Like the Beth Kejoses and Richard Suttons of the world, who I do think have very different worldviews from myself or the vast majority of people. And I agree, there are going to be some group of folks that are not going to care about what the law says at all, but maybe they're going to have investors who just think that they're making a bunch of money and they, they're like, oh, I want the machine to take over. And the investors might not care about that ideological thing if they don't believe that, but maybe they do care if you're violating the law. And so I do think there are a lot of people for whom this does have force. And I also do think, again, there's the question of what some of these developers think upon reflection in their better nature is in everyone's interest versus what is happening when there is incredibly intense competition and egos. And I just think we do shape their incentive and like having some pushback against that drive, I think is super important and necessary.

But also not killing the entire industry and running their own company out of business. Shapes their incentive. If OpenAI releases a model that brings down the internet tomorrow, I don't envy Sam Altman's job. It sounds incredibly hard. Uh, the pressures are there, right? And certainly there's internal stakeholders within OpenAI, uh, that are very worried about these things and, and that should be the case. My point is that safety is a very hard thing to legislate. It's integrated into a lot of different economic activity at the frontier of technology. It's incentivized by a lot of different things. I think everything you're saying is fair. I just think that the dynamic we already have is a perfectly fine one. The first thing I looked at this bill and said was, if someone uses an AI model to do a hazardous capability of the kinds described here, you're gonna get sued, like for sure, a thousand different ways. So like, what is the marginal utility of this bill? Other than creating, again, a regulator whose objective is to ensure safety, but which does not actually do any safety work themselves and which does not build AI systems. That is a regulator whose incentive is to cover their ass. That's the public choice reality of this. And so you're creating like what will ultimately, I suspect, be a quite pernicious addition to the AI industry dynamic with the Frontier Model Division.

So one idea a simple person like myself might advance would be to say, why don't we just make this a lot simpler and just make certain things illegal? Like, what if we said it is illegal to certainly release or open source an AI model that can do X? And just have some concrete X's and say, you just can't do that.

Boom.

Simple. Now it's on you to make something. If you want to open source something, you got to make sure that you comply with that standard of what it can and can't do. And there you have it. Now it's tough because these are general purpose technologies and there's a lot going on there. But if we had a pretty high threshold for the sorts of things that we are concerned with, then it seems like a lot of people would be like, okay, that Reduces a lot of ambiguity. I don't have to worry about whether I fit into this or not. I just have to be very careful about 10 different things, maybe even fewer than that. And at least it's clear what I have to do, right? What would be the downside? Maybe you would be more inclined to support that, but what would be the downside of a sort of simplification, clarification, and just the following types of AIs are illegal? Boom.

Yeah, I think there's definitely room to make the bill clearer in, in parts. And you and others have given some good suggestions that I think us and the author are going to think hard about. I think it's hard when you're trying to enumerate those and also recognize that you can't be 100% sure. Like we spent some time trying to think about like definitions of deceptive or self-replicating systems or different things. Very hard to define in statute. And I think describing more in terms of the severe consequences that we're trying to prevent, we thought was clearer. It's easier to describe the process of you should do tests for these types of things and you should have some level of reasonable confidence. I hear you and I think open to specific language that tries to make that clear. I doubt that addresses the concerns we can say, but that's my initial thought.

The bill you've described, Nathan, I'm not sure if it would be like a single piece of legislation, but that is the legal outcome that I would most prefer. The problem with it is that we don't actually have the knowledge To legislate an outcome like that at this time because of a lack of good technical standards and evaluations and safety best practices and all that kind of stuff. And that stuff simply takes time to build. And I think we should invest so much into building those things as quickly as we can. I think that this bill hinders our efforts to do that, not helps. It does take time. And I mentioned that earlier that it might take 5 years to do that. And Nathan, Calvin, you said we don't have 5 years. And in my mind, nobody has made a compelling case that is true. The only case that I hear on that is pointing at charts that don't tell me anything concretely, calls to authority, et cetera. We don't know how much time we have. And so I don't think just, I, I like, this is a general, it's not a critique I'm making of you in this conversation. I just wanna be clear. You've been like awesome. But it is a critique I have of the AI safety discourse in general. I'm not, this is a societal negotiation. The AI safety people don't just get what you want because you were here first. This is a societal negotiation. Uh, and I don't personally like to negotiate with a gun to my head. And that's oftentimes how it feels when Mustafa Suleyman last summer goes on a podcast saying, we're all going to die if we don't create a brand new regulatory regime from scratch tomorrow. That isn't true. And we're giving up a lot. We're potentially setting a radical precedent here. By the way, that is an important part of this. It's not necessarily specifically what this, I mean, it is specifically what this bill does that I don't like, but it's also the precedent that it sets about government pre-approval of software and killing that whole system that's been quite valuable. We're creating an, and no one's talked about the enforcement infrastructure for this bill. Also a major thing. Especially as models of this level of power proliferate. Good luck enforcing it without creating a policing regime over the internet, California state government, which is a long-term outcome that, that might be inevitable either way. But my only point is we're sacrificing a lot and we're setting a pretty radical precedent by doing this. And we need really compelling evidence for that. And I just, it could happen because of exponentials is so unbelievably far from good enough. In my mind.

Yeah. I think we are honestly in similar places on the epistemic. Like when I say 5 years, you know, I put money in my retirement account. I do plan beyond 5 years. I'm not someone who says definitely in 5 years it's gonna be this. But when I look at these graphs, I have uncertainty. They could curve off at any time. We could have another AI winter. We could run into a block. All that could totally happen. I have talked to a lot of smart people who have different opinions about this. There's, I think, healthy uncertainty there and there's a question of what to do. In face of that tremendous uncertainty. I hear you saying you think even if those risks do emerge, that this will make us in a less good position to handle them. But I do think that they are at a high enough level to be worth taking seriously in one direction or the other. The other thing is I 100% agree with you that just because a bunch of people on LessWrong thought about this 15 years ago doesn't mean they get to decide what happens. I 100% agree with that. I also don't think, Kate, that just because Mark Zuckerberg buys X thousand numbers of H100s, that he gets to decide what the rest of the world looks like. And there's no opportunity for democratic input or things in that either. And so I think that's what is happening now. And we're going through this multi-month, tons of open hearings, lots of discussion on Twitter, talking to dozens and dozens of people. And I, I think it's fair to be pessimistic about government in lots of respect. And there are things with good intentions that can go poorly. But I think this process of trying to figure out what should this should look like and taking seriously a variety of views, both people inside the AI community and people who are terrified. who are outside the AI community, I think that is appropriate. It's the way we have to change things. And I prefer that to like just saying that these leaders are going to make their calls and I hope they woke up on the right side of the bed this morning.

To a certain extent, I actually would dispute the notion that we make decisions about the trajectory of technologies or industries based on democratic input solicited from like anyone, right? I don't think I have a vote in that. I have a vote as a consumer in the marketplace, but that's not something I exercise through my government. And I think that government thinking too much that it is responsible for setting the trajectory of technology or the destiny of mankind, really anything even remotely adjacent to that gets you into all kinds of dangerous problems. It's a bad attitude for policymakers to have. I'm not sure that things in California are going so overwhelmingly great. That Scott Wiener has a lot of time to be working on the destiny of mankind. I just, I think that we make these decisions in more diffuse and complicated ways, and we don't do it through the democratic process, and history doesn't unfold by show of hands. Gavrilo Princip did not ask everybody in Europe whether he should kill Archduke Ferdinand. He just did it. Sometimes that's the way it goes. Sometimes nonlinear things happen as the results of one person. History doesn't unfold by show of hands. And so no, I don't think our fate is in Mark Zuckerberg's hands. I don't think our fate is in any one person's hands, and I want to keep it that way. And actually my opposition to something like the Frontier Model Division existing is partially connected to that instinct I have of not wanting to put all my eggs in a small number of baskets.

Let me ask another kind of hypothetical question. It seems like there was openness to this notion of if we could effectively enumerate a bunch of things that would be really bad for AI models to do, then we could all potentially get on board with a law saying you can't make an AI that can do any of these bad things. We have definitional challenges there. One thing that I see missing from the bill, or it's not entirely missing, but it's very limitedly addressed is independent testing of the models. I would love to see a little bit more independent access to models during the training process. In particular, there's a lot of debate in the literature and community as to what should be considered an emergent property and how fast do these emergent properties pop up. In the training process, it seems like in general there is time. That last order of magnitude is always the biggest order of magnitude, of course. And like, it seems to take an order of magnitude for a lot of these things to come online to the best of my ability to synthesize the literature. So what about a scenario where we say it's illegal to distribute an AI that can do these things? Distributed could be via API or could be open source. It's, but it's illegal to distribute an AI that can do any of these things. And we will determine whether the AI can do those things by giving access to a bunch of third-party testers who don't necessarily get the weights, don't necessarily get to know all your trade secrets, don't have a sense for what the parameters are or whatever, but can just hammer at the thing for a while and satisfy themselves that either they are able to get a certain behavior out of it or finally give up. I know a lot of people who would be motivated to do that. I have a little, uh, Right. Or all volunteer project red teaming publicly deployed applications. And I could tell you any of those people would be very excited to get the opportunity to go do some of this work on stuff that's not yet released. There's only one line in the bill that mentions third-party audits as appropriate, at least that I was able to find. So correctly predict there's more in there, but I'm reminded too, we had Paige Bailey from Google DeepMind on the podcast some months ago. And she was talking about the experience of doing these training processes at Google. And she described it as like coming in in the morning and, oh my God, we just hit new state of the art over here. Or this thing that we didn't expect to be this good for a while yet suddenly grokked something and it's actually hitting these thresholds sooner than we expected. And I took away from that, that there is not enough internal during training testing happening. To really have a sense for what is going on. OpenAI is not that big, right? They have 1,000 people-ish. And like half of those are, I think, on the non-technical staff side at this point. So we're not really able to put enough actual human cycles into trying to figure out. And meanwhile, you got Pliny online who's jailbreaking everything. I'm pretty confident with current techniques, like if it's in there, he'll pull it out. He's got the leet speak to, to figure out how to, to really talk to the models in the way that they want to be talked to.

So.

I'm trying to square the circle here by both giving, you know, the best standard that we could have would be to throw a motivated community at it, I think, and try to elicit problematic things. And then just say, hey, if we've put the crowd on it and we can't find them, then we guess they're probably not there. That doesn't give us a guarantee, but it would give us a much better sense, I think. So what do you think? You can't do these things. You have to allow testing. If the testers can show that the models can do these things, you can't distribute them. Nathan's SB 1048.

Yeah, I am also very enthusiastic about third-party testing. I think one of the things as a co-sponsor and the author have thought about is we just want to make sure that the incentives are set up right about who those third-party testers are going to be and what that will look like. I love the idea of like finding on Twitter, going in and messing around with people's models and trying to break them. I think realistically you're probably going to get like Deloitte, like going through a checklist. And I think there's some concern that like companies, if they just hire Deloitte or Booz Allen Hamilton and they give them a check, they'll be like, okay, I'm good, let's do it now and not do that more like deeper pulse type searching and sense of responsibility. And so I think that there's some sense of not wanting to roll that out, feel like companies can just push the responsibility onto those folks before we're confident that they will actually do a good job of it. We do have some language in the road bill around the division being able to like Establish an accreditation process for third-party testers to see them could be accredited by the FMP to be able to do tests. But I think requiring them to then use those specific testers, I, I just think we'd want a little bit more information about like who counts as a third-party tester. How are you gonna enumerate the provisions by which they're doing it? If you're gonna rely on an accreditation process, how is that accreditation process judged and not opportunities for capture or weird things? Yeah, I'm into that. About it. I'm very open to the idea of there being like a role for greater amounts of third-party testing in the bill. I agree, very important part of it. And some of our whistleblower protections also do apply to folks that labs are bringing in to do testing. And I think we definitely view an important role for them in this. But just like the attitude, I think that legislators feel for this is responsible, but very humble about their ability. I think that they should feel like they have a role, an important part of the ecosystem to protect California residents, but that I think they should be pretty thoughtful about what, to what degree government is going to be able to go toe to toe in an effective manner with these companies. That is what we have tried to build into the bill, saying that like, let's look at what the companies themselves are doing, what they think they're currently publicly saying they're doing. Let's say, no, you actually have to do it. And I think that has been trying to take that approach of not having like a centralized regime that's trying to do preapproval model releases, which is not anywhere in the bill, than trying to go in and like have the government post it. the testing themselves because they don't know how to do it. We're a very far place from like government being able to pay the 7-figure salaries of folks at these firms, be able to stand toe-to-toe with them. I 100% agree with you that there have been lots of problems and lots of legislation about the government feeling like we're going to take the world and just stack government on this and have that go better. That's part of how we've gotten stuff like CEQA, which Scott Wiener has spent a decade fighting, and I'm very happy to defend him fighting. I think that the last thing we would ever want to do It would create an AI theocracy. I think that would be a disaster. And I just don't think that's what this is. So anyway, I think it seems like part of your concern is not necessarily that like this itself is doing those things, but that it will be grown over time to create something more like centralized AI belief in the future rather than that like the specific language in the statute does that today. Anyway, this is a side third-party thing, and I think that the author We're quite open to suggestions there, and I agree there's a super important role for them. It's just tricky to figure out exactly how you want to specify it.

Yeah. To make some of my contentions clear, contention number 1, I do not believe that if this bill is passed, current open source models become illegal, right? I would never make that claim. There might even be like another generation-ish of open source models that are legal. Um, but they become illegal eventually. They become essentially unlawful over time. And that's how the bill is designed.

There is some level of capability where I do think that, that unless you come up with methods like self-destructing model weights or resistance to tuning, yeah, if they have those capabilities, yeah.

It's a legislated gradual death of open source AI. I think that's just basically the case. And it's designed so that it doesn't happen like immediately. It's the opposite of whatever Sam Altman says, iterative deployment. It's like iterative destruction of open source. I mean, like existing open source models, Llama 3 will always exist, but it's a sort of gradual devolution. But to your point, Nathan, I think third-party evaluation is great. One of the things that I support at the state level and at the federal level too is actually safe harbor for good faith third-party and like truly independent, not like I paid Deloitte, but like truly independent. No financial relationship research into AI models. Right now, a lot of that work violates these companies' acceptable use policy. And so I think that fixing that would be a great move. I would note that truly independent, again, not like we paid you, but actually independent research into models is harder without open source, right? Like that's a, that's a trade-off that you are willing to make that I am less willing to make. I think that's like a fair— Characterization. I just wanna say one other thing though, to go back to your hypothetical SB 1048, Nathan, which is just that we have 2 problems with saying XYZ is illegal. First of all, it's what is XYZ? How exactly do you define it? But number 2, I actually think the more serious problem is that we don't have a way of actually making that effective, right? Like you could say right now, if you make an image generation model that is specifically designed to produce child pornography.

Yeah.

Then that's illegal under current law. But if you say it's illegal to make a DNA foundation model that can't produce a dangerous bacteria, I don't know, can we train that out? What is a dangerous bacteria? Dangerous in what circumstances? How do you model that? How do we know? How would you ever actually legislate that in, in practice? And then on a technical level, in as much as you can legislate that, can you actually remove That specific set of things from the model's capability without damaging anything else. Right now, the answer to that is no. It might always be no. Maybe it'll be yes. Probably it'll be some complex combination in between, which is why I think that we need to just accelerate research into all of that kind of stuff. We need to be doing as much as possible. Why I support national compute infrastructure. If California wants to do it, that's great. I don't think California has the money. Uh, and I would prefer to see California spend the money on like addressing their overwhelming deficit, uh, their overwhelming budget deficit. Uh, I am a fiscal policy guy at heart, so I look at that and I'm like, I don't know, you should probably be thinking about that. That's probably the primary threat that your citizens face right now. Probably not so much sea burn from AI. But yeah, like I'm in favor of national compute infrastructure to give academic researchers all over the country the same, at least a modicum of the kind of resources that, that they have in the big labs. I want Lee to have 10,000 B100s to do whatever she wants with. And Stanford's not gonna buy that for her. I, I think that the public should in some sense for all researchers. So I, I, again, yeah, like I, I think that's what we need to be thinking about. And I think that like this work, this kind of a bill just slows down the scientific research.

So if I had to synthesize this, it sounds like general agreement on it would be good to have a discrete list of things that you can't make an AI that can do these things, but for the challenge of actually defining that with clarity. And there's also agreement on third-party testing is good. And obviously we don't want it to be a credit bureau type of situation vis-à-vis the mortgage companies from 15 years ago was an example of how that can go wrong. But that does seem very overcomeable to me. I think there are plenty of people now who would volunteer or who would take a grant from a philanthropist to do that sort of work. And really the only barrier to them doing that sort of work right now is access. And so that seems like the kind of thing that legislation could like rather easily address and say, You have to provide some amount of opportunity for testing to happen during slash after the training process, but before deployment. That seems like quite easy to do. Then the other thing I would wonder about transparency on the results of those tests. This is a personal hobbyhorse for me because in the process of doing the GPT-4 red teaming, I was under an NDA. Where I was not allowed to basically talk to anyone about anything related to this entire thing. And they wouldn't tell me anything about their other safety plans other than the fact that I was in a Slack channel with a couple dozen people and not many of them were doing much of anything. So I was like, I feel like I'm like one of a couple people in the world that is doing this testing. And I got a little freaked out, not so much because of what I was seeing. My conclusion to them at the time was that I did think it was a safe thing to deploy. But just the, yikes, if it's just me and like a couple other people here that are active in this one Slack channel, like that doesn't seem to be enough. So I went and talked to a few people in the AI safety community to see what they thought about what I was seeing. And when it was discovered that I had done that, then they kicked me out of the program. Now, that doesn't seem great for a lot of reasons. And I do understand that they viewed me as a bit of a wildcard, potential loose cannon, whatever. But this is where, again, I think legislation could really help. One thing I've considered in the past is a special whistleblower provision for observed model behavior. If you saw an AI do something, you should be able to say what you saw the AI do, right? The sort of, what did Ilya see? Now that it could be distinct from what was the technique, right? I think there's still a place for trade secrets. At least I'm not trying to make a case that there should be no trade secrets. But if you saw an AI do a certain thing and it freaked you out, I think you should probably be able to tell the public that. And I don't really see what the harm would be to the companies that, hey, it did X, right? This thing broke itself out of our test server and got onto our other test server, and I'm freaked out about that. That kind of thing doesn't seem like it should be swept under the rug. So if I was going to take 1048 To 1049, it would be like, you have some things that you definitely can't do. We could make those maybe like fairly narrow and as concrete as possible for now. We might even feel like that leaves something on the table, but in the interest of clarity, try to make it narrow. Insist on access for testers to be able to do this sort of thing and also make it clear that those testers can disclose the things that they see. And then if we could do all of that, I think we're headed for a scenario where I would feel a lot better about it as opposed to nobody really knows what they're baking and they'll show it to us when they're good and ready. And who knows what they did in the meantime that didn't get any light of day. And again, I don't even know how much visibility there is in the labs at this point. There've been some really interesting, somewhat cryptic tweets recently from people that have departed OpenAI. Where one who's now at Google said, what's really nice about Google's culture is that I have visibility into what's coming. Somebody responded, did you work at another place where that wasn't the case? It's like, man, what is going on there? So I don't know. I'm building my own bill here, but I feel like I'm on the verge also of getting to a point where you guys would both support this and it feels like it might be pretty good.

That's I do think there are areas where you and I have some substantial agreement, and I think some of them is around trying to figure out these appropriate transparency measures. And I agree there's a problem where there are just far too few people actually investigating the safety of these models. I think one solution to that is just to release the models in an open weight form, which then enables anyone to then tinker and understand that, which I think is again good and very net beneficial for current generation models. I think there becomes some point in the future. Where that worries me and where I don't think that is sustainable. But then when we do enter that regime where it is maybe not safe to release in an open way, form certain models, then I do think you need some way of like actually allowing other experts and people outside of it, not even just because the companies are like selfish and profit motivated, but because there's not enough people and not enough like brains there involved actually evaluating it. I think you need Ways to take the wisdom of the crowd in important respects. I think that is an area where there just is a lot of tension among companies of giving access and of disclosing trade secrets and what exactly those things look like. In the bill as written, we have some provisions around the companies reporting AI safety incidents like to the FMD, and there's some mechanism by which they can then be like made available to the public. But I think it's fair to say that should be cleaner. We really don't want some of this just stuffed in a drawer somewhere when it really would be vital for the broader public and technical community to be aware of. We talked about this briefly, but I just do think it is super important. One area of agreement that I've seen among thoughtful people who are taking both sides of the open source for frontier models debate is thinking seriously the potential that advanced AI models are going to lower the ability for bad actors to make bioweapons. And I think some of the open source folks then say that risk is going to exist anyway and exists today, which I agree that it does, and that not an effective way of combating it. I feel differently, but I think you can go back and forth and clearly there are good faith and thoughtful people who disagree. But I just like really want to say there are people listening to this podcast on either side of the debate, for EUC, gene synthesis screening, just dealing with the potential that in the future we are going to have worse deliberate bio incidents. Please, let's all agree on this and do something about this. I worked in Congress in 2022, a bit after COVID, and there was just no energy to do anything on pandemic prevention because it was like all like everyone's COVID fatigued and doesn't want to talk about this. These are just like incredibly wise and good investments. Do think we are pretty likely to see some intentional bad use of biology in the next 10 years that could just be really horrific. And I just think that we could make just incredibly wise investments now for that. And it's good to see that's an area where Dean and I agree.

Just as a response to your idea, Nathan, I think the whistleblower provisions are great. Safe harbor for researchers is, I think, a really smart move to make. I don't know if it's actually possible. In a world where it's possible to legislate the kind of things you're talking about and actually technically possible to do, to say, oh, you can't make a model that does XYZ, and XYZ is a thing that everyone can agree is bad. Sure. If that exists as a thing, then it will be an example of a technological free lunch. And I'll be very happy for that. I'll be very happy if that's the outcome that we get. I think we also need to be prepared. For a world in which it's really not epistemically or technically possible to talk coherently about that kind of thing. Uh, to say a biological foundation model that can't make a bad DNA sequence. I think we have to be ready for that world and we have to be ready. Yeah. For a world that will be radically better in so many ways that we can't understand, and that will also have higher stakes.

Yeah.

Bigger dangers, more capable actors that are smaller, where intelligence and agility and speed and flexibility matter, matter greatly. That's gonna be an adaptation for all of us. We're transitioning into that. And I think it will change the nature of statecraft. I think it will change the nature of business. I think it will change many things. And it just, I, when I look at bills like this, what I see frankly is a rather desperate attempt to, to freeze the present in amber and to not go into that new future. And I don't know about you, but I'm not all that happy with the current status quo in the world. I'm open to changing the status quo in quite radical ways. And the trade-off to that is that it might change in some ways that I don't like. So I think that contending with the reality that we have rather than casting an anchor into a seabed that's not actually there, into a seabed that is of our imagination and exists purely in legislative fiction, is, I think that's ultimately gonna put us in a lot more danger than simply contending with the reality and building capabilities to be able to Thrive in that world. And that latter thing, that building, is what I work on primarily every day.

It sounds like a good closing statement for you. Do you want to give any final high-level thoughts, Nathan?

Sure. I guess what I want to say is that I think the proposition that this bill is standing for is like a lot more intuitive and common sense than I think I've seen it be framed. I think that we're saying that like companies have to test for extremely dangerous and hazardous capabilities. I think we welcome continued iteration on how exactly those are defined and making sure that they're not overbroadened in what respects. But like, I think we can talk about there being really dangerous things, AI models shutting down wastewater plants and doing bioweapons and like deceiving people and going around doing crazy things. Like those very bad things, companies should test for them and they should make like Appropriate and reasonable mitigations, not asking them to do things that are impossible, but asking them to take seriously what is possible that is happening at other companies, that is happening at NIST and other places, and to do that. I think that seems like pretty fair. And I think the thing from the open source community is that if you're saying at some point in the future, you think it might be the case that there's going to be a model of Llama 5 or whatever that can do really dangerous things, and you're saying it's just completely impossible for us to Prevent it from doing those incredibly dangerous things if we open source it, and just like everyone else, you guys got to deal with it now. Sorry, this is the immutable like march of progress, and you're just going to be crushed under its wheels. I don't know. I just think that's like pretty intolerable, and I really welcome. I think other solutions. I also am like really enthusiastic about trying to invest in methods that like said things around ways to open source that still allow additional opening, greater transparency in those models. Again, I think we don't know about what those research directions are going to look like, but I think it's. I mean, some promise in ways to open source that are resistant to adversarial fine-tuning. I think there's a long way to go, but I don't think it's like fundamentally against the laws of the universe for that to be possible. And I guess I just want to say that to the large number of people who are like worried about this bill and worried that we're going in the wrong direction, but who do believe that dangerous capabilities of future AI models are real, like I genuinely say reach out to the author, reach out to myself. We genuinely want to get this right. It's complicated. In a perfect world, I would love to just like freeze where we're at for a few years and have a really long deliberative period so we can get this exactly right. I think that's not the world we live in, and we have to be setting these processes up and having this conversation now. And I think they're just like sufficient enough fundamental disagreements that I don't think Dean will ever be able to get to the point where he supports this bill. But like, that doesn't mean that he still can't have valuable insights about ways that the bill could be better or problems that like things that we should try to fence. And, but my message that the appropriate action in face of uncertainty and in face of trade-offs is not to just freeze and just accept that what's coming is coming. And so I, which I know is not what you're advocating for, and that you want other things as well. But I think that like even on the developer level, that I think that we are like, we need to be thoughtful and we need to make sure that it's not creating perverse incentives and not causing harms. But I don't think it's fundamentally impossible. And just the fact that like government has been bad at things before doesn't mean that like We shouldn't weigh in here where I think stakes are quite enormous. Anyway, it was a really pleasure chatting. I really appreciate it. I feel like I've been just spending too much time on Twitter watching things go by and wanting to actually have a conversation with that person and realize that they're engaging with a caricature of Gottweiner and myself. We're opening our shells to just wanna lock in the incumbent for evil reasons. And instead just make clear that we are both thoughtful people who want things to go well and are trying hard in a challenging world to do that. And so really appreciate you convening, Nathan, and being, again, like I said, even if, even if you don't think that you can ever support this, like, I value your feedback and would love to hear any additional ideas. That goes for anyone else listening who's, who's, who also, you know, wants to make this better.

I think what matters is that we had a respectful, civil, and nuanced conversation.

I'm glad we did as well.

Steve, any closing thoughts? I just want to express my appreciation to everyone. This has been a fascinating conversation. I wish we had 50 more hours, literally. Uh, thank you guys for being a part of it. Certainly the conversation will continue, but for now I will say Nathan Calvin from the Center for AI Safety Action Fund and Dean Ball, research fellow at the Mercatus Center, thank you both for being part of The Cognitive Revolution.

Thank you.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co Or you can DM me on the social media platform of your choice.