Governing Frontier AI, with CA Senator Scott Wiener, Author of SB 1047

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. Today I'm speaking with California State Senator Scott Wiener about SB 1047, his proposed legislation to establish safety testing and risk mitigation requirements for advanced AI models, which the bill defines, as does the Biden executive order, as models trained with 10^26 FLOPS of compute in 2024 and other models with similar capabilities, even if they require less training compute to get there in the future. It's an admittedly complicated definition, the ambiguity of which reflects the general uncertainty about how this technology will evolve going forward. In our conversation today, Senator Wiener begins by sharing his enthusiasm for the potential of AI, as well as the story and motivation behind SB 1047. He explains how he's attempting to take a light-touch approach, not to eliminate, but at least to minimise risk without harming prospects for continued AI innovation. He addresses some of the misconceptions he's seen in the online discussion surrounding the bill, and he responds to various good faith objections that have been raised. For a more thorough analysis, I definitely encourage you to listen back to our previous episode where Nathan Calvin from the Center for AI Safety Action Fund and Dean W. Ball from the Mercatus Center joined Steve Newman and me to discuss the potential benefits as well as the drawbacks of this bill from a generally pro-technology point of view. I think that episode holds up very well. In light of today's conversation. So where do I come out on SB 1047? Personally, I often describe myself as an adoption accelerationist, hyperscaling pauser, which means that, like Senator Wiener, I really want to see us get the incredible value of things like primary care AI doctors. And by the way, I'll soon have an episode on Google's new Med Gemini model, which shows just how close we are to that exciting reality. And yet, at the same time, I do believe that we are playing with a new kind of fire in AI, and that it's critical that we exercise caution on our way to superhuman AI strategists and scientists. SB 1047 is very much in that spirit, and for that reason, I am inclined to support it. While it's true that the small number of companies pushing the frontiers today seem to be proceeding responsibly without being forced to, and while I, as a longtime libertarian, absolutely recognize that a new government agency could become a bureaucratic nightmare, It seems to me that the stakes associated with frontier AI development are high enough and the competition is fierce enough that this bill constitutes a prudent step. Of course, there's more that government could and perhaps should do with respect to AI. And I did ask a number of questions about other possible rules meant to mitigate, for example, arms race dynamics between leading developers. But in the end, I came away from this conversation quite sympathetic to Senator Wiener's point that you can't do everything in one bill. And that the focus of this bill is on ensuring that serious testing is done and safety mitigations are applied before models are released. As you'll hear, one change that I would still love to see, and which would turn me from a somewhat cautious to a very enthusiastic supporter of the bill, is stronger support for independent third-party testing. I believe it's critical that independent experts who are motivated solely to find the truth have the opportunity to thoroughly test frontier models during the training process and to be able to report their results directly to the government without fear of loss of access to the models or other retaliation from the developers. This is a tricky balance to strike with open questions, including who should be trusted to test, what sort of access should they have, and what remedy should the labs have if testers begin to behave in bad faith? But as challenging as those questions are, evidence is mounting that a government requirement for third-party testing is needed if we are to achieve Senator Wiener's stated goal for this bill. For one thing, my experience on the GPT-4 red team, and if you haven't already heard it, I definitely recommend my Did I Get Sam Altman Fired episode for the full story of that experience, shows just how precarious an independent tester's position can be vis-à-vis the frontier developers. And that's assuming one can get access in the first place. My background conversations with the leaders of several different AI safety review organisations, suggest that access today ranges from nonexistent to extremely limited. And recent media reports also indicate that even the UK AI Safety Institute has not been able to get advanced access to frontier models. That's despite voluntary commitments from the labs at last year's AI Safety Summit. That, I believe, needs to change, and an amended SB 1047 would seem a natural way to make that happen. In the end, whether you support or oppose this particular bill, I would hope that everyone can recognize that the governance of advanced generalist AI systems, with their jagged capabilities frontiers and their many emergent and often quite surprising behaviours, is an important and complex topic with no easy answers. With that in mind, I definitely appreciate the care with which Senator Wiener has crafted this bill and the open-minded approach he brings to the legislative process. As always, if you're finding value in the show, please do take a moment to share it with friends, post about it on social media, or leave us a review on Apple Podcasts or Spotify. And if you have any feedback, you can contact us via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now, here's my conversation on regulating frontier AI development with California State Senator and sponsor of SB 1047, Scott Wiener.

California State Senator Scott Wiener, welcome to The Cognitive Revolution.

Thank you for having me.

I'm excited for this conversation. So you are the sponsor, or I guess as it's known in California parlance, the author of the proposed legislation SB 1047, which attempts to get the government's arms around this question of what is going on the frontier of AI development and has certainly caused a lot of consternation and attracted a lot of interest lately. So I'm really interested to get your story on how you got interested in this, what you aim to do with this legislation, and then we can get into the weeds on some of the concerns that people have and some, some possible ways to even make it better. How's that sound?

Sure, that sounds great. Thanks for having me and thanks for talking with me about, about the bill. Yes, I have the honor representing San Francisco in the State Senate, and I'm very proud that San Francisco is the beating heart of AI innovation. So much amazing and inspiring work is happening in San Francisco. Because I am a San Franciscan and represent San Francisco, I am surrounded by some incredibly brilliant AI minds and folks, not just folks at the senior levels, but frontline technologists who are doing the work every day. I'm thinking about, I've had great opportunities to talk about policy issues surrounding AI. And probably about a year and a half ago, in a number of different settings, folks in the AI world started talking to me about safety and the need for policymakers to try to get ahead of safety issues rather than playing catch-up, which is what we often do when the horse is already out of the barn. And so we started to just have a lot of conversations about what that might look like and what might make sense and how do we absolutely promote innovation. The last thing we would ever want to do, the last thing I would want to do, as Tim and Keith innovation, how we do that and also make it safe and really address that issue. And so we had an enormous number of conversations, lots of outreach last September before we recessed for the year. I put what we call, I put into print, we introduced a formal piece of legislation, really an outline of what we were looking at in terms of an AI safety bill. And I put that out there. So that it would be just floating out there for months and months. I wanted to be completely transparent about what we were doing, welcome lots of broad-based feedback. We introduced that outline, sent it around to a bunch of people, and then in February introduced the formal bill. And my intentions are really threefold. First, set some basic safety and mitigation requirements that are reasonable light touch and not micromanaging. Government should not be micromanaging this. Light touch, reasonable safety and mitigation requirements that are super doable and super possible for developers of LLMs, fine-tuned models to accomplish. We want to focus on only the most capable models, upcoming really large, powerful models that are beyond what's possible today with tools like Google and other tools that we have when we're talking about future really large models? And then how do we protect and foster innovation, including open source model development? And how do we do it in a way that takes safety into account? So those are our goals and that's how it came about.

You want to give us just a tiny bit of background on your legislative career?

Yeah, so I'm just like a gay Jewish guy who moved to San Francisco from the East Coast in my 7, for the same reasons a lot of queer people came out here to find my community. Practiced law for many years, very involved in the LGBTQ community, and ultimately ran for and was elected to the San Francisco Board of Supervisors, which is like our city council, back in 2010, representing the District of Coeur d'Alene. Used to represent, did a lot of work on the board around housing, public transportation, and And LGBT issues and so forth. And then in 2016, I was elected to the California State Senate representing all of San Francisco and part of San Mateo County, right to the south of San Francisco. State Senate districts in California are huge. I represent about a million people, and I served for a number of years as the chair of the Senate Housing Committee, and I currently serve as chair of the Senate Budget Committee. I think I got appointed budget chair just in time for a massive budget deficit, which is always a lot of fun. And my focus, I'm probably best known for my work around housing policy for trying to remove barriers to building more homes. We make it way, way too hard to build housing. So we try to make it easier and faster to build homes. Done a ton of work to make sure we're adequately funding education and a lot of work around healthcare access, including single-payer healthcare. A lot of work around climate. energy issues, criminal justice reform, so forth. A note here, I, back in 2018, after the Trump administration got rid of net neutrality protections, I don't know if people know, but let's see if anyone doesn't know, net neutrality is the notion that you or I should get to decide where we go on the internet, and AT&T, Verizon, and Comcast and other internet service providers should not be telling us or manipulating us into where to go or not go. On the internet. And so it's through something that I authored and we were able to pass California's net neutrality law, which is the strongest in the country. And so I'm a big believer in democratizing the internet and not having concentration of power, so making sure that the internet is a democratic place. Thank you.

So for this bill, one thing that is striking is that it really does focus on the frontier, and it leaves, I would say, a number of questions which you probably agree are important for another debate at another time. Those would be things like the future of professional licensing, the overall impact on the workforce, the potential for algorithmic bias or discrimination. None of those are really addressed in the bill at all. So the question would be, why? Is that something you think you'll come back to later? And how have you assembled a coalition to support this, given that, from what I understand of the coalition, I would imagine some of those folks would be pretty concerned with those issues as well?

Yeah. And one thing it's important to know, not just about AI policy, but any policy, is you can't do everything in one bill. And so we have a number of members of the legislature who have been and are working on AI-related issues and various bills that are moving forward now. So our bill, SB 747, focuses on safety evaluation and mitigation for the very largest models that are being developed. There is a bill that's pending in the State Assembly by my colleague, Assemblymember Rebecca Arcahi, relating to algorithmic discrimination. There are a few different bills around watermarking of AI-generated images. There are a couple bills on AI-generated revenge porn. There are bills around government contracting around AI-related services and government use of AI. And there are others as well, but there are quite a few AI bills moving forward. And SB 1047 is simply one of them relating to safety in particular. Okay.

Do you have an established position on these other Bills, is there any broader context on your AI worldview that you'd want to share?

Yeah, generally I'm a supporter of artificial intelligence. It has so much potential to make the world a better place, to make people's day-to-day lives better, to address some of the biggest issues of our time around climate or various huge healthcare issues. The potential is limitless in a lot of ways, and I am so excited about what AI can do to to choose life better. But we also know that there are impacts that we need to be mindful of. And so I am supportive of addressing algorithmic discrimination. The last thing we need is to have AI make the discrimination that exists in the world even more pronounced. And so that needs to be addressed. I think it's important for people to be able to know if an image is fake or not. There could be so much disinformation around audio and visual fakes and deepfakes are a real issue. We need to address it. And in terms of impacts on the workforce, it's a really hard issue. Technology always impacts or often impacts job classification in the workforce. But we're looking at impacts on the workforce at a very large scale. And I know that there in a In a theoretical perfect world, if AI made work or made it possible for less work to exist, in an ideal world, the benefits would be spread around to everyone and everyone could just work less and have more free time. But we know that in this society, in this world, we are very bad about spreading around benefits. And what I don't want to see happen is that AI generates huge economic benefits that are enjoyed by a few, whereas most people are made worse off. Because they no longer have jobs. We need to be intentional about what the future looks like in terms of AI's impact on the workforce. That is a huge mega issue that's not going to be solved by a bill or one bill, but that's something we need to focus on.

Hey, we'll continue our interview in a moment after a word from our sponsors.

You said the goal is to have a focus on the frontier. The goal is to have a light touch. How would you describe the burden that the law would place on frontier developers? And how would you say they have reacted to it so far? It's been striking to me that I haven't really heard like an official position from the companies that would seem most likely to be directly impacted in the near term.

Sure. So I'll describe what the bill Does, and I'll also describe after that all the elements that I rejected that some people wanted to put in the bill. My goal here is to be light touch, is to not to micromanage, not to you know undo burdens on people. That's something I don't like passing laws that make people's lives harder just for the sake of making them harder. That's not what that's not who I am, not what I ever want. I do. I want my the laws that have my name on them to. Actually make the world a better place. So the bill, it applies to models that are size threshold. We use the same size threshold that the Biden executive order on AI safety used, which is 10 to the 26th FLOP. And because we don't want to tie everything to where we are in 2024, we want this to be flexible over time. If you're developing that model of that scale, then before you train The model, and before you finalize it and make it available to other people, you need to perform a safety evaluation on the model. There are various kinds of evaluations, right? Red teaming is one of them, but there are others as well. So you have to perform that evaluation. If the evaluation, the testing shows that there is a significant risk to the model leading to catastrophic harm, and I'll tell you in a minute how we define catastrophic harm, then you need to take reasonable mitigations to reduce that risk, not to eliminate risk. Eliminating risk in life is usually not possible. And when we try to eliminate risk, that will take a lot of the joy and innovation out. We'd all be sitting in a coffin in our basement, basically doing nothing. Life is about risk. And so if there are, we want people to at least try to mitigate or reduce those risks. When we talk about catastrophic harms, how we Define it in the bill is that it will cause lead to the deployment or development deployment of nuclear, chemical, biological weapons. That it will cause harm damage to critical infrastructure five hundred million dollars or more, or that it will create trigger some sort of cybercrime or some sort of crime that would be a crime if a human did it causing five hundred million dollars or more in damage. So we're not talking about small things; we're talking about. large-scale destruction and damage. Some people say that thresholds should be higher. Some people say, wait, why $500 million? Isn't $50 million enough? And that's a topic of conversation we could have. And then if there are other harms that are sort of equivalent in scale and destruction to those. And when someone engages in the safety evaluation, you then have to certify to the Department of Technology that you've done so. And if then one of those harms occurs from your model, Then, and you have not done the safety testing, or that you already did the safety testing and it showed that there was something really bad that was likely to happen, you didn't do anything about it, then the Attorney General can sue you. I also just want to note that there are a number of things that I rejected, including in the bill, because I wanted this to be light touch. The first is there are some who wanted us to be able to call it private right of action so that anyone could file a lawsuit against the developer if something were to happen. I did not accept that idea. We limited enforcement to the Attorney General of California, who with their limited resources, so they will focus their enforcement on the really bad actors, not just on someone. There are people who wanted us to have a licensing requirement so that you can't train or release any of these, a lot of these models, Unless the state could commission and gives you a license. I did not want to go in that direction because I don't want the state to be in the middle of all that. Some people wanted us to ban certain kinds of models. We didn't go there. And some people wanted what's called strict liability. Strict liability means if you cause a harm, you're automatically liable whether or not you acted reasonably. I rejected that as well. That's basically what this element of the bill does and some of the items that we did not. I I do just want to address one thing because there was some conversation on Twitter saying some very inaccurate things about the bill. First, they said the bill was being fast-tracked, which is false. The bill— we first announced this 9 months ago and it's on the regular slow boat. And so there's plenty of opportunity for additional dialogue, for amendments. And so it's not being fast-tracked. It's a super transparent process. And the other piece is there was some people saying, oh, this Developers are gonna go to prison for this. You don't make honest mistakes and go to prison, which is also false. The only criminal aspect of the bill is about perjury. If you literally intentionally maliciously lie to the government, that's perjury, just like it would be if you lie on a driver's license application or any other documents with the government. If you actually intentionally lie, misrepresent, that can be charged as perjury. It's not about if you make a good faith mistake or an honest mistake, or you just submit something that was even something sloppy, that would not rise to that level. So anyway, that's, I'll stop there. Yeah.

So that's an important distinction. So just to make sure I get it in layman's terms, basically, obviously these frontier models are very unusual technology in that we don't really know what they can do until they've been created. And then we're still figuring out new things and better ways to prompt GPT-4 a full year after its public release, 18 months after it was finished training. So there's this whole process. What you're saying there is that a developer is responsible for doing some amount of reasonable testing to try to get their own arms around what can this thing do. And then it's their job to report, obviously, honestly to the government that we have tested this. Again, I'm a little fuzzy on exactly how much they're supposed to test, how exhaustive that's supposed to be. Maybe you can clarify that for me. But if they do an honest testing and reporting and they say, hey, we don't think we have a problem here, we can go ahead and distribute this model, and then something happens, they could be Sued would be the base case, and only if it was later found that they actually knew better and were covering up information that they had willfully not reported would they be subject to criminal liability.

Do I have that right? Yes, but the suit, lawsuit, it was only viewed by the Attorney General of California, not by other random people. So it's very limited, very focused enforcement by one law enforcement agency.

Gotcha.

Okay.

So how have the developers reacted to this? I feel like there's a lot of crosstalk in that dimension as well. Some people are of course saying, oh, this is a ploy by the developers to create a regulatory capture environment. Others are saying that it's going to prevent them from doing what they want to do. What has your conversation with the developers been like?

It's been a wide array. We're talking to all sorts of people. And there are a lot of people who, some publicly, but many quietly say, this is the right approach. Thank you for doing this. Largely, they don't want to be public about it. I'm learning a lot about the politics of Silicon Valley and there are a lot of these people who want to be funded. They want to, they, and so there are a lot who don't want to be public about their support. Some do though. And we've had some very prominent folks in the AI world who are supporting the bill. We've had a lot of meetings, both with the large labs, with the mega tech companies, but also with smaller developers. The large tech companies, there's been this narrative that this is about regulatory capture and trying to box out small folks and startups. That is completely untrue. That's not what the bill will in any way do. None of the big tech companies are supporting the bill. We're in conversation with them to try to get feedback, but none of them are supporting the bill. And in fact, TechNet, which is the trade association for all the big tech companies, the Metas and Googles, Amazons, several others, opposing the bill. The ideas in this bill did not come from big tech companies. It actually came from a lot of technologists and folks who focus on AI safety. We have a number of startups that are supporting the bill. I think that also speaks volumes. A lot of folks were just sort of watching and staying away. We have a particular conversation with the open source community. Folks who are, think, who believe that open source should be treated differently from other models, and we're in active conversation with folks in that community. I think open source has enormous potential value in terms of spurring innovation and democratizing access to these models. But there are of course also risks. If you release a very powerful open source model, people could do amazing things with it and could also potentially do harmful things. So that's a continuing conversation. We definitely are not looking to undermine open source.

Objections that I've heard, you mentioned that the big tech companies in their network are opposing. One of the other objections I've heard has been that, well, all the leading developers are basically doing this already anyway. So what's the point? They've all put out model cards and walked us through, here's what we're doing for testing. Anthropic's got the responsible scaling policy. OpenAI's answered that. Google's getting there too. Even Meta with their open source approach has done pretty thorough testing. So, interested to hear what are they saying that they find a problem with? And how would you answer the opposite end of that says, if all the big guys are doing this, why do we need to make it a law?

Yeah, just to be clear, the big tech companies, they have not come out individually. Essentially, we're having conversations with, I think, all of them. I've spoken to the big folks, with the small folks, with the academics, with Absolutely. Everyone, super open door. And yeah, there is testing. So for the folks who are doing reasonable good testing, this bill will have, will have very limited impact because they're already doing it. And so it's not an issue for them. There are other labs that we're aware of who are maybe not doing the best testing. There, we've heard some, this has been This is on Twitter, so I'm not seeing this on Twitter. Some folks who have concerns about Meta's safety testing, whether it's really where it needs to be. And so I'm just repeating what I've seen online. So there are diverse views about whether all of the large labs are doing testing to the degree that they need to. And I've also found in life when it comes to any industry, any ecosystem, It's good to have minimum standards and not just trust that every lab is going to do the right thing. There are plenty of labs that will do the right thing. They're taking safety very seriously, will go above and beyond on testing and on guardrails and mitigations. But that doesn't mean that they all will. And I think it's good to have a level field where there's a minimum standard. And again, this is light touch. Safety rules. We're not looking to micromanage, but we want to make sure everyone, if you're creating a model of this scale, and these are huge, powerful models, just at least do that baseline safety testing.

On the open source question in particular, this is obviously one of the hottest topics. The cause around models that have similar power to what 10^26 would get you in 2024, Has a lot of people projecting out a couple of years and saying, look, we're getting a lot better at this stuff. It's going to become pretty affordable to create something of that power over the next couple of years. And so the threshold effectively, in terms of dollar budget, falls from the rare air today where we're talking hundreds of millions of dollars and relatively few companies can do it, to maybe a couple of years from now, just a couple of million dollars, and maybe like a ton of companies could do it, or even groups or clubs or whatever could rally that many resources. It seems to me like that is right, unless there's some sort of conceptual breakthrough that would allow people to really definitively say that their stuff is safe. Is that sort of a long-term bullet you're willing to bite as of right now, where you basically would just say, hey, look, if there's no way to say that this stuff is safe, then you really just can't open source it into the public? Is that kind of your— default position and hope that there is a breakthrough?

We know that no one can predict the future, but we know that right now and in the future, I think we can see that it's going to be incredibly expensive to develop models of this scale, this magnitude. And so we're talking about significant undertakings to develop models of this scale. And so we think it's reasonable to say it's just too A safety test to at least, again, not eliminating risk, to at least try to mitigate the risk. With respect to open source in particular, we want to protect open source. I mean, very, very clear about that. We're not looking to eliminate open source. That's not what I want to do. It's not what we're going to do. We're working right now, as we speak, in good faith with a lot of people who are a lot of really smart people in the open source space to try to figure out how we can address some of their concerns in a way that still promotes safety in the open source context. And we know that when people, if you take a very powerful open source model, when you're building on that, there's a good chance that what you're building on top of it may be very small in terms of turning that model into some sort of real-world application. And so we do think that if the model developer is best positioned to, to do that safety analysis. However, we are really actively receiving feedback from open source developers and experts to make sure that we are— again, I don't want to, I'm not looking to create any kind of structure that people can't meet. So we're going to continue to work on the open source issue and I'm going to continue having, having open door on it.

One of the best ideas I've heard there would be the idea of maybe getting a little more precise on how much more you can do on an open source model before the responsibility becomes yours. Right now, if I read the legislation correctly, it's like open source developer puts out a model, anybody does downstream additional training or whatever on that, that's still a derivative model. And so the original open source developer retains the responsibility. Some of the ideas I've heard have been like, if you do 10% additional compute, then maybe you should now own the ball. The original open source developer can be off the hook. Is that the kind of thing that you're entertaining now?

Yeah, I think that's a fair conversation and we're open to exploring that and other ideas with the open source folks. Okay.

Hey, we'll continue our interview in a moment after a word from our sponsors.

One other idea that's just like super simple is, what if we just made it illegal to distribute AIs that do certain things? What's the fault in that idea?

Yeah, there's certainly a case to be made that there are certain types of activities that quite should not be allowed. I don't think we want AIs to be creating biological weapons, for example. That's probably the most commonly discussed idea, but we don't really know First of all, what the capabilities of these models are going to be in 2 years, 5 years, 10 years. Even if we decided there are certain things we want banned, we may not even know what it is we would want to ban. And so we decided in this bill, rather than saying what you can and can't develop in terms of capabilities, let's require safety evaluation. And again, I'm not going to pretend that this bill is going to solve all the potential problems. That could ever result from current AI models. But I think it at least takes that step of requiring some introspection and evaluation. And then if you discover, oh wait, this is going to generate biological weapons, trying to mitigate that. And the conversation about whether to restrict or ban particular AI capabilities, that will be an ongoing conversation, but it certainly is a fair conversation to have.

I was surprised also that there's no AI sort of having to identify itself when it goes out into the world. This evolved, Noah Harari, the first rule should be AI must identify itself as AI. Is that just something that you think will be handled by other legislation?

And I should, I do have a list of AI bills in front of me. I think that might be a bill pending this year. I can't remember, but that is certainly an active topic of conversation. And I, I think it's in that there's a dull atmosphere, but I could be wrong about that. But that's certainly an important issue.

I think one of the big ideas that people really worry about in AI broadly is the idea of an arms race between countries, certainly, but even between firms within the United States. The bill obviously insists on a certain amount of testing. But in a world where it's really easy to switch from one language model to another, there is, I think, risk of a sort of winner-take-all dynamic between the companies where giving the testing kind of the shortest possible window that they could give it starts to maybe become attractive. Because I think we might even see this in the next couple of weeks here. Google's going to have their event, they're going to release something, and then it's expected that OpenAI is going to try to come back over the top and steal their thunder. And so you can imagine inside the labs, the conversation might be like, look, we're launching at this date when we can most effectively compete our rival, and you got to have your testing done by then. And whatever we can do by then, we'll call it reasonable. Do you worry about that at all? And do you have any ideas, or has there been any conversation about things that legislation could do to try to mitigate this sort of winner-take-all AI arms race dynamic that seems like it is Starting to take shape.

You mean in terms of people cutting corners?

Yeah, exactly.

Ultimately, under this bill, if someone cuts corners on safety testing, those really shoddy testing or sees problems and just sort of congenitally pretends not to see them, they're going to have to file a certification, and they and the maker be there something. Terrible happens that could lead to liability. The Attorney General could file a lawsuit. Yeah, people especially again, this is light touch. We're not micromanaging. We're not just setting broad parameters, and people can certainly violate the law, act in shoddy, irresponsible ways if they want to. But then there's going to be a risk to them. Maybe they'll get away with it, but that's not life often works that way. At times when people do. problematic things and get away with it, but sometimes they don't. And so they're— and so we're talking about catastrophic harms. So they, developers will need to be mindful. And I think ultimately, well, many developers want to do the right thing. They want, they don't want to release a model that, that's going to cause catastrophic harm. And so I think that we will have good compliance.

The penalties are— how would you describe the penalties? Because I see in the bill like a 10% to 30% sort of fine, 10% to 30% of what it costs to train the model, I believe. And then I think there's like additional sort of possibility for kind of punitive damages or whatever. But I could imagine a situation where you're at Google, for example, and you're like, man, OpenAI is eating our lunch. How much did this thing cost to train again? Oh, it cost $500 million to train. Okay, that means we could be on the hook for $150 it's worth it, right? Do you think that the financial penalties, sort of probabilistic and deferred into the future as they are, are enough to be really persuasive to these companies?

I think so. And especially punitive damages, if it's— and punitive damages requires very high, it's a high threshold of malice. And if punitive damages are triggered, those are typically based on your Net worth. And so punitive damages brought against me as a public servant who doesn't have a lot of money are going to look very different than punitive damages against a trillion-dollar corporation or whatever the size is. And so I think the incentives are good in the bill.

I started my career very briefly in the mortgage space, and it just happened to be at the time when the mortgage industry blew up the American economy. And then it became clear that while there were a lot of people nominally watching after the behavior of the lenders and all the different steps in the value chain, that had eroded to be more of a rubber stamp. And that is a big part of what the safety community most worries about.

It's okay, these people are going to be under pressure.

It's such an easy change to switch from one API call to the other. I can switch from OpenAI to Google to Anthropic literally in one line of code. So it does create these sort of, you have the best model, you win the day dynamics. And then there's this sense of, okay, we're racing. Maybe we want to do this, but do we really want to do it? Do we really want to find everything we could find? Do we really want to take all the time that we could take? It seems like there is likely to be a space where it would be plausible for the developers to have a reasonable defence. Look at all the things we did. This is very reasonable. But then somebody who's really in the know would say, think about all the things you didn't do, and here's all the things that I really think you should have done and you didn't, but it's still reasonable. This leads me to, I think, one of the most common suggestions that I heard in preparing for this, which is a more forceful requirement for truly independent third-party testing. Has there been any conversation around that sort of thing where Who provides that testing, under what circumstances, how long they have, exactly what sort of access they have. There's a lot of particulars there. But I do worry, along with a lot of these safety people, that the motivation or the sort of standard of reasonableness could be met. And yet we might want a higher standard if we're really talking about catastrophic, or as you well know, the safety community even worries about extinction-level risks. Is reasonable enough, or should we sort of say, hey, you have to allow in these sort of Red Hat testers to go in and dig deeper than you might dig on your own.

Yeah, the bill, and again, we are walking a fine line here where we want to have good safety protocols without being too intrusive, right? And we're already— there are people who are criticizing us for being too intrusive. I don't think we are at all. Other people are saying it should be more intrusive. There are, as you know probably better than I do, there are some philosophical divides. Within that AI world. Right now on this issue, the bill calls for developers to use third-party testing, quote, when appropriate. And I understand that is a little vague and certainly an issue we'll continue to take a look at. We're open to the possibility of a greater role for third-party testing, but we also want to know more about who those third-party testers Would be, and I think as you alluded to earlier, and you said, I guess it's just some big accounting firm that comes in and checks the box, or is it real? Our goal here is just to have good testing, and we're open to different ways of approaching that. Again, with that, I know you're going to get sick of me using this phrase, but it's real. As long as it is light touch, that is our goal here, to micromanage. And there are people who would like us to micromanage, right? They want People who want you to have to get permission from the government and get a license before you train or, or in any way release a model at this scale. And I respect that point of view, but that's not where we chose to go in this bill.

Yeah, I do think it's a tough one to figure out exactly how you would specify who the third parties should be and again, exactly what kind of access they should have. I had a personal experience as a member of the GPT-4 red team. And I was really taken aback by what a leap it was. And the public hadn't really seen anything like it. My feeling at the time was that the testing that they were doing appeared to be inadequate. And I became sort of lost in the wilderness, like under NDA on the one hand, like not trying to blow up their spot on the other hand, but like feeling like I was one of just a couple people in the world that had the time to devote And I literally dropped everything else I was doing and worked on it full time for a while. And I felt like, where do I go? One of the problems that I had was that I was totally at the pleasure of OpenAI. I do think one thing that would be really useful would be some protections for the third-party testers, some perhaps like accreditation or some status that they need to achieve, but also some ability for them to report their findings without being at the risk of immediately being cut off because the access dance That they have with the developers is a really tricky one.

Yeah, I think so. In a, in a totally different context, I authored a law that was passed and signed last year. It was a law to require large corporations to disclose their carbon emissions, including from their supply chain. It was a huge fight with a lot of the large business associations, but we got it passed and the governor signed into law. It puts California in the forefront on, and in that law, we require these disclosures and the disclosures have to be audited. And the California Air Resources Board sort of certifies the auditors that are qualified and trusted to do that auditing. Different context, not just subject matter, but also because carbon emissions are not necessarily like trade secrets and whatnot. I guess they could be, but are typically not particularly sensitive in terms of some sort of confidential part of a business model or a technology. Here, obviously, there's a much higher sensitivity in terms of this is like the core product of the business, and there's a lot of confidentiality issues as well. So it's a little touchier, but I think that this is an absolute worthy topic of conversation about how to make sure the testing is as solid as possible.

What about sort of a similar Transparency requirement for model development as well. What if you were to say you have to just disclose how big your training runs are going to be? If you're going to do 2 times 10 to the 26, okay, but you got to put it on record. That would maybe help address these sort of race dynamics where the different developers don't know just how fast the other companies are going, how big. And you might also imagine extending that to things that people have seen models do. I do think there is definitely a place for trade secrets. I don't think total transparency makes sense, if only because then we're also sharing all our secrets with all the governments of the world, which we probably don't want to do. But I've often thought, does it really harm the company if somebody says, hey, I saw an AI do X and it freaked me out? Because I was under an NDA where I couldn't even disclose that sort of thing. And people have been fired, of course, for saying that they think their AIs are conscious and whatnot. I don't I don't necessarily think they are, but I also don't rule it out. It certainly seems like something we should be discussing. I wonder what you think about these sort of— are there ways to put strategic transparency, either requirements on the company or sort of opportunities for very careful, narrow, but hopefully illuminating disclosure for employees or testers that could help everybody at least get a sense for kind of what's going on across the environment?

Is that possible? Yes. And is that worthy of a conversation? Yes. In this bill, we're really focused on just creating, which does not exist now, a baseline requirement to do safety testing, which a number of the labs are already doing, and some are probably doing better and more consistently and thoroughly than others. And we want to make sure that labs that are creating these huge models are doing that baseline testing. Are there other things that types that may be reasonable to put on top of that potentially. But we're really focused on let's just, let's do this, take this important step that is already going to be controversial. We're already seeing that controversy. So that's what we're focused on.

Yeah, makes sense. So on the core testing concept, I hear you. The bottom line seems to be we want to make sure that this testing is happening. We want to make sure that it's good. Quality testing, want to create the right incentives for that openness. You've expressed several times to discussion about potentially a bigger role for third parties or a more kind of forceful insistence on that. Where should people come to have that conversation with you, particularly those that are motivated to do that testing, that feel like they have something to contribute but are currently boxed out of contributing in the way that they would like to?

Yeah. So people can reach out to my office. We're super easy to find. They can also message me on Twitter, @scott_wheeler, W-H-E-E-L-E-R. You can message me there. You can reach out to the office. We really welcome feedback and ideas, and people have been very gracious and helpful in pointing out maybe things in the bill that could be better or things or problems that we may not have anticipated, and we very much welcome that feedback.

Yeah.

I can never guarantee that everyone's going to be in complete agreement with everything in any bill that I author, but I do really value feedback to make the bill as good as it can be. And there are times I've had bills where there are— there's one philosophy of legislating saying, if you're opposing my bill, then I'm not going to listen to anything you have to say. I don't subscribe to that philosophy. My view is that even if you are fighting me on a bill, if you then come forward and say, hey, There's something you missed and this is going to play out in a problematic way. There are times when I'll say, oh my God, yeah, you're totally right. Thank you for bringing that to my attention. We changed the bill, even though the person's going to continue to oppose it. Because my take is I just want my bills to be as good as they can be.

I appreciate what a complicated situation this is, what, how much uncertainty there is on so many of the key questions. And I know you're balancing a lot in terms of different constituencies and trying to create something that has a positive effect and still has a light touch. I definitely appreciate the fact that you're focused on these frontier tales, sometimes dismissed as speculative, but I don't think wisely dismissed as speculative risks. I appreciate the work that you're putting into this and the openness that you've expressed to further input as well. Any other closing thoughts from you?

I appreciate that. First of all, I'm excited about what AI has to offer, and I appreciate the engagement that we're Seeing, I would also just one thing I wasn't asked. My only request is this is true in life, not AI policy, and that's just about our bill. But if you see something on Twitter or elsewhere online about the bill, just please don't assume that's what the bill does because we've seen some real inaccurate information about the bill on Twitter in particular over the last week. I'm not saying it's all. malicious. There are some people who just see things or hear things and then just post about it. And feel free to reach out to us and ask, because we're happy to answer questions.

California State Senator Scott Wiener, sponsor of SB 1047.

Thank you for being part of the Cognitive Revolution. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co Or you can DM me on the social media platform of your choice.