SB1047: The World's Most Important and Problematic AI Policy Debate

Today on the AI Daily Brief, we are talking about SB 1047, the world's most important and problematic AI policy debate. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. To join the conversation, follow the Discord link in our show Notes. Hello friends, Quick note before we dive into today's show. As you will see it is much longer than our normal episodes and because of that there will only be this main discussion. We will not do the headlines, but we will be back tomorrow with our normal format. But for now, let's dig into SB 1047. Welcome back to the AI Daily Brief. Perhaps the most controversial discussion in the artificial intelligence space right now is the discussion around California's SB 1047. On the one side you have those who say that this is common sense light touch regulation, exactly the type of guardrails the industry needs to evolve in a way that is good for everyone and not just good for the companies driving AI. On the other side you have those who see this as having the potential to absolutely chill AI development and innovation. A fraught, if some believe well intentioned attempt to regulate that will do nothing but create more legal landmines that drive entrepreneurs to simply build less. Now, as you might guess from the fact that I do this daily AI podcast and I have an AI enablement company and superintelligent, I find myself more aligned with one side than the other. However, for the purposes of this episode, I'm going to try to represent each side faithfully with the idea that all of you are smart enough to make up your own minds around what you think about this. However, I'm also going to try to explain why the discourse around this seems so fraught. If you've spent any time reading about this or seeing how people talk about it, it feels more tense, more fraught than you might expect from a state legislature bill. Perhaps that is just the current political climate in America in general, but I think it actually might be reflective of something more. In short, there is an underlying and more fundamental disagreement about the risks that we face from AI that makes it very hard to find common ground with this discussion. So let's first do a little bit of background. Here's how the Wall Street Journal sums it up. SB 1047 requires that developers of large AI models conduct safety tests to reduce the risk of catastrophic harm from their technology. It defines catastrophic harm as cyberattacks that cause at least 500 million in damage or mass casualties. Developers must also have a kill switch that ensures that AI can be shut down by a human. The bill has a computing power threshold defined in terms of how much it costs to train. That level is set at around $100 million. In terms of where the bill is, SB 1047 has been passed by the California Senate and two of the state's assembly committees. The Democrat controlled legislature in California is now back in session. And so the push to get changes made has gotten frantic now, as the bill could be on its last steps before being sent to Governor Gavin Newsom for signing or veto. The bill was drafted by Scott Weiner, who's a Democratic state senator from San Francisco. He was my representative when I lived there back in the teens, and Weiner has tried to argue strongly that he is not anti innovation and that he is not anti oversource. He argues that the bill is not overly prescriptive, in fact that it only codifies safety standards that the industry has already set on its own. Gdsuper.com gives a few additional details that help understand the bill. First, as we discussed, the bill would regulate AI models based on the amount of money and computing power used to train them. They point out, however, that the bill also seeks to regulate covered model derivatives, which include copies of covered models, modified versions of covered models, and covered models combined with other software. The bill would also set up a new state agency called the Frontier Model Division. Next, there would be a variety of compliance obligations for developers of covered models. Quote, before training a covered model, a developer would have to implement administrative, technical and physical cybersecurity protections, the capability to promptly enact a full shutdown of the AI, and a comprehensive written safety or security protocol that the developer would have to provide to the state. Again, that's before training the model. Now, before releasing the model, the developer would have to assess whether the model is reasonably capable of causing or enabling critical harm implement reasonable safeguards to prevent the model and its derivatives from causing or enabling critical harm Ensure, to the extent reasonably possible, that the covered model's actions and the actions of covered model derivatives can be accurately and reliably attributed to them. You might already beginning to see some of the problems for people who are against this legislation. There's a lot of words like reasonable, reasonable safeguards, reasonably capable that create a ton of legal ambiguity. Legal ambiguity, of course, leads to expensive legal fees. In addition to the compliance rules around the developers, the legislation would also regulate operators of computing clusters that were used to train covered models. Those operators would have to obtain a prospective customer's identifying information and business purpose assess whether a prospective customer intends to use the computing cluster to train a covered model, retain records of customers, IP addresses used, and implement the capability to promptly enact a full shutdown of any resources used to train or operate models. Basically, it would give operators of the computing clusters extreme surveillance requirements and significant power and responsibility over their customers. Next, there would be significant labor protections for employees of the developers, basically protecting them if they have to become whistleblowers. This, I will say, is probably the least controversial part of this whole bill. So let's talk now about the pro position for this. We'll get into some specifics, but here's how I might sum up some of their points. First, there is a general belief that industry can't be trusted to regulate itself. There is an argument that only big models are implicated and so the concerns around little tech and open source are overblown. There is an argument that this is light touch relative to the extreme AI existential risks, a point that we'll come back to. There is an argument that the rules have the flexibility to be applied in ways that accommodate the reality as the reality becomes clear over time. In other words, that this doesn't lock us into overly prescriptive policy. And there's an argument that this largely reflects voluntary commitments that have already been made and so why the logic follows Wouldn't companies who already have made those voluntary commitments be willing to enshrine them in the letter of the law? Sunny Gandhi, the VP of Political affairs at Encode justice, wrote an op ed called California Shouldn't Buy Big Tech. Talking points that AI regulation will hurt innovation. He writes, you might think this is a debate between big tech and slow government. You might think this is a debate between those who would protect technological innovation and those who would regulate it away. Or you might think that this is a debate that decides if AI development will stay or leave California. These arguments could not be more wrong. SP1047 is about ensuring that the most powerful AI models, those with the potential to cause catastrophic harm, are developed responsibly. We're talking about AI systems that could potentially create bioweapons, crash critical infrastructure, or engineer damage on a societal scale. These aren't science fiction scenarios. They're real possibilities that demand immediate attention. He goes on to argue that the arguments from the tech industry that say that this will drive innovation out of California is simply not true because the bill only applies to companies spending hundreds of millions. He calls this fear mongering. He also explicitly calls out the effective accelerationist movement. He calls them tech zealots that dream of a world where AI develops unchecked, regardless of the consequences. They list concepts like sustainability, social responsibility, and ethics as enemies to be vanquished, and feverishly dream of a world where technology replaces humans. Sonny argues that he is mischaracterized as being anti technology. He says he rejects that completely. I'm a digital native that sees AI's immense potential to solve global challenges. I'm deeply optimistic about the future of technology, but I also understand the need for guardrails. Now, I want you to put a pin in that section about the effective accelerationists and the characterization of them as tech zealots and that line that these aren't science fiction scenarios. Recently, some of the loudest supporters of the pause movement, Turing Award winners Yoshua Bengio and Geoffrey Hinton, have also written a letter expressing their support for the bill. That letter was also signed by Lawrence Lessig and Stuart Russell, and they described the bill as, quote, the bare minimum for effective regulation of this technology. They say there are fewer regulations on AI systems that could pose catastrophic risks than on sandwich shops or hairdressers. They write, as some of the experts who understand these systems most, we can say confidently that these risks are probable and significant enough to make their safety testing and common sense precautions necessary. In an email to Time magazine, Bengio reinforced the argument that industry can't be trusted to regulate itself. He wrote, I worry that technology companies will not solve these significant risks on their own while locked in their race for market share and profit maximization. That's why we need some rules for those who are at the frontier of this race. That was, of course, why Geoffrey Hinton said that he left Google last year, namely that he was concerned that the competition around AI was making them act in ways that he no longer agreed with. The letter says, relative to the scale of the risks we are facing, this is a remarkably light touch piece of legislation. Now the pause AI movement goes even farther than the four authors of this open letter saying the language that Bengio, Hinton, Lawrence and Russell use should tell you SB 1047 is better than nothing. But it's not enough to keep us safe. We need to globally stop AI companies from gambling with our future. We need an international pause cause. By the way, they quote tweeted Dan Hendricks, the director of the center for AI Safety, who was integrally involved in writing this legislation, which, as we will come to, is part of the challenge with having any sort of common ground discussion of it. Now let's talk about the con side of this argument. A huge variety of industry organizations, led by venture capital firms and the tech companies themselves, have organized an effort to stop SB 1047. The arguments against it are very diverse. There are competitiveness concerns, particularly with China, as well as in terms of keeping California as a bastion of innovation versus seeing startups locate elsewhere. Some see this as a boon for regulatory capture, with only the biggest companies able to comply. Some point out that this doesn't address immediate concerns and that it's not based on specific immediate risks. Some argue that it's too early to know what the actual risks are. People point to arbitrary lines and concerns around developer liability and the fact that this creates significant new governmental power. StopSB1047.com has a collection of many of these arguments. The website lists a set of unintended consequences of SB 1047. They write that it could put a chilling effect on AI investment and development in California, that it penalizes developers for outcomes that are obscure and requires testing that doesn't exist yet. They write that the bill's vague definitions, coupled with stiff legal liability, create substantial uncertainty and economic risk for AI developers and business owners. They argue that it could unintentionally make AI systems less secure by driving research into the shadows that the bill creates systemic disadvantages for open source and startup developers, and that it disincentivizes AI research and innovation in the US Giving other countries, notably China, the opportunity to overtake the US and AI development. Andreessen Horowitz, who has been one of the loudest voices in this fight, wrote a 14 page response to Senator Weiner in response to a letter that he had written regarding A16Z's public comments about SB1047, they write. Your July 1, 2024 letter made certain legal claims that are inconsistent with a plain reading of the proposed bill in its current form. Notably, they point to SB 1047 applies to startups because of its arbitrary and shifting thresholds, SB1047's criminal penalties will deter innovation. SP 1047 creates new civil liability 1047 will hurt California's economy 1047 is troublingly vague. SP 1047's kill switch requirement imposes executive burdens on new AI models, and SP1047 privileges closed source models over their open source counterparts. I'm not going to read all 14 pages of this, but you can hear these themes coming through over and over again. I think from an outsider's perspective. Some of these specific discrete concerns that ring most clear are around the vagueness of the language. Anyone who has spent any time around. Regulation knows that the more vague things are defined, the more interpretation there is, which means more power in the hands of regulators and lawyers and higher burdens and compliance costs on the subject of the regulations. And this is true even in the case of well intentioned regulation. And indeed, well intentioned regulation is exactly what Fei Fei Li, who is sometimes called the godmother of AI as opposed to Geoffrey Hinton's godfather of AI, says the bill is even though she thinks it's wrong. Her main arguments in an open letter are that one SP1047 will unduly punish developers and stifle innovation. She writes, in the event of misuse of an AI model, SP1047 holds liable the party responsible and the original developer of that model. It is impossible for each AI developer, particularly budding coders and entrepreneurs, to predict every possible use of their model. SB 1047 will force developers to pull back and act defensively. Our second argument is that SB 1047 will shackle open source development. She argues specifically that the kill switch requirement would devastate the open source community. Third, she writes that the bill would cripple public sector and academic AI research. She writes, open source development is important in the private sector, but vital to academia, which cannot advance without collaboration and access to model data. She also says, most alarmingly, this bill does not address the potential harms of AI advancement, including bias and deepfakes. And again, I'm sorry to keep saying put a pin in this, but this really will become part of the central tension of this entire conversation. Ultimately, she writes, in various conversations with President Biden over the past year, I have expressed the need for a moonshot mentality to spur our country's AI education, research and development. SB 1047, however, is overly and arbitrarily restrictive and will not only chill California's AI ecosystem, but also have troubling downstream implications for AI across the nation. She argues that she is not against AI governance, but that we should, quote, work together to craft AI legislation that will truly build the technology enabled human centered society of tomorrow. A few other con arguments just before we wrap up Today's episode is brought to you by superintelligent, the platform that helps teams maximize AI. Super is of course, the platform that we've been building that pairs fun, fast, practical video tutorials with step by step instructions to get you actually using AI and from there unlocks information about hundreds of use cases that show you how people are actually getting value out of AI right now. Now we have just launched super for teams. This is a new add on experience that allows teams to share more information about what AI they're using, how it's working, and how to get more value out of it. Whether your company is 25 or 2500, superintelligent is going to be the best platform for unlocking information about how to get the most out of AI right now. If you'd like to learn more about the super intelligent teams offering, go to Besuper AI Partner and send us a note so that our team can get right back to you. Once again, that's Besuper AI Partner. Today's episode is brought to you by Venice. Venice is a private, uncensored, generative AI app. It accesses open source models to enable text, image and code generation without the fear of being spied on or having your data exploited. Discuss anything with Venice without concern about it being monitored, sold or given to advertisers and governments. Venice is different because your conversations and creations are kept securely within the browser, never stored or accessible by Venice. Unlike other AI apps, Venice won't tell you what's okay to say or not. Venice won't patronize you. It simply provides direct access to machine intelligence. No topics are off limits, no ideas are taboo. With Venice, you're in control of the AI as you should be. Pro subscriptions are available for $49 a year or $8 per month. AI Daily Brief listeners receive a 20% discount on Venice Pro. Visit Venice AI NLW and enter the discount code nlwdailybrief. That's NLW Daily Brief. All one word. The first, notably comes from Anthropic. Anthropic is of the big tech labs, the one most associated with AI safety and the effective altruism movement, which is part of the origination story of in short, Anthropic does not support the bill. In a letter they wrote, ensuring the safe development of AI technology is a worthy goal, but the current version of SB 1047 has substantial drawbacks that harm its safety aspects and could blunt America's competitive edge in AI development. Our letter proposes to refocus the bill on frontier AI safety and away from approaches that aren't adaptable enough for a rapidly evolving technology. And then there's the academic community. Russell Wall, the deputy director at the Stanford Institute for human centered AI, says SB 1047 does not offer the regulation California needs and instead attempts to combat unsubstantiated X risk while not focusing on the immediate harms of AI. And this is really where we start to get into the rub of this, another open letter comes from a group of University of California students and faculty from seven campuses. They write, as academics, we unequivocally oppose this bill for concern that it may seriously harm both the research and educational objectives of California universities in AI. We call on our representatives to seriously consider these harms when weighing whether to pass this legislation. Industry actors are not the only ones who vociferously oppose this bill. But more than any other open letter, this note calls out the point that the entire bill at core is predicated on a belief in a very particular set of AI X risk. They write, as experts in artificial intelligence, machine learning, and natural language processing, we must stress that both the existence of meaningful AI risks and the proposed methods of assessing model risk alluded to in SB 1047 are deeply dubious. There is no consensus in the scientific community on whether and how language models or other frontier AI systems may pose a threat to the public. It is questionable whether a language model can act as an information generating system that is any more empowering to a nefarious actor than a search engine any would be. Bioweapon developer can just as easily do Google searches and scour textbooks to find any of the kinds of chemical, biological, radiological, or nuclear weapon harms that have so far been demonstrated. Furthermore, we do not believe that existential or catastrophic risks from AI development, the primary motivator for most in the AI safety community, are sufficiently evidence based to guide policy. We find it deeply troubling that a law with potentially severe intellectual and economic harms is built atop such shaky intellectual ground. For example, Senator Weiner's response to earlier criticisms of SB 1047 leveled by Y Combinator and Andreessen. Horowitz notes a point of common desire to regulate AI for safety between the bill's promoters and its critics, and that the late feedback on the bill is a surprise to him as he posted an outline in September 2023 for the purpose of soliciting early feedback. We would like to emphasize that only those who already believe in existential risks of AI systems, a group that, while containing some field luminaries does not hold a consensus position in the AI research community, would participate in discussions around safety legislation in its early stages. As we reject the frame that the risks alluded to in the bill are real enough to warrant regulation, it is hard to offer constructive criticisms to improve the bill, and we instead call on our representatives to seriously reconsider passing such a bill at all. Any fact based debate around this bill must start from the questionably factual nature of AI risk. And this is the real heart of the problem. I think that was incredibly, clearly and succinctly put. But to try to simplify this, what this group of academics and students are saying is that this bill is predicated upon the belief in the AI X risks that the AI safety community is so concerned about. When I read that letter from Sonny, he made clear that these were, quote, not science fiction scenarios. When I referenced the letter from Bengio and Hinton, it said relative to the huge scale of the risks. But again, that is an argument predicated on a belief in those risks. The point that these folks are making is that there is not consensus on those risks. In fact, there is fairly significant disagreement about it. And so when a bill is explicitly designed to address those risks, there is no common ground to start from. In other words, this bill didn't start from a question what are the real risks of AI and how might we address them? It started from a very specific assessment of the risks of AI that come from the AI safety movement. Ars Technica pointed this out in late July as well. In a section of an article called Attack of the Killer AI. They pointed to the shutdown provisions and said this kind of language in the bill likely reflects the particular fears of its original drafter, center for AI Safety co founder Dan hendricks. In a 2023 Time magazine piece, Hendriks makes the maximalist existential argument that evolutionary pressures will likely ingrain AIs with behaviors that promote self preservation and lead to a pathway toward being supplanted as the Earth's dominant species. If Hendrix is right, then legislation like SB 1047 seems like a common sense precaution. Indeed, it might not go far enough. However. Critics argue that AI policy shouldn't be led by outlandish fears of future systems that resemble science fiction more than current technology. They quote writer Daniel Jeffries, who SB 1047 was originally drafted by non profit groups that believe in the end of the world by sentient machines like Dan Hendricks, center for AI Safety. You cannot start from this premise and create a sane, sound, light touch safety bill. Tech policy expert Neer at Weissblatt adds, if we see any power seeking behavior here, it is not of AI systems but of AI doomers with their fictional fears. They try to pass fictional led legislation, one that, according to numerous AI experts and open source advocates, could ruin California and the US's technological advantage. And so here we really come down to it. Why there is so much acrimony around this discussion. Hold aside the details of where lines are drawn around how big a model has to be to be implicated by this? The specific structure of the new regulatory bodies that have power over these rules, the culpability of developers of open source systems. All of those things are incidental to the core disagreement, which is about just how risky AI is. In many ways, this is really a debate about the precautionary principle, the idea that some technology is dangerous enough that we need to regulate it in advance of it coming to fruition. In Forbes, James Brogle, a senior fellow at the Competitive Enterprise Institute, the precautionary principle, which advocates for banning potentially harmful technologies until they can be demonstrated to be safe, can be an appropriate response in cases where there are clear and imminent existential threats. For example, it makes sense to restrict public access to fissile materials like enriched uranium, or to prohibit the everyday sale of grenade launchers. The problem is that existential threats from AI are not obviously imminent, and the US is locked in a race with China. Moreover, the likely benefits of AI, from revolutionizing research to curing some of the world's most intractable diseases, are so substantial, withholding them from the public is nowhere near realistic. The battle then here really is not about specific remediations to AI risk, but about the nature of AI risk in general. On the one side are the ea, or effective Altruist aligned organizations that are extremely focused on AI X risk. This includes groups like Open Philanthropy, the Center for AI Safety, and the Future of Life Institute. On the other side are the Accelerationists, the group that Sunny Gotti called tech zealots who dream of a world where AI develops unchecked, regardless of the consequences. The group that he argues feverishly dream of a world where technology replaces humans. There is absolutely a transhumanist element to parts of the accelerationist movement. But there are also a huge number of people who disagree fundamentally with the AI safety community's assessment of X risk, who have absolutely no desire to see technology replace humans, and who, frankly, would be pretty pissed at being painted with such a broad brush. In the middle, of course, are regular individuals. In that same essay, Gotti points to recent polling that suggests that 66% of voters don't trust tech companies to prioritize AI safety on their own, and 82% who support the core provisions captured in SB 1047. This, I think, reflects two things. One is just some core amount of common sense that incentives are strong and that a role of government is in many people's minds, not only to foster innovation but to set limits to the power of private enterprise. But we are also dealing with an environment where there is an increasing base level hostility to technology. Nowhere is that more expressed than in these conversations around artificial intelligence. And frankly, the tech industry does not by and large have credibility to address this. Over the last decade, tech has lost the American public when it comes to popular opinion, and I think that things are only getting more fractured as many in tech decide to loudly weigh into the current presidential election in the United States. So where do we go from here? Well, this is firmly moving out of the state discussion level. Yesterday, Zoe Lofgren, hardly a tech revolutionary firebrand, wrote an open letter to Scott Wiener. Lofgren is the ranking member of the House Committee on Science, Space and Technology, which has jurisdiction over AI. She writes in the letter, as you know, I rarely contact the California legislature to provide my views on state legislation. I had not expected SB 1047 to move quickly. Now that it has, I feel an obligation to provide input. I have carefully reviewed the proposals in SB 1047 and I have some concerns that I wanted to relay to you. To be clear, I firmly support AI governance to guard against demonstrable risks to public safety. Keyword here being demonstrable. Unfortunately, this bill would fall short of these goals, creating unnecessary risks for both the public and California's economy. The surrounding AI safety is still in its infancy. The current state of the technical solutions that would underpin implementation of SB 1047, including standards, benchmarks, and evaluations, is significantly underdeveloped. The bill requires firms to adhere to voluntary guidance issued by industry and the National Institute of Standards and Technology, which does not yet exist. For example, even though we do not yet have the standard evaluations necessary for a developer to confirm with confidence that an AI system could cause a critical harm. The bill bases its liability provisions upon such hypothetical guidance. Such premature requirements based on underdeveloped science call into question from the outset the efficacy of the bill in achieving its goals of protecting public safety. Further, Lofgren continues, SB 1047 seems heavily skewed towards addressing hypothetical existential risks while largely ignoring demonstrable AI risks like misinformation, discrimination, non consensual deepfakes, environmental impacts, and workforce displacement. There is little scientific evidence of mass casualties or harmful weapons created from advanced models. On the other hand, there is ample evidence of real world AI risks like discrimination and misinformation, which are already a problem in our communities. By focusing on hypothetical risks rather than demonstrable risks, the efficacy of this legislation addressing real societal harms is called into question. Now she also talks about some specific unintended consequences. But I think overall it's very clear that it is the underlying core of this bill which causes her concern, her recommendation I'm deeply concerned that much of the bill is legislating requirements without a sound evidentiary basis. I strongly urge the legislature to put this bill aside for further study and consideration. I am confident of the good intentions that you, the principal author, have, but I'm equally confident that this bill would not be good for our state, for the startup community, for scientific development, or even for protection against possible harm associated with AI development. Now, as I mentioned at the beginning, Legislature is back in session and so we will be hearing more, not less of this. Ultimately, as I think it's probably pretty clear if you've listened all the way, I do not think that the starting point for our regulatory considerations should be strictly the belief set of one group when it comes to perspectives on AI safety and AI risk. I find myself in the pause camp. Not pause AI development, but in the camp of pausing writing bills before there's a chance for much more conversation and debate. That doesn't mean we should stop, and I'm not even upset at the AI safety movement for trying to push this forward. They are a group that has, whether you disagree with them or not, a set of opinions that in our system of government they get to advocate for through all the means available to them. But I think that on a fundamental level, the conversation needs to widen when it comes to making policy. These are difficult questions, but they are not impossible ones. And I think the more that we can involve a broader cross section of people and perspectives here rather than the extremely loud voices on the far ends of either side of this debate, the better the result is going to be. For now, though, that is going to do it for this quite long episode of the AI Daily Brief. Until next time, peace.