SB 1047: The Bill That Could Hinder AI Progress ft. Anjney Midha

Can you break down what is SB 1047?

SB 1047 is a proposed law, but this one is the most harmful to little tech.

Being a know it all used to be considered a bad thing, but in business it's everything. Because right now most businesses only use 20% of their data. Unless you have HubSpot, where data that's buried in emails, call logs and meeting notes become insights that help you grow your business. Because when you know more, you grow more. Visit HubSpot.com to learn more.

Hey, welcome to the Next Wave podcast. I'm Matt Wolfe. I'm here with Nathan Lands and today we've got a really important episode. Today we're talking with Anjani Mida and he's a general partner over at A16Z. He was on the ground floor for companies like Midjourney and Luma and Anthropic and some of the biggest AI companies in the world. And today his message is really important. He's talking about legislation that they're trying to pass right now in California that could kill AI. And this bill is called SB 1047 and it will really, really hinder AI progress if it gets passed. And in this conversation we're going to talk to him about what this bill is, why you should care about this bill, some better options for AI regulation and what we can all do about it to make sure that the right regulations get passed and the wrong regulations do not. So this is a fascinating conversation with a lot to learn. So let's jump right in with Anjane Mida. Well, welcome to the show. Anjanay, thanks so much for joining us. We're really excited to talk to you about AI and AI regulation and AI investments. So thanks for joining us today and how you doing?

I'm doing great. Thanks for having me.

Why don't we start by getting a little bit of background. Can you, can you break down what is SB 1047 like in layman's term? What, what do we need to know about it?

Yeah, so look, basically SB 1047 is a proposed law. It's a California state bill that, that's making its way through the California legislature right now. That's part of a much broader wave of, of like 750 or so new pieces of AI legislation that have been proposed in the US since Biden signed the AI executive order late last year.

But this one is the most harmful to little tech. You know, little tech is startups, open source researchers, academia. You know, unlike many of those well intentioned builds which say, hey, Folks, AI is a powerful new useful technology. And like many useful technologies like electricity or the Internet, which has good and bad uses, we should be thoughtful about how this technology is used. We should punish bad actors for doing bad things with that neutral technology. Unlike that rational approach, this bill is drafted to attack underlying model researchers, scientists and developers. And among other things, it's trying to place civil and criminal liabilities on developers of AI models as opposed to focusing on the malicious users of those models.

Right.

So as proposed by this bill, overseeing these new laws would be a Frontier model division, which is kind of like a new dmv. They want to form a new regulatory agency that would have the power to propose requirements on startups, on researchers, on academia, that would dictate if a researcher or an engineer could ultimately be thrown in jail or not. Now it's so crazy that when this bill was proposed amongst tons and tons of other bills, most people read it and said okay, crazy bills like this get proposed all the time. This is never going to get anywhere but the California Senate. Bast SB 1047 in May, 32 to 1.

Oh wow.

And so this bill is now slated for California assembly vote in August, less than 60 days away. If passed, we are one signature from Gavin Newsom away from cementing this into California law. And so this is a incredibly dangerous piece of well intentioned but incredibly misguided regulation that is trying to make AI safer by focusing on the underlying model instead of the malicious misuses, which is.

Really where we should be focusing on if that's passed. I don't see how you build an AI startup in California. I don't. Why, why would you like take that risk? You'd go to Texas or somewhere else.

There's, I mean no rational AI researcher or scientist is going to risk being thrown in jail.

Right.

Just to pursue their research in California. Right, right. You know, I think that if I had to be really sympathetic, I think he's probably trying to elevate the attention that AI gets. But it's just proposed such misinformed drafting that's completely devoid from the way these models are actually researched, trained and developed in the real world. When it comes to frontier AI research, a local legislator who has no background in AI development, technology development, and whose co authors are, frankly, I believe the most experience that the bill's co authors have is four months real world experience in a lab that has actually developed and productionized these models is a four month internship at Google.

Wow.

One of the co authors of the bill is a well meaning high school Think tank. I think it's like staffed by a couple of high school researchers or something.

I mean, both from a substantive analysis, we've put out tons and tons of critiques of the substantive pieces of the bill. But the process around this bill. Oh my God. I mean, it's a joke. Yeah, right. And that's what ultimately led us to launching this website, Stop SB 1047, two weeks ago. Because, you know, after many attempts to provide the senator and his team with feedback on the problems with the bill and how to address it.

And being ignored, you know, our founders and the little tech community just got super frustrated when every new revision of the bill just ignored all that feedback and instead made the bill even worse.

And so, you know, with this August vote deadline looming, it's just become so important and urgent to amplify those voices that are being ignored. Right? Like startups, researchers, the open source community at large to voice their concerns. And so we just wanted to amplify those concerns that Scott Wiener's team has been ignoring. I have learned now, even if you're not interested in politics, politics takes an interest in you.

Right.

A lot of legislators, I think, especially in other states, are being quite thoughtful about saying, you know what, we're open to feedback, give us feedback, and they're making revisions to the bill that actually address that feedback. That's not the case here.

This is a process that's been led by a legislator who keeps saying, I'm open minded to feedback. You know, takes a bunch of founders time and company's time. And then when you see the new draft, it addresses none of the core issues.

So for like a, a theoretical example of like, what this could mean, and, and you can definitely correct me if I'm wrong, but so if, let's say there's like an open source model out there that, you know, just some people developed and they put online, made it open source, somebody else grabs that open source model, uses it to like hack into a government system. I don't know, something like that. They use the model to like, do, you know, do some bad actor stuff. The people who made the model are just as liable as the people who actually did the hacking. Right?

That's right. What, what they're saying, what if you read the bill, what the bill is saying is if you open source a model that meets some, you know, criteria that they're, that they, that they've put, which is completely arbitrary, we can get to that in a second. But if you open source a covered model, you have to certify that this model cannot be used for any catastrophic harms. And if somebody downstream picks up your model and does something bad with it, fine tunes it, changes it, modifies it in ways that you didn't control and does something bad, you are liable as the open source developer for the harm they did. And they're placing this perjury, you know, penalty on that developer. And you might go, okay, Anj well, perjury is, is kind of, that's pretty severe. If you perjure in the, if you're guilty of perjury, you get thrown in jail. Yes, you do. And so what this bill is proposing is that if an open source model developer fails to certify app appropriately, and by the way, there's no real definitions proposed yet, all they're saying is this new agency will have the full rights to determine these, these definitions in the future. Yeah. You could potentially be held liable.

Yeah.

And I think that's just crazy.

Yeah, yeah, definitely. I mean, so is this only open source or does this apply to the closed source models as well?

They're, they are proposing civil and criminal liabilities on all model developers. Open source, closed source, if. Yeah.

So how, how, how is this like disproportionately affecting the smaller businesses building open source than it is the OpenAI's, Microsoft's, Googles of the world?

Oh, this is classic regressive tax. Right. If you just think about a concept of a regressive tax versus a progressive tax, a regressive tax is something that disproportionately hits less resourced people than people with more resources. Right. And the way they've drafted the bill by putting all of this burden of definitions that have no crisp, precise definition today, what's going to happen is if this bill passes, this agency is going to get lobbied by big tech, who has armies and armies of lawyers and compliance experts to shape the definitions.

In their favor. And tiny startups, open source researchers, academic labs who don't have all those resources will just be left out in the cold. We've seen this happen with multiple industries and that's what's going to happen here as well.

It actually sort of helps some of these bigger companies with the regulatory capture that, you know, they're not outright saying they're going for, but they're probably going for. Right.

100%. You know, let's take one example from the bill which, which the sponsor, the bill sponsor, Scott Wiener, keeps saying, oh, look, this is, this, my definition of what a covered model is only applies to big tech companies because it only gets triggered by $100 million training threshold. Okay, well hold on a second. The big tech companies training budgets are in the billions. So first of all, if all you cared about was really just attacking and regulating big tech, you would start your bill with the number B4 billion.

Right?

Number two, what even is a training budget? There's no such canonical definition today. This space is so early that you know, if I sampled the 16 different budget AI model startups that I've invested in over the last three years for their definition of training, every single one has a slightly different meaning. Right. Pre training vs post training vs fine tuning vs computing Latent representations like past training runs. If I took Llama's. If I'm a startup and I took llama 3, which cost, call it about 100 plus million to train and then I fine tuned it. Does their training expenditure applied in mind too? None of these definitions have at all been. The bill's authors have proposed zero definitions around these pretty important issues. Right?

Do you think that's purposeful? Because obviously if you leave it vague like that, that gives them so much power and control over all this. Right?

Look, I think there's the generous interpretation and.

The less generous one. The generous one. There's this idea of Occam's razor. Right. The simplest explanation is usually the right one. When I first read the bill, I was so worked up, I was like, wow, this has been maliciously vague.

Right.

To put this burden on model developers. When I then looked at the bill's authors and their backgrounds, then I realized that they just don't know what they're talking about. Right. I kid you not. There's literally zero, I think beyond one researcher on that team who spent a four months inside of a lab as an intern. They don't have any experts on the drafting team who've actually trained models, who've deployed them, who've worked at startups for an extended amount of time that are frontier model companies. I mean, I just think they don't, I think they're well intentioned. I wish I could tell you they had the competence to have done this maliciously. I think there's good reason to believe they're just way in over their heads with no real world experience here.

Right, Right. Don't attribute the malice what you can explain with ignorance or whatever. I don't remember the exact quote, but that seems to be what's going on here.

Right.

I'm curious, so if, if you were sort of like an advisor to help with like creating some regulation, are there things that you believe should be regulated? Or do you think it should just be, you know, open door, let's just push forward, accelerate at all cost? Or are there some areas where you're like, okay, this is, these are areas I think should be regulated.

Oh, I'm absolutely in favor of regulation. Let's make it clear. Models are powerful tools like electricity. They can be used for good and bad and we should focus on preventing people from doing bad things with it. But this approach to regulating the underlying technology and placing burdens on researchers instead of placing the burdens on the misuses of the models is completely misguided. So I have an issue with this particular piece of legislation. I don't have a problem with regulation and thought especially regulation that's thoughtful, that's drafted in partnership with industry that puts America first, that doesn't just hand away our entire AI startup industry to China. I mean.

Yes, I'm absolutely in favor of regulation. If you were asking me if you were drafting.

Legislation with policymakers to make sure AI is developed safely and responsibly, what would you prioritize? I'd probably look for three basic principles in that drafting. One, focus on the misuses, not the models. Right. Focus on the malicious users, not the underlying infrastructure. The second would be to prioritize concrete security problems over these sort of super theoretical, borderline sci fi, doomsday Terminator scenarios that they're calling AI safety. Those are not our most pressing safety issues where a model autonomously goes rogue and launches a cyber attack on our power grid. Like that's, that's the plotline of a Schwarzenegger movie.

Right?

Right.

What is happening? And I know this because our portfolio companies are being attacked by this. We get, we get approached by law enforcement agencies all the time is in fact like good old fashioned spear phishing, misinformation attacks, identity theft. That are, where these attacks are increasing in speed and scale because bad actors are using AI tools.

Right?

It's the same attack vectors. We have laws that say these are, these are illegal. We don't need more laws to say these should be even more illegal. What we do need is laws to bolster enforcement. You know, give, invest in defensive tools that our agencies can then use to fight these, these increasing speed and scale of AI security. That's the problem we should be focusing on. Right? So anyway, that's the second thing. Let's, let's prioritize concrete AI security over sort of doomsday safety scenarios that are, that have no, that have very, almost zero empirical evidence that, that these will ever come to pass. And, and then I think the third thing I would do is to really prioritize open source development in the United States to maintain the competitive edge we have globally.

Right.

Because us placing these burdens on our startups, our open source researchers, our universities, is not slowing down China. They're full steam ahead. But if you prevent our open source ecosystem from collaborating, from putting out models that people can research, can fine tune, can red team to make them more secure, you're going to hurt us and you're hurting the US national competitiveness. Nobody else, while everybody else races ahead. So those are sort of the three simplest principles. We provided that feedback ad nauseam, to be honest to.

The Senator, but none of the amendments to the bill have addressed these core issues.

Speaking of China, I don't know if you saw the news today, but it looks like OpenAI is going to be banning ChatGPT in China. It looks like this is possibly in collaboration with the US government or at the direction of the US government. So I do wonder if we're going to end up in a scenario where OpenAI and the other major AI players who are closed source, if they're already in collaboration with the US government behind the scenes. You had a member of the NSA who joined as I think a board member of OpenAI recently, Mira Moradi, the CTO. She openly said in an interview that yeah, they collaborate with the US government in terms of like showing them the new models before they come out. I do wonder if that's going to lead to a world where, yeah, the closed source models, they're collaborating with the US government because the US government sees this as like a national security, like it's, it's, it's a, an asset to the US but also it's a, it's a security threat, it's a risk as well. And if they will actually end up pushing that, there should be no open source. Because of that, I'm almost kind of like what Founders Fund is saying, right? Where it's like, and you know, and that's one area where I'm a little bit conflicted because like I saw some of the people at Founders Fund saying that like open source AI can be dangerous, right? And like, and that's actually going to help China. So I'd love to hear your thoughts on that, like how, you know, kind of a 16Z is more on the, on the side of open source. And it seems like Founders Fund is like slightly against open source for AI.

Look, I, I think any arguments that claim that Open Source AI is a threat to national security are either frankly misinformed so that they're just.

Coming from a place of not knowing the true state of reality on the ground, or they're malicious in that, in that they're designed to hold the United States back. And let me explain what I mean there, number one, is a very, I think, miss sort of informed understanding of the state of info information security at the best labs. Right? There's this idea we have that oh, the closed source labs are so protective and secretive of their, of their weights that, that China doesn't have them and we somehow have this amazing competitive advantage over China. For over 10 years now, the Chinese government has had a state sponsored program to infiltrate targets with a valuable IP development in the United States. It's not AI specific. This is in all kinds of industrial processes. That.

Is a nationally sponsored strategy by the government of China to exfiltrate valuable IP from the United States to China. And while the FBI and other enforcement agencies can't comment on ongoing investigations, I will tell you that you don't have to look too far to look for public evidence that this is already happening at the Frontier Labs. Just two months ago, there was an engineer from Google who was caught by the FBI boarding a plane to China with TPU schematics on a thumb drive. Oh wow. We're not talking sophisticated exfiltration here guys. Thumb drive. Okay, so I would be number one. I think we, any national security game theory that, that, that folks are binding on should just must take into account the reasonable likelihood that Frontier Model Labs in the United States are already infiltrated by adversarial nation states. Frankly, I think there's good evidence already from our enforcement agencies that ongoing investigations that will soon become public will, will make that clear. But, but you, you just have to go read the news to know that this is happening. So, so number one, any, any national security strategy that says oh, we're ahead and they can't get our weights from closed source labs is you've already given away the game. Okay, so let's start from an operating assumption that at best we are at par with them where they have our Frontier developments today.

I'm not even sure we can claim we're ahead. Let's just say the goal is to remain at parity.

Right?

The idea that open source is somehow going to give away our national competitiveness fails to take into account that the way we got to the Frontier in the first place was through collaboration between researchers of different labs. And the current big tech argument that oh Open sourcing our weights will allow adversarial countries to get them does one thing and one thing only. It allows them to stop having to publish their research, stop having to have a convenient excuse to tell their best researchers who want to, by the way, publish their research. The way the best AI researchers.

Get more sort of feedback on their research is by presenting openly.

The scientific process is you put out your research, you share about it publicly, other people then provide feedback, and then you improve.

That entire process of open collaboration at the frontier of AI is about to basically be all but dead. And one of the biggest ways that we have shot ourselves in the foot is by preventing academia in the United States to contribute to that research.

Open source, for example, today is the only way that allows Frontier University labs to. To contribute to research at all. If llama 3 was not open sourced, if Mistral was not open sourced, Stanford, Berkeley, MIT.

These institutions, the postdocs, the PhDs, there would have zero way of contributing to AI research. And so I think if you believe that the public university system and open collaboration between labs is critical to keeping our national competitiveness ahead, then turning off open source is a great way to keep us behind, right? Especially at a time when our labs are already infiltrated. So if the enemy already has our best and then we're slowing down our best.

The most likely steady state is that we lose our national competitiveness and we fall behind. Right? So I think that either people don't know just how people advocating for these who are arguing that open source is bad for national security, frankly, a lot of them just don't know what they're talking about because I don't think they're investors in enough Frontier Labs at this point. And frankly, I think there's a bunch of people who are culturally misguided because they think that these doomsday scenarios are more realistic than they really are.

So I have one sort of last question about the SB 1047. It's a little bit of like a devil's advocate question. So if this is like a California bill, you know, wouldn't people just argue, well, just go do your research outside of California. Like, I don't know, just. I'm curious your thoughts on that.

Yeah, it's a good question. So, unfortunately, this, the drafting of the bill was amended to be even more clear that the bill stretches across state lines. You know, up until last week, there was some debate like, oh, Ange, like, it doesn't say that this applies outside of California. They, the bill's authors went outside out of Their way to make it clear that the statute would reach across state borders.

Oh my God.

So really, I mean, I mean, I believe the bill's authors has been actually promoting that as a feature of the bill, not a bug. So this, this legislation is nationwide, whether we like it or not, or they're proposing it to be nationwide. So the only, I think the most likely scenario will be that our best researchers, our best teams will move offshore to this emerging kind of region across the world that I'm calling an AI sanctuary. Basically, there's sort of three things you need now as a world class startup or a model research team. You need cheap electricity, right? Cheap, abundant, sort of sustainable, clean electricity to run the massive amounts of compute you need to train these models. And the last thing is you need regulatory.

Certainty and protection to train these models where you're not being, as a researcher, you're not being held with civil and criminal liabilities. And you know what, Frankly, I have been shocked by how many.

Nations have reached out since we started.

Publicly speaking about the bill, saying, hey, please send us your best and brightest. We will gladly protect them with our regulations. And I think that will mean that our best companies do offshore to places that are offering them cheap and abundant energy, compute and regulatory protection.

Yeah. So let's talk about what people can do. If you know, you're listening to this and you're going, okay, yeah, this, this definitely sounds like we need to stop this from happening. What can we do about it?

Yeah, I'm glad you asked. So stop sb1047.com it's a public website that's a hub for researchers, academics and anybody else concerned about the impact of the bill can go and write to their legislators. So if you oppose the bill, please visit the site. We've got a templatized bill that you can then customize for yourself and send it to your assembly representative. We have a list there where you can easily pick who your representative is. You know, we released the website last week and in the first four days we had the community send over 375 letters to the Assembly. And so this is an important issue that a lot of people, a lot of startups, a lot of academics and a lot of open source researchers are concerned about. But we need to get the word out to even more people. We have less than 60 days before the final assembly vote on this proposed law. So please tell others about the site, share the information, raise awareness among those who will be impacted by this bill. We basically think little tech deserves to have its Voices heard. And so if you visit the website, we make it super simple for you to understand how this bill impacts you, if you're little tech and how to take action, which is to send a letter to your representative and make your voice heard.

You know, the message about helping small startups, like I personally feel that, but I think a lot of people are going to resonate more with the fact that this is very important for the future of America. You know, it's kind of appropriate that this is like right after July 4, Independence Day, celebrate, we're celebrating America. And I personally, that's, that's the thing that inspires me is like this is very important. Like if we just like hand AI dominance to China, like the reason America's been so successful is that we were, you know, we were the leaders in culture for a long time with like entertainment, we were leaders in technology, Internet. And so that's why freedom is spread around the world. And if we don't win an AI, it's probably gonna be the opposite of freedom that's spreading around the world through China. And so I think it's a big problem. And like, and the opposite of freedom when you have AI is, can be quite scary when you can use AI to mass control people. And so I believe that it's really important that America wins this. And I think that more people probably will resonate with that kind of message versus, you know, obviously you still kind of. Other people we like, yeah, we want to support the startups. Right. But a lot of other people, I think that's a more powerful message.

I think, I think you're absolutely right. And there's a professor at Berkeley, Ion Stoica, he testified in Sacramento last week saying that this bill, while this bill is called safe and secure innovation for Frontier Artificial Intelligence models, in its current form, it would do the opposite. It will hurt the innovation in California and it will result in a more dangerous rather than a safer world. And he goes on to explain that. First, if SB 1047 passes when it comes to open source models, he predicts that within one year we will all use open source models developed overseas, likely in China. Why? Because this law will discourage building open source models in California and likely the United States and Chinese open source models are already very competitive and three of the top six open source models are already Chinese, according to the Berkeley LMSIS Chatbot arena evaluation. The second is that if SB 1047 passes, then California will lose its competitive edge when it comes to AI, because as a researcher in a fast moving field, you don't want to be constrained by such limitations. So you just go elsewhere when you can do your best research. So more and more PhD students of Chinese origins will just go back to China, while others might consider going to places like other adversarial countries so where they can enjoy huge funding for their research. And this is already happening, according to him. And he's a leading academic at one of the preeminent American university labs. And so when he's saying it, we really have to, I think, sit up and pay attention. And then the last thing he did talk about is how SB 1047 incentivizes companies that sell to enterprises to move out of California, since most of their customers and enterprises already have headquarters out of California. And so for the California market, they could just provide, they will have to basically provide inferior models to conform with SB 1047, which will mean that the state will turn from a leader to a laggard. And that's not a future any of us want. And so you're right, Nathan, that this is not just a California issue, this is an America issue. And I don't think enough people across the US realize just how dangerous this piece of legislation is for all of America. And I think more people should be talking about, about, about it the way you are.

I think on that note, that's probably how we'll, we'll wrap up the episode, but everybody can head over to stopsb1047.com to get more details about the bill as well as more details about how to help prevent this bill from actually getting passed. And Anjaney, thank you so much for hanging out with us today. This has been a fascinating discussion and I think it's really going to open a lot of people's eyes. I don't think a lot of people even realize that this is kind of happening behind the scenes. It doesn't seem like it's getting a lot of publicity right now. So I think it's important that we have these discussions and let people know that this is happening. This is what the California government's shooting for. So if you like what we're getting out of AI right now and you like the progress we've seen, we need to do something about this. So I appreciate you sharing all your thoughts and all the details about this because I do think it's going to be eye opening to a lot of people.

Oh, thank you guys. I'm a huge fan of the BOT and you know, helping us spread the word and, and, and get the message about the cause out is. Is deeply appreciated. So thank you.

Absolutely amazing. Thanks again.

All right, thanks, guys.