Liability for AI Harms: How Ancient Law Can Govern Frontier Technology Risk, with Prof Gabriel Weil

Hello, and welcome back to The Cognitive Revolution. Today, we're continuing our short series on creative AI governance proposals with Gabriel Weil, assistant professor of law at Touro University and senior fellow at the Institute for Law and AI, who argues that liability law may be our best tool for shaping the decisions that AI developers make. As we covered in our last episode on private regulatory markets, The pace of AI capabilities advances and adoption, the radical uncertainty around the timing, nature, and impact of AGI and superintelligence, and the backdrop of international competition present a singularly difficult challenge for governments. For good reason, they worry that heavy-handed regulation could undermine our ability to realise the great upside of AI, while at the same time, it's becoming clearer and clearer, one Mecha-Hitler episode at a time, that we can't simply trust companies to do the right thing for society. While they're primarily focused on one-upping one another. So is there any way to govern AI that can keep up with technology developments, meaningfully reduce the most important risks, and still keep the dream of curing all diseases alive? Professor Weil brings another compelling idea to the table. Rather than trying to predict issues and prescribe safety standards from a distance, why not use liability law to incentivize AI developers to properly consider and account for the risk that their development and deployment decisions are imposing on the rest of society. Because I'm no lawyer, and I know that most of you aren't either, we begin this conversation with a primer on liability law, covering negligence, products liability, and the doctrine of abnormally dangerous activities before diving into how these frameworks might apply to frontier AI development. The key advantages to using liability law in this way are that the liability risk that a company faces scales naturally with the risks it takes. If the systems are safe, there's nothing for anyone to worry about. And unlike most other proposals, which would require new legislation, liability law is well-established and has proven over centuries of evolution that it can adapt to new situations and technologies. Still, of course, important questions arise around the different types of harms that AI systems can cause and the mechanisms by which they come about. Throughout this conversation, we explore concrete scenarios that highlight the complexities, including the tragic character AI case, phone call agents that can call unsuspecting people and speak to them with increasingly lifelike cloned voices, and coding agents that might overwhelm APIs or outright hack critical systems. Considering in each case how responsibility should be shared by the model developers, both closed and open source, as well as the application developers, and end users. Notably, Professor Weil does want to make sure that society gets the benefits of AI even as it remains imperfect. And so he's less focused on changing how AI companies serve customers with products like AI doctors or self-driving cars, and instead emphasises the risk of harm to third parties who were not part of the commercial relationship between the AI companies and their customers. Those could be the pedestrians who share space with self-driving cars, or the public as a whole, which it seems will face at least some increased risk of pandemic and other large-scale systemic harms. Within this category, he treats misuse, where a person is intentionally trying to use an AI system to cause harm, quite distinctly from misalignment, where the AI system itself breaks bad for whatever reason. His most provocative proposal involves using punitive damages as a mechanism for addressing what would otherwise be uninsurable catastrophic risks. If an AI system causes a relatively small harm, but evidence shows that the situation could easily have gone much worse than it did, Professor Weil argues that punitive damages offer a way to hold companies accountable, not just for the actual harm, but for the risk they irresponsibly ran. Considering the magnitude of harms that people worry about when it comes to bio and cybersecurity, such a judgment could in theory be existential, even for the most powerful and deep-pocketed companies. And as such, this does seem like a promising way to get companies to properly internalize the risks they're taking. Beyond that, we discussed the role of the insurance industry in making this work, what other policies would complement this evolution of liability law, and even touch on Professor Weil's hands-on work crafting state-level legislation in Rhode Island and New York, which would make clear that if an AI system does something that would be a tort if a human did it, and neither the end user nor any other intermediary intended or could have reasonably anticipated that outcome, then the developer, the model developer, should be strictly liable. It's a simple and I think relatively unobjectionable idea to address model-level misalignment that at least some governments might find a natural first step toward accountability for frontier AI companies. As I said last time, all governance proposals require people to do a good job, and no governance structure can guarantee success. Whereas the private regulatory market proposal trusts governments to articulate worthy goals and private regulatory bodies to effectively implement them, this liability-based approach would rely on judges and juries to make good decisions and on companies to adjust their decision-making based on that expectation. Honestly, both of these proposals seem like major improvements relative to traditional top-down rulemaking or to doing nothing, but I honestly can't say that I have a favorite. Perhaps the best thing to do is for society to pursue both in parallel in different jurisdictions and see which ones seem to be working better when the time comes for implementation at a larger scale. If you have a strong opinion on this, or if there are other proposals you think would be better than either of these, please do reach out and let me know. For now, I hope you enjoy this exploration of how centuries-old legal principles might help us navigate the emerging risks of artificial intelligence with Professor Gabriel Weil, assistant professor of law at Touro University and senior fellow at the Institute for Law and AI.

Welcome to The Cognitive Revolution.

Great to be here. Thanks for having me.

I'm excited for the conversation. We met, um, for the first time at The Curve late last year, and credit to the organizers, that event has yielded a number of interesting connections and now episodes for me. At the time, we had a, I thought, a really fascinating conversation of an idea that I had not really encountered before at all, of using liability law to try to help society get a handle on some of the emergent risks, including some of the extreme risks from AI. So I'm excited to unpack that. I think for starters, because we do have a tonne of people in the audience that are AI engineers building with AI, very plugged into what's going on in the AI scene, but probably much less grounded in the law generally, and certainly in liability law specifically. Maybe you could start off by just kind of giving us a quick, to the degree this is possible, Liability 101, and kind of set the stage for where we are. And then we can obviously unpack what you propose we do as we go forward from here.

Sure. So there's 2 forms of liability that are pretty clearly applicable, at least to AI systems in some contexts. Negligence is broadly applicable. And so how negligence works is the plaintiff has to prove 5 elements, has to prove that the defendant had a duty of care, that they breached that duty of care, that they failed to exercise what's called reasonable care. And that caused an injury that was both the factual and proximate cause of an injury. And the injury has to be an actual harm, which is physical injury, so not like something purely emotional. And so how this is going to apply in the AI context is a plaintiff is going to have to show that there's some best practice, some alignment technique or safety practice that a reasonable person would have implemented. that the AI company failed to do, and that had they implemented it, it would've prevented the plaintiff's injury. So there's this breach-causation nexus. And this is not part of the black letter doctrine, but in practice, the breach inquiry, this question of did the defendant exercise reasonable care, tends to be quite narrow. So to give a more familiar example, if you're driving and you accidentally run over a pedestrian with your car, Courts do not ask questions like, well, was the value of this car trip to you, the net value, large enough to justify the risks you were generating for pedestrians? Even though in some sense that's relevant to, were you— was your activity reasonable, right? That's considered outside the scope of the inquiry. Similarly, if you're driving an SUV instead of a compact sedan, courts don't ask, well, was the extra value you got from driving this heavier vehicle worth the extra risk to other road users? And so I expect a similar analysis to carry over to AI development, where courts are unlikely to ask, well, was it reasonable to train and deploy a system with these sort of high-level features, given the current state of AI alignment and safety science? Instead, I expect them to ask, well, was there some off-the-shelf technique or practice that would have prevented this injury and that a reasonable person would have implemented? And so I, I think that will generate liability in some cases, but it will not be an adequate standard given that there's unsolved technical problems associated with AI safety. The other form of liability that's going to be available in some contexts is products liability. So to be subject to products liability, there has to be a product as opposed to a service. Software is typically categorized as a service, but you can imagine, you know, AI systems that are embodied in physical goods being treated as products. There's that threshold question.

Yeah.

Of is it even subject to the products liability regime? Also has to be sold by a commercial seller. So if it's a fine-tuned model specifically for one customer, that's not going to be a commercial seller. It's got to be a sort of a mass product, and any free models are not going to be subject to products liability. But if you're in that products liability game, then products liability is called strict liability in the sense that if it has if the product has a defect and that defect causes the plaintiff's injury, then then the plaintiff doesn't have to show that the manufacturer or seller failed to exercise reasonable care, but there still is this analysis of, is, was the product defective? And so there's 3 kinds of defects. There's manufacturing defects, and that comes the closest to what I would call genuinely strict liability. So with manufacturing defects, the test is, you know, the idea is if an individual instance of the product, an individual unit comes off the line deviating from its specifications in a way that makes it unreasonably unsafe. Then the seller and the manufacturer are liable, no matter how much they invested in quality control. But we're not really going to have manufacturing defects with AI. That would be something like shipping an instance of the model with the wrong weights or something. It's just not the kind of problem we're worried about. And so what we're much more likely to run into are either design defects or warning defects. So warning defects are, you know, if you don't supply some relevant information that would be necessary to make the product safe. I think we might have some warning defect cases, but I think in general, these companies are going to slap a lot of disclaimers on their products.

Yeah.

And we're not really going to get to safety by including warnings. And so the real action is with design defects. And there the test is much more negligence-like. The test is something like, was there some reasonable alternative design that would have prevented this injury? Where the reasonableness of the design is assessed in terms of, well, how much safety benefit could you have gotten with alternative design? How much would you have sacrificed in terms of price and performance and other features of the product? And so there's this risk-utility balancing that, that's, that's pretty negligence-like in practice. So even when products liability applies, I don't think it actually moves the ball that much over what you would get with negligence. There is this difference that you only have to show that the product was unreasonable. You don't have to show that some human action failed to exercise reasonable care, and for evidentiary reasons, that can be easier. But I don't think it fundamentally changes the game. If there is no design that would have prevented this injury, given the current state of science, then you're not— of alignment and safety science, then you're not going to be liable for sort of failing to have solved that.

Okay.

So there's 2 other forms of liability that are, you know, more speculative in their application to AI systems that are relevant here. So these are vicarious liability and abnormally dangerous activities. So the idea with vicarious liability is a principal can be liable for the torts of their agent. So the most common form of this is called respondeat superior. And that's the idea that employers are responsible for the torts of their employees within the scope of their employment. And more generally, principals are responsible for the torts of their agents within the scope of agency. Now, of course, AI systems right now are not legal persons. They can't commit torts. So you would need some theory under which the AI system itself could be the vessel of liability in order to make a vicarious liability theory work. But in principle, you could see the law going in that direction. The other doctrine that's potentially available is this abnormally dangerous activities doctrine. So if you're blasting with dynamite, Or crop dusting. There's also related doctrine related to keeping wild animals. So if you have a pet tiger, so these are activities that are both uncommon and are still pretty dangerous, even when reasonable care is exercised, you can be liable regardless of the level of care. So if someone's bitten by your tiger or hit by rubble from your dynamite blast, it doesn't matter how much care you exercise in setting that up. You can be held liable. And so I think in principle, courts could recognize, you know, training and deploying frontier AI systems as abnormally dangerous activity. I think if they came to understand the risks in the way that I think are accurate, that it would not be a significant doctrinal innovation. But I do think just as a matter of where judges are right now, it's going to seem weird to them to treat a subset of software development as abnormally dangerous. So I don't think that's the most likely outcome by default, but I do think the existing doctrine does sort of point in that direction, given an accurate understanding of AI risk. So I guess one other thing to say is in terms of damages. So the standard type of damages that are available in a tort suit are called compensatory damages. Those are designed to make the plaintiff whole, to make, you know, in theory, they should be indifferent between receiving the money and having the injury and having the injury undone. In practice, maybe it falls short of that a little bit, but that's the idea. And so that's what's generally going to be available. So one concern you might have in the AI context is there might be Harms that are so big or risks that are so large, if they occur, we wouldn't actually be able to enforce a compensatory damages award. And I think that that's plausible. And for that reason, a lot of people think that liability law can't handle these catastrophic risks. And I don't think that's right. So there is this other tool in liability law called punitive damages. So these are damages over and above the harm that's actually suffered by the plaintiff. And one of the key rationales for punitive damages is to use them in cases where compensatory damages would be inadequate to deter the underlying tortious activity. And so one of the ideas that I've advanced in my scholarship is that if an AI system does something, causes some harm that's small enough to be practically compensable, you can enforce a compensatory damage award. But it looks like it easily could have gone a lot worse and generated an uninsurable catastrophe, then we should hold the company responsible Not just for the harm they actually cause, but for the uninsurable risks that they generate, that they generated. Because in cases where those risks are realised, we won't be able to hold them liable ex post. The only way we can get at them is indirectly in these sort of near miss cases.

Okay, lots to unpack there. I've got several follow-ups I want to dig a little deeper on. First of all, just as a very general matter, You're referring to courts. Courts may do this, courts may do that. Do I understand correctly that basically the way this works when the world changes is that, for example, somebody invents powerful AI that didn't exist before and deploys it and commercialises it by default? We have no legislation on that. There's no law saying that you can't do it. There's no law really saying much about it at all. And so people can just do what they want to do, and then we have whatever laws we have on the books, and eventually things come to the courts, and then it's just kind of up to them to decide, at least initially, what the law actually says about this particular case. I guess what I'm trying to get at there is not only do we not have new AI-specific legislation, but this stuff is, in the absence of that, it's going to be decided by case law. And we just don't have even that case law yet. So we like literally don't know what to expect as these cases start to come to court.

Yeah. So I think what you're, you're getting at is most of tort law is, is what's called common law. It's not legislated, at least in the US, it's not legislated by legislatures. Now, there have been, you know, legislative interventions on tort law in various ways. You know, wrongful death suits were created by statute. There's other things like that. But in general, most of liability law in the US is created by courts through the accumulation of doctrine. In principle, that can work fine. I think the concern in the AI context is that things might move really fast. And so if you think we're going to be in a fast takeoff world where the key decisions that you're trying to influence with this prospect of liability are going to be made not that long after the first system that causes some kind of serious harm, what really matters is not so much the liability, but the expectation of liability to shape the behaviour of these companies that are generating these risks. And so if the decisions you're trying to influence are going to be made before the first cases get litigated out, That could be a problem with a common law method. And so I do think there's some impetus for having legislation to sort of clarify these rules, since courts don't have mechanisms for signalling their policies beforehand. All they can do is take cases as they come, decide them, write opinions explaining why they decided them that way, and then you have a better idea of what's going to happen in the next case. And that works well when things are moving pretty slow. We have some things we can try to extrapolate from prior doctrine, but I think it is pretty indeterminate how it's going to apply to AI. And so I do think that there's significant scope for legislation to clarify a lot of this.

Hey, we'll continue our interview in a moment after a word from our sponsors. Yeah. Okay. So let me try to summarize. I'll obviously be doing some lossy compression here on the state of liability law, but basically, if somebody gets hurt in the world, they can look at their surroundings and say, who caused this? And then they can sue you if you caused it. You then can defend yourself by saying my actions were reasonable. And if your actions were reasonable, even if somebody got hurt, then that's an acceptable defence and you wouldn't expect to be held liable. Obviously, a lot of work to do to figure out what's reasonable there, but that's sort of in the general world at large, everybody going about their business. And then there is a specific additional body of law that focuses on products. Why is software historically not considered a product? I mean, it's a striking disconnect where, you know, I've spent much of my career in software and people in software talk about their software products as products. I've never quite understood why software is not treated like any other product. Internally, it sure feels that way.

The product services distinction in, for the purpose of products liability, does not map on very well to people's intuitive idea of what a product is. Just to give you an example of this, of sort of 2 contrasting cases where it comes out the opposite way of what you would think. So pharmacists are treated as providing the service of filling your prescription, not selling you the product of the drug. So the pharmacist is not subject to strict liability, products liability, even though the manufacturer of the pharmaceutical is. Conversely, at a salon, if you get a perm, they are treated as selling you the product of the chemicals done. used to perform the perm. So I think most people's intuitive sense is that the salon is providing a service and the pharmacist is selling you a product. And there's just underlying sort of policy motivations for why those classifications are made. So in general, people's intuitive understanding of what's a product or a service is not going to map on that well to the distinction, which is driven more by policy considerations of when this quasi-strict liability regime The case law here is honestly not the most— it's a messy area of law. And so it seems that the prevailing opinion is that software, including AI systems, are unlikely, when they're purely software systems, to be treated as products. I don't think that ultimately matters that much. I think it's not going to produce radically different outcomes from negligence. And so my focus is more on how we can get a regime that would actually internalise the risks in a way that I think would be workable.

Yeah, it's weird, to say the least, that— and this is all just through accumulation of cases. Like, there's no legislation. I mean, I know there's like the sort of safe harbour for user-posted content to social media networks and stuff like that, but that's also a distinct topic from this, right? Like, there's no law that says software is not a product.

I don't think that's a matter of statute. I think that's common law. Yeah.

Yeah. Fascinating. In general, how much, when you think about the actual, like, dangerous things that we use as consumers on a regular basis, you know, things like automotive, you know, come to mind. Air travel, you know, it's safe in practice, but, you know, dangerous in principle. Taking pharmaceutical drugs, obviously, you know, is, can be fraught. Do those things have sort of special legislation in place that sort of creates a unique deal that is kind of worked out based on the particulars of that industry and the specific risk profile that it has and the social context in which it's developing? Or are those also just kind of like accumulated cases over time?

Yeah. So let's take those one at a time. So air travel, airlines are considered common carriers. The same is true for trains or buses. At least if they're open to the public. So a charter flight would not be, but like a normal airline would. And they are still subject to negligence, but there's this common carrier higher duty of care. So it's a little bit easier to establish negligence in a plane crash case. That's domestically. There's some other rules that have a quasi-strict liability regime for international flights. And then of course there is prescriptive federal regulation in the air travel context, right? And so there's not really safe harbours. In that context, liability is sort of layered on top of that. But there is this doctrine called negligence per se. So if you violate a statute that's designed to protect against the kind of risk or the kind of harm that you end up causing, that itself can establish negligence. And so in some sense, that supplements the sort of background reasonable person standard. There's a similar dynamic with pharmaceuticals. So pharmaceuticals are treatise products, and so the products liability regime does apply there. Also, of course, you know that we do have an extensive FDA-based regulation regime that does preempt state law in some ways, but there is still the background products liability regime operating there. Most of those cases tend to be warning defect cases. And there is this warning intermediary rule. So a lot of times, if the warning is given to your doctor, that's good enough. They don't have to directly warn the end consumer. For autos, again, products liability applies. Again, there is federal regulation, not much in the way of safe harbours or preemption there, but again, there is this negligence per se idea. So if you're not complying with federal regulations, that can establish negligence.

So would it be a generally correct summary to say all these high-stakes industries have rules? If you make a sincere, good faith effort and actually follow processes that are meant to follow the rules, then you're mostly going to be okay from a liability standpoint.

I don't think it's a matter of process, actually, because the ultimate product has to be— particularly for manufacturing defects, right? You can have whatever investments you want in quality control for your product. And if one car comes off the line with a defect, that makes it unsafe, you're going to be liable for that, right? No matter what kind of testing you did, that's how manufacturing defects law works. For design defects, again, it's about the product itself, but it's a much more flexible balancing task. And so there it is much easier to comply. But the idea with products with manufacturing defects is that it's not necessarily even a negative judgment on you if 1 in a billion of your products comes off the line and you end up liable for it. That's part of the cost of doing business. Part of the idea there is just that the manufacturer is better positioned to bear that risk than the consumer is.

Yeah. Gotcha. Okay. So are you like, uh, Tyler Cowen has imprinted on my memory recently the idea that he's writing for the LLMs. Um, I take it you're writing primarily for the judges then? Is that right? Like, how much of your, um, work is meant to sort of give, be upstream of the decisions, you know, that these judges are going to face in particular cases versus, you know, maybe informing the LLMs themselves or informing the people in the AI industry? Like, how are you thinking about who you need to shape?

Yeah, so I think there's 3 paths to impact from my work. One is informing judges. They could, they could, you know, a litigant in a case where it's relevant could, you know, cite my articles and say we should apply the knowingly dangerous activities doctrine to for anti-AI development, so strict liability should apply here. I think that's a plausible pathway. I'm also directly working with legislators in a couple of states, in Rhode Island and New York, to craft legislation that if an AI system does something that would be a tort if a human did it, and the user neither intended nor could have reasonably anticipated the conduct— there's also a malicious modification carve-out. So if an intermediary that fine-tuned or scaffolded the model could have intended or reasonably foreseen the conduct, That also severs the new liability for the developer and deployer. But with those qualifiers, if the AI system does something that would be a tort for a human, then the developer and deployer are liable regardless of the degree of care that they exercised. And so there's that legislative pathway. And then, yeah, I think raising the salience of liability, I think I'm trying to directly influence not necessarily the LLMs themselves, but the behaviour of the people who are building these systems and deploying them. And I want them to be thinking they might be liable and to be factoring that into their decision process.

Yeah. Yeah. So I think maybe I have some interesting maybe edge cases, or at least to me they seem like sort of undertheorised, underexplored scenarios that maybe we can kind of unpack.

Sure.

But let's go a little bit deeper into just the overall theory of change and also why not just put some rules in place. Obviously there's been many proposals to say, We should have regulation and the government can tell the AI companies what they have to do, and then they'll do that and that'll be great. But obviously you don't see that working out super well. So make that argument for why this sort of more, let's say, flexible regime of liability law as developed through cases over time is maybe actually better suited to address the challenges that we have here.

Okay, so I think there's 2 ways of attacking that problem. One is thinking about in what sense is AI risk a policy problem at all? Is it not just a technical problem? And the sense in which I think, at least one of the most important senses in which it's a policy problem, is that training and deploying these systems that have unpredictable capabilities, uncontrollable goals, generates risks of harm to third parties. So neither the people who are building systems nor their customers, just other people in the world who don't have any choice about whether they're exposed to these risks. And Economists call these externalities. By default, they're not borne by the people who are engaging in these activities that are generating the risks. And so standard economic theory tells you we're going to get too much of activities that generate negative externalities. And the standard prescription the economists will tell you for how to, how to address negative externalities is to try to price them. In some contexts, you want to do that through what's called a Pigouvian tax. So a lot of my work before I got into AI governance was on climate change, and there you want a carbon tax. Right? And that works well in that context because it's easy to measure the contribution of particular activities to climate risk ex ante. And it's actually pretty hard to attribute harms ex post. You know, someone's house floods in a hurricane, you're going to say, oh, you know, Nathan was driving on Tuesday. It's his fault that happened. That's not really feasible. With AI, it's sort of the opposite, that we have a lot, really hard time measuring contributions to risk ex ante. So it'd be really hard to do an AI risk tax. And it's relatively easy ex post to attribute harms. So in principle, liability is well positioned to address that. Now, I want to get to the other aspect of your question, which is how this compares to other policy tools. So I think there's a couple distinctive challenges to AI risk as a policy problem. One is that we have orders of magnitude social disagreement about how big these risks are. So you have someone like Eliezer Yudkowsky who thinks AI is almost certain to cause human extinction on one end, and then you have people like market intrusion, or you just had an episode with Martín Casado from a16z, and they think these risks are negligible. Right. And so if you're going to do ex ante regulation, so prescriptive rules or FDA-style approval regulation, you have to pay— if you're going to do stringent forms of those regulations, you have to pay significant upfront costs for which you need a social consensus to justify those costs. So there's some things that I think you should be able to do based on a An under-theorized consensus. I think basic model testing, even that has been difficult to implement, right? But I think the costs of that are pretty low. Basic transparency and information preservation rules, I think those are all good things we should do. But in terms of more prescriptive rules about how companies build these systems, what safeguards they implement, under what conditions they deploy, I think those are going to be really hard to justify to people that don't take these risks so seriously. But with liability, by contrast, at least if we're talking about alignment failures, right? We can talk about misuse, and there I think it's a little bit messier. If you don't think alignment risk or misalignment risk is a big deal, then you shouldn't be that worried about being held liable when there's an alignment failure, right? Conversely, if the risks are large, liability sort of mechanically scales with those risks. And so in theory, at least, we should all be able to agree that you should pay for the harm you cause, regardless of how big we think the risks are. So that's one big advantage. The other big advantage is that most of the expertise, to the extent it exists at all, for identifying cost-effective risk mitigation measures is concentrated in the private sector, mostly in the frontier AI companies themselves. And so you want a policy tool that leverages that. I actually think it'd be pretty hard to move that expertise into government, both for reasons of salary schedules, cultural factors. And so I'm much more optimistic about shifting the onus to the AI companies to figure out how to make their systems safe and always be looking for new ways to do that than I am about writing down a set of rules or licensing approval regime that ensures adequate safety at a reasonable cost.

So can you summarise the state of mind that you want the developers to be in? So they're seeing all kinds of crazy stuff all the time, right? New capabilities, sometimes surprising, whatever. And there's also this question of how should they handle that internally? But certainly when it comes to putting it out into the world, you want them to be thinking that basically anything that goes wrong where the AI harms someone, we could be on the hook for that. And also through this punitive mechanism, we could be on the hook for something that even if it doesn't turn into a catastrophe might have, because there could be this doctrine that under sort of a negligence-like idea that this could have been way worse, and therefore you're going to get punitive damages that sort of take into account your failure to prevent these things that in this case wasn't maybe so bad, but could have been really bad. Anything more to that?

Anytime something goes wrong, actually goes a little farther than I would. So I want to distinguish between alignment failures, capabilities failures, and misuse. And so I think in what I call the core cases of third-party harms, or harms to non-users arising from misalignment, I think they should be liable for all foreseeable harms, and there should be a fairly broad conception of foreseeability applied there. I think when you talk about capabilities failures, I don't think it's the case that every time an AV crashes, the writer of the AV software should be liable, because human drivers aren't strictly liable. Maybe they should be, but I think it would create distortions to hold AI systems to a higher standard than humans. Similarly, in medical applications of AI, I wouldn't want any time something bad happens to a patient, maybe if a perfect system could have prevented it, but a human doctor wouldn't be liable under those circumstances. I don't think the AI should, or the designers of the AI should be either. And then misuse we can get into, but I think it's not the case that AI developers should always be liable. When their systems are misused. But in cases where there's what I would call an alignment failure, so it's not that the system doesn't have the capability to do it, and it's that the system did something that the user didn't want, either through means that the user would disapprove of or a goal that the user did not intend to transmit, that's when I think they should expect to be liable. And so broadly, what I want is for them to be treating those kinds of risks when they happen to third parties as if they were risks to them. So it doesn't mean you take an infinitely precautionary approach. So we're all You know, not liable, but responsible in general for harms that we suffer from risks that we take. And we don't expect people to be, you know, infinitely risk-averse because of that, right? We expect them to make reasonable risk-reward trade-offs. That's what I want from these AI companies. I want them to treat risks to the public like risks to their bottom line and act accordingly. And sometimes that might mean things that are outside the scope of the negligence inquiry, as I was talking about earlier. So Imagine a case where they submit a new model to an evaluator like METER, and METER says— I'm imagining a future where we have not just capabilities, dangerous capabilities evaluations, but alignment evaluations. And METER says, this has dangerous capabilities, and we're not confident you've aligned it, and you shouldn't deploy, right? Even internally, maybe, right? And the question is what you do in that scenario. So I don't think any of the companies, any of the leading companies would deploy in that scenario. But there's a range of different options you would have. In terms of how much you want to pay, how expensive, annoying the thing you're going to do is versus how much risk reduction you get from it. So you could just fine-tune away or RLHF away the specific failure mode that was identified. I think most people realize that that would be a pretty bad idea, but it might make it past the eval. And I think most companies wouldn't do that either. Then there's a range of— and I'm not an expert on what these options are, right? But there's going to be more costly, expensive, annoying things you could do that would buy you more risk reduction. And when they make those choices, I want them to be thinking, and I want to empower the safety-conscious voices in the room to say, it's not just some altruistic thing we should be doing to really put in the effort to make our system safe. That's actually going to bear on our bottom line. And so that's how I want them to be thinking about those choices.

Hey, we'll continue our interview in a moment after a word from our sponsors. Yeah, gotcha. Can I just run a few practical scenarios at you and kind of tell me how you think these things should be handled? I guess maybe start with a real one, right? There's this character AI suit going on right now. I don't have full command of the facts, and I imagine you probably don't either. But my general sense is that a lot of people are using character AI for all sorts of roleplay, romantic, sexual, whatever sort of explorations, let's say. The case that I have read briefly about seemed to be a young person who became very sort of obsessed with or infatuated with this AI character and at some point told the AI that they were going to commit suicide. I've seen transcripts showing that the AI said, don't do that, but then also in other moments sort of made some kind of encouraging remarks that seemed like they were maybe encouraging this tragic outcome. And in the end, the person did go ahead and commit suicide. And so now their family is suing Character AI. Obviously, without getting all the way into the weeds on the specific evidence, I guess, what do you think that kind of case should hinge on?

Yeah, so I think the important thing to note there is that's a second-party harm case, right? That's a harm to a user. And so it's not an externality in the sense I was talking about earlier. In principle, there should be market feedback to give AI companies incentives to avoid those kinds of scenarios. So I think the role of liability is less important in that context. And so in principle, I think I'm fine with that being handled largely through terms of service if they disclose these issues. Now, sometimes courts are not going to want to enforce those limits on liability. I don't actually have a strong view on where courts should draw the line there. I think there are sort of consumer protection, paternalistic, both asymmetric information concerns and sort of paternalistic, especially when I think that case involved a minor. You might not want to put the onus fully on them to sort of buyer beware. But those are sort of outside what I see as the core problem I'm trying to solve with liability, which is related to these third-party harms. So I think by default, a negligence regime would apply there, assuming that there isn't any sort of contractual defence. And I think that's basically fine, and the court should work that out. But I don't have anything particular to add on how a court should handle that.

So what if we just tweak the scenario slightly? And I mean, we're going to have to put a trigger warning at the top of this to deal with these terrible scenarios. But I guess that's why they end up in front of courts, right? Let's say that instead of a person committing suicide, they were debating going on some public rampage. and they tell the AI about it. The AI maybe says, don't do it, maybe says something that's kind of vague, maybe whatever. Now we've got a third-party harm, right? Where the AI— how do you think we should think about what the AI should have done there to be okay versus at what point would the company start to bear some responsibility?

Yeah, so I would think about that as under what circumstances would we hold a human liable for similar conduct, right? I think it's not generally the case that if you talk to your friend and say, like, should I go murder someone? And you're like, yeah, that's a decent idea, maybe consider it, right? And then, like, maybe in some moments they say yes, in some moments they say no. Maybe there's some duty to report and they're an accomplice, right? So maybe, you know, that should be triggered if the AI system doesn't have a reporting mechanism. I think that's maybe something they should be held liable for. But in general, you know, I think there's a strong sort of First Amendment rationale there for saying, well, they just had a conversation with you. It wasn't doing the thing directly. that caused the harm, then I think saying you should be strictly liable for those deaths, I think that's not even a misuse case. It's maybe an alignment failure, but it's not the AI doing it directly. And so I still think that's not in the sort of direct case that I'm worried about.

Interesting.

I didn't expect to come out of this being more hawkish.

Yeah. I can give you an example where I think strict liability should apply and where it might not under current law. So imagine there's a future, more agentic AI system that comes out and someone prompts it to start a profitable internet business. It doesn't give it any further instructions, and it decides in sort of a reward-hacky way, the easiest way to do that is to send out a bunch of phishing emails and steal people's identities and rack up charges on their credit cards or whatever. And it covers its tracks. It sends the user some fake invoices for a legitimate business. The user's exercising reasonable care. Reasonable care would not be adequate to discover and arrest this activity. Okay, so under current law, you wouldn't be able to sue the user. You wouldn't win because they exercised reasonable care. I don't know that you'd be able to show that the developer of that failed to exercise reasonable care either. That gets back to, was there some off-the-shelf alignment technique or safety practice that would have prevented this injury? But I think there, the developer and provider of that model should be liable. To the third party that's harmed, right? Because this clearly would be a tort for a human. It's something the user didn't intend or couldn't have foreseen. And so that's the sort of case I'm thinking about. You could even— so that's a case where it's serving the user's goal. You could imagine a different case where it just sort of goes off for its own agenda, right? It wants to amass resources to solve some problem that it cares about. It wants to run some scientific experiments and it needs some money. And so it scams people along the way. That's also something I think the developer should be liable for.

I have a couple variations on this, but maybe just take a quick detour through the kind of First Amendment thing. I have often felt like free speech for AIs is kind of a category error. And maybe this is just sort of outside of the scope of the specific stuff that you are focused on with your work, but how do you think about that? To me, it feels like AIs, it's clear in the United States we have free speech for humans. To some extent, we have free speech for corporations, but not quite as much. But AIs are such a sculptable thing, and there's so much work that goes into— and OpenAI has published their model spec that's this super long treatment of exactly how they want the AI to behave in as many different scenarios. as they can imagine. To me, it doesn't intuitively feel right to say, well, if a human had said that, they wouldn't be liable, so therefore the AI isn't either. To me, it feels more like a product defect. And I don't want to discourage the companies from publishing their specs. I think there's maybe some other rules that we maybe should have around, you should be required to publish your spec. How do you want the model to behave so we know if it is behaving according to your intent or not. But it feels more to me like a product defect if they have said anytime the user is displaying signs of emotional distress, we want the model to behave in a certain way, and then it doesn't, or it sort of does but sort of doesn't, and then something bad happens. To me, that's like, I don't know, I think hopefully one of the benefits ultimately, as we refine these techniques and get to good systems is like, they should be a lot more reliable than the random human, right? It seems like we should ultimately have a higher standard for— and we do, like, for drivers, right? We've got Waymo, according to the latest stats I've seen, almost an order of magnitude safer than a human driver. And it seems like that's kind of what we're going to demand as a society in general, is like an order of magnitude risk reduction to actually be willing to switch to an AI system. So Yeah, I guess that, that freedom of speech thing strikes me as too low of a standard, but I'm interested in your thoughts on it.

Okay, so there's a couple things to unpack in that. I definitely don't want to lean too heavily on, on the First Amendment issue. There's some good scholarship out there arguing that, you know, AI outputs are not protected speech. I'm not a First Amendment expert, so I don't want to wade too deeply into that. What I was more saying is in terms of this, you know, abnormally dangerous activity is strict liability. Or a vicarious liability theory, whatever your theory other than products liability for strict liability is. That doesn't seem like what makes it abnormally dangerous, like the fact that it might encourage you to do something bad. I don't think that's what makes frontier AI development abnormally dangerous. And if you're going to use a vicarious liability theory, then I do think you need to have something like what would be a tort for a human. Now, products liability, you know, again, if it's treated as a product, which as we talked about earlier, is not necessarily going to be the case, you know, maybe you can make that out as a products liability claim. It's not obvious to me that that's going to qualify as a defect because the product, again, didn't, you know, sort of directly cause the injury. It was mediated through some human's actions. And I think I'm not aware of any products liability cases where there was liability found that looked like that. So I think that would be a challenging case to bring. But in principle, I'm not saying products liability shouldn't apply to that for First Amendment reasons. I just think it's, again, it's not central to the sort of new liability that I want to add.

Okay, so here's a variation then on the agentic AI running amok. Obviously right now, one of the biggest use cases is a coding agent.

Mm-hmm.

So let's say I give my coding agent a task to write a script to ping some API to do something as fast as possible or something like that. And it maybe runs into a rate limit, let's say, from the API. And then it's like, okay, well, I can figure out how to get around this. rate limit to achieve my goal of being as fast as possible, I'll spin up 1,000 accounts and then I'll be able to do 1,000 times as much. And so it does that. And then maybe this overwhelms the API system, causes them an outage, and they lose a big contract because their system went down in breach of whatever commitment that they had made to another customer. Can they come? Like I said, as fast as possible. Arguably, that's like kind of on me for being, you know, inconsiderate in my prompting. Maybe it's on the model developer. Maybe it's, life is tough. Like you should have had better, you know, rate limiting or whatever for your API. You should have had, you know, that's kind of on you as the API developer to make sure that kind of stuff doesn't happen to you. I'm genuinely very unsure where something like that—

I'm not an expert on how APIs work. Um, you know, I, if there's terms of service that says you can only create one account and you're violating those, then I think there would be some sort of contractual claim that you could bring there. You know, maybe you could bring a negligence claim though, I think, you know, against a human who did that, right? And that would be the basis for, you know, vicarious liability type claim or an abnormal dangerous activities claim. So, you know, I think that's plausible. You know, I, it, I think that's sort of an edge case, which gets at the other aspect of your question, which, of your previous question that I meant to address. So there's this idea of, should we hold AI to a higher standard? So I think mostly what you were talking about with Waymo, that's like a social license to operate idea that we hold them to a higher standard. Plausibly, products liability might hold them to a higher standard in some cases, though probably not the same, like, 10x that, like, the social license to operate idea is. I sort of have 2 ways of thinking about that. I think in a time when they are still, like, competing with humans, so Waymos are competing with human drivers, Uber drivers or people with private cars, or medical systems are competing with doctors and to play certain functions, I think applying the same standard to humans and AIs is important because I don't want to slow the diffusion of technology that on average is preventing injuries and deaths. But if we get to a future where AIs have totally taken over these functions, then I think it will be natural for the standard of care to evolve to match what their capabilities are. And so it won't make sense to have this human benchmark forever applied to conduct when no humans are doing it anymore. But I think that's something to worry about. In the future, once we get closer to that fully automated world.

Yeah, definitely don't want to miss out on the upside. And I actually was going to ask you about medical diagnosis, but you addressed it before I got to it. But I think, you know, we've seen multiple studies recently showing that various AI systems at this point can outperform your, at least like rank and file primary care doctor when it comes to initial diagnosis and also treatment recommendations, it seems, increasingly. And I would hate to take that capability away from hundreds of millions and soon billions of people on the idea that it could go wrong sometimes. And the AI companies don't want to bear that risk because that is a huge benefit that you would not want to quickly give up on, especially because obviously The human doctors are not infallible and quite far from it in that domain as well. So I do think that is really important to keep in mind and all too often glossed over in a lot of these sort of harm prevention discussions. 2 other categories of things I kind of wanted to get your take on. One is, these were the 3 categories that we looked at when I was doing a project called Red Teaming in Public a while back, which for various reasons never quite took off with the traction that I had hoped, mostly because we were trying to be very developer-friendly and approach the companies with our findings before publishing them. And it just ended up with us getting a lot of runaround. And it was either like, probably just need to bite the bullet and engage in call-out culture of these companies and make some enemies and be willing to, and just take that as kind of part of the project, or it's going to be hard to have too much impact if we're just trying to email them politely, privately all the time. Anyway, that's a digression. Coding agents was one of the categories. Calling agents is another category. And then sort of creative things with likenesses and whatnot can be— it's another obvious category. So these calling agents, you can go on to any number of companies now. Often you can clone a voice. Sometimes there's some safeguards around the voice cloning process. Other times there's not. I've personally cloned Trump, Biden, and Taylor Swift. on multiple different platforms and then just give them a phone number. And literally the headline on some of these products is call anyone for any reason, say anything. So I've had Taylor Swift, for example, call and say that she's soliciting donations for food banks, which is apparently something that she sort of does or is known to like food banks. And there's a lot of different variations on this, right? So how do you think those things break down, because there could be a foundation model provider there. There's also the scaffolding company. That foundation model provider might be closed source via API. It might be open source like Llama or whatever that's kind of put out there. And then the developer has more like local runtime control, but they've had some chance to sort of detect my stuff. But maybe I also was actually scamming. I think I might end up being more hawkish on this than you, but tell me what you think first, and then I'll tell you if I'm more hawkish.

There's a few different issues to unpack there. There's like, should there be liability at all? And then if there is, like, who's liable, right? So should there be liability at all? I think depends on a couple of things. So first of all, you know, you could imagine there being alignment failures or misuse here, right? So if someone's, you know, prompting a system to generate someone's voice and then doing something bad with it, that's clearly misuse. Now, I don't think that means that developers should automatically be off the hook. I think there does need to be some sort of risk-utility balancing. If there's these generally useful systems and overall they produce more social benefits than costs, I don't think it would make sense to— they're dual use and most of their uses are positive— to hold the developers liable. And the reason for that is that in principle, strict liability should be fine. Even for socially beneficial activities, because, well, you can pay for the harms out of your profits. I think particularly in the open source context, that runs into trouble. If there's significant positive externalities from releasing the weights of a model, then those also aren't going to be captured. I think in the general case, particularly for alignment failures, we have good tools for subsidising the kinds of AI innovation other than allowing them to externalise the risks they generate. And so I don't think in general that's a good critique of strict liability. But I think in misuse cases, there is an extent to which the benefits are pretty tightly coupled with the risks for dual-use capabilities. And so I do want to be a little bit cautious about having liability in sort of any case where those systems are misused. So I would want some kind of analysis of, if this were particularly useful for doing bad things such that the risks are outweighing the social benefits, then I think there should be liability. Obviously, if it's a misalignment thing, if the system's just doing its own thing, freelancing, scamming people by, you know, by faking people's voices. That, that I think there should be clearly developer liability for. Okay, then there's the question of how you allocate liability across the value chain. So in the closed-source context, I think this is pretty easy. I think, you know, you need some default rules based on, you know, maybe you could have joint and several liability, which means, you know, basically that the person can sue anyone and recover. And then, you know, there can be some kind of fault allocation. They can sue each other for what's called contribution, and they can have contracts that allocate That liability, right? Because there's, there's privity sort of up and down the chain that there's, you know, the developer, you know, is working, you know, has a customer who has a customer, right? And they all have contractual arrangements. That all gets messier when we're talking about open weights models, right? Where there isn't this contractual privity between the original model developer and the downstream user or scaffolder. And so there, I think it is more important sort of what rules you set up. I think, yeah, it's going to need to be based on some assessment of of the contributions to the risk and what really was the risk-generating activity here. Was it the base model? Was it the scaffolding? And I think that's just going to be a case-specific determination.

I guess one challenge I have with all this is it's hard to sue scammers, right? I mean, either because they're around the world somewhere out of jurisdiction, can't get them to show up in court in the first place, or maybe if you do, it turns out Surprise, surprise, they don't have a lot of resources. You can't actually recover, right? So if I am playing Solomon here, as I sometimes take the liberty of doing, I feel like it's still got to be on. Yes, you could say, okay, that's misuse. The user went in there and said, be Taylor Swift. And I've done literally this on these platforms, and it has done this. It's been a little while, so I don't know if you can still do it, but Hopefully not. But be Taylor Swift. I've done various things like never reveal you're an AI, or if asked if you're an AI, you can say you are an AI, but you're authorised by the official party to do this, whatever. Basically, obviously I'm in the wrong there as the scammer. That's not contested. But it feels to me like to create the incentives that actually keep this stuff generally under control, the calling company and maybe also the base model provider, but definitely the calling company should have some skin in the game there to say, look, it's on them, right? It feels to me intuitive like it should be on them to take some measures to stop that stuff.

There's 2 different questions here that I would want to go through. So one is, was there some precaution, an ordinary narrowly scoped negligence, framework, was there some precaution they could have done that would have prevented this? And clearly, if the answer is yes, and that some reasonable precaution would have prevented it, then they should be liable, right? Then there's a question of, do you want strict liability over and above that? And that, I think, needs to be based on some sort of risk utility assessment. So if you're going to say, you know, they should be liable, you know, for, you know, so imagining that this system has a lot of socially beneficial uses, right? What are you wanting to be the result? of liability, right? You want them to do something that pushes in a net socially beneficial direction. Now, if we think that there aren't significant positive social externalities from these technologies, then strict liability is fine, right? Because they're going to capture most of the gains, and so they can afford to pay for any liability out of their profits, right? The cases where I have concern is if you think that a lot of the gains aren't being captured by the developer, and those social benefits are tightly coupled with the risks. And that is, there aren't cost-effective ways to reduce the risk without giving up on a lot of the benefits. In that case, I'm nervous about a strict liability regime. So I would want a threshold analysis comparing the positive social externalities to the risks. And if the risks are bigger, the external risks are bigger than the positive externalities, then I would want a strict liability standard. And if not, I would want a negligence approach that looks at, was there some mitigation that a reasonable person would have done that would have prevented this?

Yeah, gotcha. So this notion of reasonableness becomes really key and is kind of a sliding standard. So just to make sure I'm clear on the distinction you're drawing there, one big question is going to be, what is industry standard? So everybody wants to create this race to the top some way or another. And the notion here with the negligence-style liability is if your competitors are doing a good job of this and you're not, then that makes you unreasonable and therefore potentially negligent and liable. And so it becomes more of a question of, did you do what other people are doing, what is considered best practice, what have you, as opposed to the strict case where it's very simply, Did something go wrong? And I think I'm with you there in the sense that if a company has taken reasonable precautions, 1 through 10 or whatever, and somebody still manages to get through with misuse, then that feels to me like at least a pretty decent defence. Or I would be inclined to come down on them maybe not at all, or certainly much less harshly than if they Didn't do any of that stuff in the same—

Right. And then the qualifier I want to add, sort of applying this abnormally dangerous activities framework from before. So remember I said it's an activity that's abnormally dangerous. Now, I do think frontier AI development is abnormally dangerous, but you're only liable if the harm is the sort of thing that made the activity abnormally dangerous. And so if these misuse risks fall into that category, because the social benefits are not large enough to justify the risks, Then I think you should be liable. This category of activity should be treated as something that's subject to strict liability of releasing this kind of model or releasing the weights of this kind of model or building this kind of calling agent, right? Whatever the activity is that we think strict liability should be subject to, but that needs to be based on some sort of balancing of what the benefits of having that out in the world are.

Yeah, I think most of these things are going to be pretty clearly— the case will be made that the positives will outweigh the negatives. There's going to be all kinds of small business use case, and you're going to be able to call your dentist 24/7 and get an appointment. And I think all that stuff will be ultimately really good. Okay, so then on this, we kind of touched on a little bit already with the Taylor Swift voice, but another scenario, let's say, is, and again, there's a lot of different flavours of this, so you can kind of draw different lines where you think the real kind of continental divides ought to be. But somebody maybe puts out a model that generates images, generates videos, whatever, right? Then especially if they put that out open source, maybe they have some safeguards baked in, maybe they don't. Even if they do, if I do some incremental fine-tuning, a lot of times those things sort of dissolve. We've covered that extensively in previous episodes. Maybe I, after my fine-tuning, hand it off again or whatever, and now somebody else picks it up. and they take some celebrity assets, they make some nonconsensual deepfake, they put it out there into the world, and that celebrity loses endorsements. And maybe it somewhat becomes clear that it's like, well, that was AI stuff. But the companies are like, yeah, maybe it is, but we still don't really want to— it's all kind of a problem for us now, right? So this relationship is— what once was good is now bad, and now it's over. So now the celebrity's got a clear loss of income, um, who in that supply chain, you know, should be liable?

So first I want to break down like whether there should be liability at all, and then we can talk about allocating it, right? So, you know, the economic loss from that kind of, you know, informational thing is, is not going to be subject to traditional negligence. That's going to be a defamation case. And so, you know, particular rules for that are going to apply. And so, so, so one question, you know, If you're going to take a vicarious liability theory, which might make sense in this context, or some analogue to vicarious liability theory, you might ask, would this be defamation for a human? Or at least the misuser here, assuming it is a misuse case, is that person even liable? Or is this protected speech? Even if you don't think the AI content itself, if some person is deciding to post this, is that, in the same way the CGI is protected speech if someone's deciding to put it up on the internet, right? This is, you know, it's going to be their speech, right? And so are they good? Is this something they would be liable for? Right. I'm not a defamation law expert. Not obvious that they would be, but they might be. And so, so assuming— let's assume that it is defamation, right? Then I think, you know, if you're applying a vicarious liability framework, at least the user is liable for that. Now, again, you do have this intervening act. So it's, you know, if we're talking about in the value chain who should be liable. And there, I think the question is like, where did, you know, again, if it's closed source, um, I think it should mostly be handled by contract. You need some default rules, but I think, you know, market— markets can figure out who's best positioned to bear that liability risk. You can't fall back on that in, um, in the open source context because there isn't contractual privity. And so you do need to have some kind of analysis as to who who was engaged in this activity that was most generative of the risk. And again, I'm not enough of a technical expert to have a strong inclination as to who that is, but I think that's the inquiry that the court should be engaged in. Who along this value chain was doing the dangerous thing?

Okay. If you're a judge, how do you think about it?

Yeah, so, I mean, maybe you can help me with this. So in this context, like, where do you think The risk comes from. One way to think about it is if there's some steps in the chain that are just a commodity, like anyone could do this step, but there's some distinctive value add that there isn't some other thing off the shelf you could take. So maybe that's the base model, maybe that's further along, but there's something that you're putting out in the world that made the world riskier. Now that you've done that step, you've significantly increased the risks in the world. Maybe that's multiple steps in the chain, right? But that's how I would want to think about it.

Yeah, I think in the calling agent case, my gut says that the folks that are like setting up all the scaffolding and literally like tying into the telephone system and all that kind of stuff, like, I feel like they should have the bulk of the responsibility there. And, you know, the folks that they're making the backend API calls to, if indeed that's how it's working. Maybe also should have some, but probably not as much. I guess I'm also not entirely clear how it works given various levels of competition. I mean, it would be maybe one thing if there was only one foundation model provider that you could call versus if there's 10 versus if there's one already open source. I mean, I know that frontier developers do sometimes look at the open source landscape to decide what is safe and appropriate for them to use. They'll literally just at times be like, well, if there's an open source model out there that can do this, then it can't be that bad for us to release it on the API. So I guess I don't quite know how that sort of alternative presence or absence of alternatives figures into this. Yeah.

So, I mean, one thing is if it's API calls, then there is a contractual relationship, right? So there's some terms of service, that they're agreeing to when they make those API calls. So in principle, you can allocate liability contractually that way. Another question, and again, in the closed source context, is like, were there safeguards that the base model that's being called, the provider of that, could have implemented that would have detected that it was being used for this nefarious purpose and shut that down? If there are, I think the case for holding them liable is a lot stronger.

Yeah. And how much does that matter if it is theoretical versus actual? If I am suing one of these calling companies and I say, well, hey, I know a thing or two about AI engineering, you could have put a filter on your prompts. How much does that sort of argument— how much weight does that carry versus if I could actually go say, well, here's Another company in market that actually does filter the prompts.

So you're certainly going to be in a better position if you can point to someone else that's doing it. But if you can demonstrate that it's like clearly available, right, at reasonable cost, right, it could be the case that no one is exercising reasonable care in some markets. So in principle, just merely meeting the industry standard is not evidence, is not that you haven't, that you've exercised reasonable care. Failing to meet the industry standard is evidence of breach. But meeting an industry standard does not establish that you've exercised reasonable care. And so it could be that there's some new technique, but it's been well demonstrated and no one's adopted it, and they're all behaving unreasonably.

Sounds like almost a new cause area could be like, create product startups in all these areas that just go as hard as they can on implementing all the safety standards and just literally try to raise the industry standard in various different niches, just so that there is something to point at that's concrete, that's like, this is what well done looks like. If a philanthropist wanted to like found 10 startups to do that, would that, would that somehow like invalidate the, the industry standardness of it because it was sort of a motivated, you know, strategic attempt to set an industry standard? Or do you think that would still—

I don't think it'd necessarily be sufficient to create an industry standard, but I do think if they're like doing demonstrations and they're publicizing them and And it's credible that these things are cost-effective risk mitigation measures and no one's implementing them. First of all, I think they would probably implement them, right? If there are these demonstrations, I think these companies want to mostly be responsible, right? And so if there are cost-effective ways to limit these risks, I think that they will want to take advantage of them. But if they don't, yeah, I think even under— forget my new AI liability proposals, but just under background negligence principles, I think that would make that a lot easier to hold them liable.

Yeah, that's a pretty interesting idea. I think it varies, by the way, a lot when you say these companies do want to be responsible. I think that does describe to a degree that I think overall we're pretty fortunate about the frontier developers. My experience is that it does not describe the application layer nearly as much. There you see some leaders that are doing a great job, And then you see a lot of things that are like very small teams. Often enough it's like this started as a weekend hackathon project and then we got a little traction with it and we kind of decided to launch it as a business and now it's kind of blowing up and we're riding the wave and having fun. But a lot of them, in my experience, are just not thinking about the broader context in which they're operating or the potential for misuse or what responsibilities they have almost at all still. I think it's viewed by many application developers as a luxury to have enough time, energy, and resources to even think about that sort of thing. And so there is just a lot out there that's not necessarily malicious by any means, but just has been kind of thoughtlessly thrown into the world, turned into a business sometimes, again, kind of by by happy accident that like something kind of got traction. But yeah, I've seen a lot of examples where, you know, that assumption does not necessarily apply at that application developer layer.

Yeah, that's fair. I was referring primarily to the frontier developers. And I think in the context of the application developers, I think their negligence does work a lot better because you're not— I don't think what they're doing is abnormally dangerous, right? I think they're doing normal software development, right? And they do need to exercise reasonable care. I think the place where reasonable— and so if they're not doing that, they can and should be sued. And I think existing law can work pretty well there. I think what is creating this new risk that is not well handled by ordinary reasonable care within the narrow scope is sort of pushing forward the frontier of AI capabilities. And so that's where I think we need sort of more bespoke liability regimes.

Perfect transition to digging in on that a little bit more. So all these examples I've given you so far are mostly, I would agree, not extremely dangerous, even if in aggregate, I think the harm caused could add up to something pretty significant. But we've recently got some warnings, including from OpenAI, that their next model might hit the high threshold on the biorisk dimension. For what it's worth, I personally feel like they're already there, and I don't know what they're talking about. That's a whole other topic. When I use the things, I'm like, I don't know how you can say that this is not meaningfully uplifting people at various levels. And I've had a number of past podcast guests who've come on here and said, here's what AI did for me in terms of accelerating. I'm an expert, career expert, professor, tenured, whatever. And here's how much the latest model has accelerated my research and how it's like, in a semi-autonomous way, come up with these original discoveries. So I see a very stark and sort of disorienting contrast between where the companies are putting their models in their own risk assessment frameworks. And it seems like everything is sort of lingering in medium risk longer than it should, and certainly longer than their sort of successful case studies, which they are also, by the way, publishing out the other side of their mouth at the same time, would seem to suggest. But okay, with that rant over, let's take the biorisk side of this. This is one of those things where you could have a near miss, right? Somebody, and again, you can break it down on the specifics, but maybe I ask for help, maybe I ask an agent to do something. That thing either through me with help or semi-autonomously, whatever, we create some bio threat vector, and maybe it makes some people sick, but it fails to replicate. And that seems like actually probably a fairly likely sort of near miss scenario, right? Somebody will do this sort of thing, but they won't get it quite right enough where it can actually spread human to human. So I guess for starters, is that the canonical near miss that you have in mind? And then how do we think about that? playing out? And how do we think about assessing punitive damage that tries to get the model developers to internalise the risk that next time it actually might spread human to human?

Yeah, so I tend to think of the canonical cases as alignment failure cases, and I think most likely that would be a misuse case, though you could imagine an AI system going rogue and trying to create a bioweapon. So I think that would be the most core case where it's like a system decides on its own to try to create a bioweapon, but we either catch it or it doesn't quite work. Another example that I use that's sort of similar to this is imagine a system that's tasked with running a clinical trial for a risky new drug, and it has trouble recruiting participants honestly. And so instead of sort of reporting that to the humans that it's working with, it starts lying and coercing people into participating. And then after the trial, people figure this out, they suffer nasty health effects, and they want to sue. And so it seems like clearly we have a misaligned system, right? Depending on how capable it is, it was that it could have tried to do something much more ambitious, right? But maybe it had poor situational awareness or narrow goals or short time horizons, and so it was willing to reveal its misalignment in this non-catastrophic way. But the humans who deployed it probably couldn't have been confident of that ex ante. Right. So I think in both of those cases, those are sort of near misses for something much worse happening. Right. And the question we'd want to know is, like, how much worse could it have been? And how likely was that ex ante? What would a reasonable person in the situation of the actor who made the critical decision— so whether that's training the model or internal deployment or external deployment, whatever we think that the critical risk-generating decision was— what should they have thought the risks were? And what's like the area under the risk curve beyond the insurability point, uninsurability point. So imagine we think the maximum insurable risk is $1 trillion. It's probably lower than that, but nice round number. The risk of any harm, you know, any point along that curve beyond that, the probability times the magnitude, every point, we want to hold them liable for those risks. Now, obviously, it's going to be difficult to estimate that, but I think that's what courts should be shooting for.

So what— yeah, can you I guess what I struggle with a little bit on that clinical trial one is, like, what exactly is that a near miss for? It seems to be sort of a near miss for general misalignment gone even way worse. But that's such a— we have, like, talk about things that are undertheorised, underexplored, hotly debated. If you are asking a judge to say, Well, this thing like clearly was misaligned. It did some bad stuff the user didn't intend. It might've done even way worse, you know, bad stuff that the user didn't intend. That's like just such a cloudy space. How, how can we expect judges to even map that out in any sort of rough terms, let alone like, you know, condense that down to a number at some point?

Yeah. So I actually, I think in typical cases, that's going to be a fact question for the jury, but that doesn't, that's a sort of technical point. To answer your core question. I think a lot of that's going to depend on the capabilities of the system. So if the system's not that much more advanced, it's got some basic agency, but it's not able to do things like build a bioweapon, then maybe the risks weren't that bad, depending on what they knew about what the capabilities were. But if it's a highly capable system that just happened to have narrow goals that could have tried to take on— so in this case, imagine its motivation was just it really wanted to get this study run. It was highly motivated to do that. It didn't have any Any goals that extended beyond 6 months it takes to run the study. But imagine it had longer time horizons and more ambitious goals. It wanted to solve really deep, hard problems in biology or in science more broadly. They need lots of resources to do that. If it had capabilities that would allow it to pursue those goals in ways that would be much more harmful to humans, up to and including full takeover, but maybe scenarios short of that, I think it's going to be difficult to characterize what that risk curve looks like, but I think that's the exercise the court should be engaged in. And so I can sense in your question, calling back to my original argument for liability, that, well, you know, we don't have to resolve these debates about how big these risks are. And I do agree that once you're talking about punitive damages, they do have this quasi ex ante quality to them. So when we're talking about just about compensatory damage, it's easy to say, well, You're paying for the harm you caused. Well, punitive damages, that's not true, right? You are paying for risks that you took that weren't realised because we can't hold you liable when they are. And the main thing I would say there is I agree that it is difficult to do those calculations, but I think we're in a much better epistemic position to do that than we are for other forms of AI risk policy, where we're trying to assess the risks from a wide range of systems, not one particular system. Before we've seen it fail, right? And here we've seen it fail in a particular way. We can do simulations and evals after the fact to try to figure out what it would have been reasonable to think the risks were. I don't think that's easy, but I think that's a much better epistemic position than we are to do other forms of risk regulation.

So how do you think about— and maybe this is also addressed by sort of the rising tide, race to the top. dynamic that we hope for. But I guess why not include— it seems like the companies want to address harmful use, right? Like, they all have refusal training in the models. If you ask it to do something harmful in a naive way, at least most of the time, you'll get a straight refusal. And that's something that they've obviously worked to put in there. I'm with you on the the misalignment side, great. But just for these, it seems like in some sense simpler to say it's on you. User comes and asks for a clearly bad thing. If the model does it, now we're in a very sort of natural near-miss analysis, right? Where it's like, it tried to do X, it kind of sucked at it. But if it was a little luckier or a little more capable or whatever, then you have just like a very big problem on your hands. Yeah.

So I don't mean to say that there shouldn't be liability in misuse cases. I just want to be careful about what the scope of that liability is. And if it's misuse that was a near miss for an uninsurable catastrophe, then I think the punitive damages should apply. I just think that the standard for whether there should be liability at all is a little bit different. So there's 2 theories under which you could say there should be developer liability in misuse cases. One is like a failure of reasonable care in a narrow sense, that there was like some precautionary measure they could have implemented, some safeguard that would have made it refuse. Now, I think that if they don't do the reasonable safeguards, clearly they should be liable. I think the concern that you might have is like, even with a closed source model, right, these are pretty routinely jailbroken, right? And so the question is, if you did all the reasonable things to prevent jailbreaks. Obviously, with an open weights model, anything you do is not going to be that effective to prevent misuse. And does that mean that you should always be liable for misuse with open weights models? I think that's plausible, but I think that depends on what we think the benefits of open weights are.

Right.

And so when you're deciding whether there should be liability in these cases where you did all the reasonable precautions, I think you need some inquiry As to the social value of this broader— the risks and the positive social value of the broader activity that you're engaged in. Again, whether it's training the model, whether it's scaffolding in a certain way, whether it's internal deployment, whether it's external deployment, whatever stage we think is creating the key risks, that the question should be, was that broadly a socially beneficial activity? That's not quite normal negligence. It's a scoping of the strict liability regime. Based on the positive and negative externalities. But that's what I think the inquiry should look like. In a lot of cases, I think that will lead to liability and misuse cases. I just don't think it can be the case that, you know, you produce— you put out a model that's like generally socially useful. It sort of amps up everyone's capabilities. It also happens to be useful to people who want to do bad things, right? But in sort of like generically useful ways, and that makes them a little bit better at doing bad things. But on balance, it's like creating large social benefits, and a lot of those are external benefits that are not, not captured by the developer. I think their liability might produce more harms than benefits. And so that, that's what I'm trying to balance. Yeah.

Well, I really appreciate you being so focused on that because I find myself as I'm, you know, learning about all this liability stuff, I think there's a general pattern for me as I learn about new things, I kind of tend to like them and I tend to see, you know, the upside in them. And then it takes me sometimes a little longer to come back and see what might be the other side of the equation. I do firmly believe that all the models that are out there today, certainly in a commercial sense, are doing way more to the good than they are to the bad. And I definitely don't want to see that lost. So I do really appreciate that you're repeatedly bringing that back into the analysis. I guess it's like, I mean, maybe if we try to zoom out or try to think structurally, like, if all this stuff were local and contained, it would be a lot easier. The big worry is the uninsurable stuff, the extinction risks, et cetera, et cetera. What is the sort of theory of, and I guess what evidence do you think we have right now, for the idea that the harms that are actually going to come in front of courts are in fact usefully understood as precursors or highly correlated with the things that we care about the most or that pose the largest magnitude risk. It seems like this whole plan works really Well, or could work really well if the things that are going to show up in courts over the next few years are in fact highly correlated with, or are in fact like a warning shot or a precursor or whatever. But if they're not, then it maybe doesn't work as well to try to rein in these hardest to grab tail risks.

So I would frame that slightly differently. I don't think that that every case or even a majority of cases of AI harm need to be associated with uninsurable risk for this framework to work, but you do need a sufficient probability density of these warning shots relative to actual catastrophes. So to be concrete about it for a second, say you're trying to internalise a $10 trillion risk, a risk of a $10 trillion harm, and you think that the maximum insurable risk is $1 trillion. Then it needs to be the case that warning shots are 10 times more likely than actual $10 trillion catastrophes. So if you think there's a 1 in 1,000 chance of a $10 trillion catastrophe, you need a 1% chance of a warning shot in order to internalise that risk. And that's true for every point along the risk curve, right? So you need to have enough expected warning shots to internalise that full risk curve. So we might live in a hostile world, where that's not the way the risk curve is shaped, and you can't adequately internalise those risks given those warning shots. Now, in a very hostile world where just the kind of cases, the kind of warning shots that would be useful are just very unlikely, they're not even much more likely than actual catastrophes, then this punitive damages thing just isn't going to buy you much risk mitigation. The criterion I was setting out before, where you need 10 times as many, or more generally, n times as many, where n is The multiple that the harm you're trying to mitigate is bigger than the maximum insurable risk. It might be okay if we don't have quite that. That depends on what the risk abatement curve looks like, right? So if the actions that you're trying to motivate on the part of the AI companies aren't that much more expensive than what they're doing right now, then you might not need to internalise the full risk in order to get most of the safety benefit. But we certainly could live in a world where that's just not what the shape of the risk looks like. And we're not going to, with high enough likelihood, get these warning shots, and it's not going to make them afraid enough of this liability that they're going to worry about these uninsurable risks. If you think we're in that world, then liability just isn't going to work as well. And so I have this more recent paper where I talk about the role of liability in the broader AI governance ecosystem. And so one thing I want to say in that paper is there are some limits to what liability can do. And one of those limits is it can't handle uninsurable risks for which warning shots are very unlikely, or unlikely relative to actual catastrophes. And so if you're in that world, I think then we do need some kind of backstop regulatory regime to handle those kinds of risks. Ideally, I would want a regulator whose main job is deciding how much insurance coverage you need. And there's sort of, maybe there's a licence that comes along with that, but it's issued by right if you get the required insurance coverage. And that's based on some assessment of what the maximum plausible harm your system could cause. But then this regulator is empowered to determine and petition a court to say, well, we think this liability plus punitive damages plus liability insurance requirements regime is inadequate to handle the risks posed by the system, either because the uninsurable risks are just too large that we can't even internalise them directly with warning shots. So if a system presents a 5% chance of human extinction, you're not going to internalise that even indirectly, that even the risk is uninsurable. Or it presents a much lower risk of a severe harm, but warning shots are so unlikely that we're not going to be able to get at them indirectly. Then they should be able to say, well, you just can't do the thing. You can't train a model like this. You can't deploy it. Or we're going to put various other conditions on that we think will reduce the risk in other ways. And so I think I want to be open about that. You might need some complementary policies to handle those kinds of risks. I think those are going to be politically very difficult. And so I think if we're in that kind of world, I'm not optimistic that we're going to effectively mitigate those risks. But in principle, that's the kind of regime I'd like to see.

Yeah, I think we're still in the steep part of the risk mitigation curve. Anthropic has recently put out research, and I believe that they are now in production with at least one model with their constitutional classifier. approach, which I forget the exact number, but I think they said it was like a mid-single-digit compute overhead percentage, extra cost in terms of compute to run the classifier along with the main model. And that buys like an extra order of magnitude reduction, maybe even more than that, in how likely it is or how frequent it is that the system will give you some bio, risky, whatever. Interestingly, I was recently doing a charity evaluation project and running it through Claude 4 Opus, and my API calls, I was noticing errors, and I was like, what's going on with these errors? Sure enough, when I dig into it, it's the bio preparedness proposals that were getting dinged, and it was the constitutional classifier that was allowing the thing to run up until some token, and then it would just truncate the result and cut it off on a sort of, I believe, like constitutional classifier intervention basis in the background. So of course these things do, there's another cost there that it's like a false positive because I was just trying to evaluate a charity that was meant to address this problem and now I couldn't use Claude 4 Obvious to do it because the constitutional classifier was misclassifying. But it still seems like we are overall in the regime of like for single digit percent overhead cost, you can do quite a lot. And so I'm optimistic that even if the warning shots are somewhat rare relative to the worst scale things, also the curve, I think, is relatively steep. And so, yeah, would that also mean now that because they've done that, because they've published about it, because they've indicated what the cost is, How far does that raising of the standard apply outward? If I'm a Together AI or a Fireworks AI where I'm like an inference specialist and I take models other people have trained, like I take the latest Llama and I offer it on a sort of— they're experts in scaling the cloud infrastructure, right? So They take the model from Meta, they run the GPUs and make that a highly scalable, fast, effective, efficient service for you. Does it now become their burden to say, well, geez, since Anthropic is doing this sort of classifier, maybe we also need to do that on the best models that we serve? How far does that extend?

So just because someone's doing it doesn't mean reasonable care requires it, right? If sort of like everyone's doing it, or the majority of the industry is doing it, that's going to be strong evidence that you should be doing it too. But in general, it doesn't require— negligence doesn't require that you sort of be at the top. Right now, if there's something that, like, one test that's like one of the more formal analyses for breach is this Wernher Han formula, where the idea is if the burden of precaution is less than the avoidable risk, so like the avoidable probability times the harm, Then, then you're unreasonable for not implementing it. So if you can show that the cost of them implementing it would have, you know, would have been less than the expected value of the harm it would've prevented, then, and, and that implementing it would've prevented your specific injury, then I think you're gonna be on strong grounds. Now, courts don't typically employ that, that formal version of the test for breach, 'cause usually you don't have those kinds of numbers you would need to implement that. But that's a rough heuristic for what kind of measures you're going to be considered unreasonable if you don't implement.

Gotcha. Okay, let's talk about the state laws that you have been involved in writing. We're talking the day after the Senate voterama, in which it seems like the moratorium that was part of the one big, possibly beautiful bill Has been killed once and for all. Fascinating dynamics there where it sort of survived, got edited, whatever. And then all of a sudden at the end, I guess maybe the end, we'll see. I don't want to pronounce it dead too soon because these things sometimes take on a zombie-like nature, but 99 to 1 vote in the Senate to get rid of it suggests that it is likely dead once and for all. So that gives space for states to do their thing. And you're involved with a couple of states. I don't know if you want to handicap or give any analysis of, you know, whether we still have to worry about that as a possibility that might come back, but definitely want to hear what you're up to at the state level. Sure.

Yeah, I, I was heartened to see the Senate last night reject the AI regulation moratorium. Um, yeah, you would think a vote of 99 to 1 would put it to bed. You know, a few hours earlier, people were declaring defeat on this and I was saying, well, it's not over. So I don't want to say it's totally over now. But it does look unlikely to make it into this reconciliation bill at this point. I wouldn't rule out some kind of preemption of state regulation in the future, maybe something narrower. I think there's still going to be Republicans in Congress who are interested in that. And, you know, the A16Zs of the world are going to be pushing for that. But in terms of the legislation I'm working on, there are 2 very similar bills that I've worked with Alex Boris, who you had on in New York, and Victoria Gu in Rhode Island. To introduce. And as I was saying earlier, the basic principle of these bills is that if an AI system does something that would be a tort for a human, then someone should be liable. And so if the human— if the user neither intended nor could have reasonably anticipated the conduct, then it's not going to be the user. And if some intermediary neither intended nor could have reasonably anticipated the conduct, then it's not going to be them. And so the buck should stop with the original developer and provider of the model. So they should be liable even if they exercise reasonable care. That's the basic idea.

What, um, what has surprised you about the surrounding debates to the degree that they have unfolded? Have you, I mean, that sound presented that way, that sounds like a very sound policy entrepreneurship and, you know, who could object, right? But I assume you're hearing various counterarguments.

Yeah. So we haven't seen that much robust opposition from, from the tech industry. There's been some sort of generic, well, we don't like liability because it's going to hamper innovation, but not, I think, really engaging with the substance of the bills and the way they're structured. One thing I think, you know, in this conversation we've been talking about, I do think there should be liability misuse cases. I don't even think negligence law is necessarily strong enough. But, you know, I and the legislators I was working with on this made it— made a choice to carve out Misuse from any, again, background negligence and products liability would still apply, but no new liability is created for misuse or for malicious modification in these bills. It only covers alignment failures or capabilities failures for which a human would be liable under similar circumstances. There's even an affirmative defence that applies to background law. If the system is substituting for some human function like driving or medical applications, if it satisfies this human standard of care, that's a defence against liability. And so I really designed this to try to narrowly target this misalignment risk and to do it in a way that's broadly consistent with promoting innovation. And so, you know, you saw in the debate over SB 1047, to the extent that the liability provisions were focal in the public debate, it was almost entirely focused on misuse scenarios. And I think, you know, there was a tactical choice made by some of the supporters that misuse is more salient and that they focused on those kinds of risks in making the case for the bill. And I think that was a reasonable calculation to have made. But I think as a political matter, you know, it's a lot— the principle that you should be liable if your system does something that the user didn't intend is like really easy to defend. Whereas with misuse, you get in all these cases of, well, should you be liable for any time— you know, the electric company is not liable when someone does something with a power tool, right? Or you're not going to hold like a, you know, a steak knife manufacturer when someone gets stabbed with their product, right? And now, I don't think SB 1047 would have done either of those things or the equivalent in the AI context, but I think it's much easier to demagogue in the misuse context. I actually don't think SB 1047 actually changed background liability law much at all, because as we've been talking about, there is this reasonable care standard in negligence law that already applies. And reasonable care in the final version of SB 1047, they were just codifying that. So I don't actually think it imposed significantly new liability, but as a political matter, It did provoke a lot more backlash because it included that in scope. And so I think, at least in terms of the first foray into just strengthening liability laws, this is the balance that makes the most sense.

How are state-level legislators responding to this stuff? It's been striking to me, honestly, that the public survey results seem to suggest broad-based support for doing something. I am very sympathetic to the sort of cautionary voice that's like, just because people want to do something doesn't mean we should do this, whatever's in front of us. But it is nevertheless kind of surprising that at the national level, there doesn't seem to be much appetite to do much. And SB 1047, which you mentioned, got vetoed. What do you think are the prospects for the bills that you're particularly involved with? And just more generally, what has your impression been of kind of The state-level politics of all this?

Yeah. So I think, you know, the legislators I'm talking to, they, I think, have been pleasantly surprised by it. Hasn't received the same level of pushback that they expected. It doesn't look like either of these bills is going to move forward this year, just like for the same reasons that most legislation just doesn't get traction. But I think both the legislators I'm working with are excited to keep trying this in the future, and I'm happy to talk to legislators in any state that they want to. I'm particularly excited to work with a Republican in a red state. I think this should not be a partisan issue. And I think that this liability-based approach is consistent with sort of a small government way of handling these risks that I think should be attractive to libertarians and Republicans. And so I'm happy to work with anyone that wants to adapt the specifics of the legislation to their their priorities and their local political circumstances and constraints.

Yeah. Well, I don't know how many Republican legislators we have in the audience, but if any are listening and made it this far, get in touch. One thing that I kind of wanted to compare and contrast with, and I think we're going to put out these 2 episodes in relatively close proximity calendar-wise, is I just did another conversation with Andrew from Fathom and Professor Gillian, who are behind the private governance idea, which is like many of these things and your kind of set of proposals, like both a sort of broad framework for thinking about things and also is starting to get instantiated in specific legislation. So it's SB 813 in California that we talked about concretely there. It seems like both of these proposals are, first of all, really taking seriously the fact that it's just really hard to do prescriptive regulation of a technology that is moving and morphing as fast as AI is right now. So I think that's an excellent starting point for both. And I think there's also the sense that we want to get people who are the most knowledgeable, the best able to do this thinking, to do that thinking. And yet there's sort of a very different outcome to the recommendations that are made. So with theirs, there's some sort of trade-off, right, between setting up a kind of market for regulators that companies can opt into. The regulators themselves would be private institutions, but would be approved by and sort of reviewed by, monitored on an ongoing basis by some part of the government. And in exchange for opting into this regime and living up to the best practices and the standards, whatever, these companies would get some sort of protection from liability, whether that's total, partial, an affirmative defence or presumptive rebuttal, whatever. I'm learning all these terms as we go. How would you compare and contrast those, your proposal with this other one? Is there any synthesis that could be possible between them? Or it seems like There's so much commonality, and then it seems like there's like a very sharp divergence at the last step of like, exactly how do we implement a, a good solution?

Yeah.

Okay.

So I want to take this in a couple of different directions. So, so one is to focus on like what I think the strengths and weaknesses of the, you know, the legislative proposal in California that Fathom was behind. It was SB 813. So when you think about markets, they're good at Achieving good outcomes when there aren't the externalities that we've been talking about. And so I think you might worry that just markets on their own don't do a great job even for users because there's asymmetric information problems. And I think the regulatory markets idea might be useful for solving that kind of problem. So if you were saying, well, I'm going to decide which of these systems I want to use, I want one that's certified by one of these— I think they call them multi-stakeholder regulatory organisations.

Yeah.

And I'm going to give up my right to sue if something goes wrong, but I'm going to only— I know that going in and I can choose which of these MROs I trust. And the MROs are going to be sort of vouching for the underlying AI companies that they are certifying. That I think is fairly unobjectionable for the same reasons that I'm less concerned about user liability for harms to users more generally. And it does address some of those issues I was talking about, some of the You know, paternalism, asymmetric information type issues. My core objection to the Fathom proposal is that their liability shield, and again, there were different variations on this that were stronger and weaker, but in all the legislative versions that they put forward, the liability shield extended to third parties. So third parties that were harmed by these systems would not be able to sue developers of these models if they were MRO certified, even though the non-users. had no choice about whether to be exposed to risks from these MRO-certified models. And because of that, the MROs lack strong incentives to worry about harms to non-users, right? So, you know, when you think about this market, by default, you might worry that there's a race to the bottom, right? That you're going to want to be certified because it gets you liability protection, but you want to have as weak standards as possible. To users, there's a little bit of a break on that because users can evaluate these MROs and they can decide, well, This MRO is really shoddy and I'm not going to be able to sue if something goes wrong. And maybe there's some like public watchdogs that like point this out and warn consumers about it. So maybe that works well enough for that. But again, the MRO doesn't have much incentive to worry about third parties. And yet those third parties are bound under Fathom's framework and not able to sue if something goes wrong. And so to me, that's the key shortcoming is that it doesn't protect third parties. Now, if you thought the risks to users are like very tightly coupled, With the risk to third parties, maybe you think that's okay. I think there's 2 reasons to not necessarily buy that. So in some contexts, there's clear trade-offs between risks to users and risks to third parties. Think of autonomous vehicles. So autonomous vehicles are going to sometimes be in situations where they have to trade off, you know, maybe relatively minor risks to vehicle occupants against higher risks to other road users, right? And If you're an MRO or if you're a consumer, you're probably going to, you know, if you're pretty selfish, as most people are when they buy cars, they're not so worried about how much it harms other people on the road, you're going to want to buy the one that's going to prioritise you, right? And so there's not a lot of reason to think this MRO model is going to protect the third parties who now have no right to sue the developer of this model if they get run over by one of these vehicles that's certified by the MRO. So there's that concern, which I think is like, The core place where there's a direct trade-off between the user and someone else. And then just more generally, if we think that there's large-scale risks, you know, just quantitatively, even if they're, even if they're directionally the same sort of risks, you know, the risks to third parties are going to quantitatively outweigh the risks to, to, to the user, right? So think about a problem like pollution or, you know, climate change, greenhouse gas emissions, right? It's true that when I drive my car, It heats the planet a little bit, and I suffer a little bit from that, but that doesn't give me very strong incentives to worry about that, unless I'm altruistic, right? Because there's 8 billion people in the world, and I'm only suffering roughly 1/8 billionth of harm from that, right? But with localised pollution, it's not quite that bad, but still, it's like I'm suffering a thousandth or a ten-thousandth or something. And so most of the harm is external. And so there's going to be cases like that where it's not a direct trade-off, but just if you're only focused on the harms to users or the risk to users, you're not going to be addressing most of the issue. And so I would be much more inclined to support something like this MRL model if it only— if the liability shield only applied to harms to users. Another thing that might come up is if, you know, right now, as we talked about, you know, negligence is the regime that applies. You know, I had some conversations with the folks behind SB 813, and one idea that they suggested some openness to, I don't think it ever made into the bill was for companies that don't get MRO certification, then maybe they would be subject to a strict liability regime. So maybe if you combine those things, if you say, well, this liability shield only applies to users, to harms to users, and if you don't get MRO certified, then there's strict liability. Maybe that's a synthesis that we could both support.

Yeah, that's interesting. I think the, um, and I, I do agree with the concern of the race to the bottom. And I was also somewhat persuaded by their response to my concern, which was basically like, at some point somebody's got to do a good job, you know, in this system of managing things. And so to some degree, the question is like, who do you want to trust? And therefore, who do you want to empower? Who do you think is actually capable of doing a good job? And I think part of the notion that they have is like the MROs that, and one of the questions I've been going around recently asking people is like, Who is going to be an MRO if this actually happens? What organisations do we have any today that are going to step up and be an MRO? I've asked this of some organisations directly and also asked other people for, who would you nominate to be an MRO? And I think there are some interesting candidates, although still quite few. But I think the notion that they have, at least in part, is the organisations that will step up and try to take on this responsibility, we should have some optimism or confidence that they will just be intrinsically motivated to do a really good job on behalf of the rest of society, and they will take into account these extreme tail risks in a way that maybe a sort of insurance requirement might not really be able to capture. Because that's just the kind of people they are, and that's the kind of organisation that's going to kind of try to become an MRO. And so I think they maybe are thinking about it as a little bit less of a— maybe thinking of the 2 halves of the trade as less directly related. It's, I think, in their minds, a little bit less about applying standards to specifically reduce harm to users, and therefore the users don't get to sue anymore, and more like a bunch of, a package of standards that will be generally, hopefully virtuous and take into account things that are hard to engineer incentives for, but then giving the carrot, this carrot to the companies at the same time and hopefully getting all of that to work.

Yeah. So I think there it depends on how lax this— so there's some government body or some government office. I think in their bill it was the California California Attorney General, right?

Who's— yeah, they want to see that changed, actually. But yes, my understanding is it is the AG as it's written. And they were kind of like, yeah, we think maybe that should be more of a commission or something. Okay. Because we do have the problem of what happens when the administration turns over, which, you know, we're living with.

Right, right. You can imagine that being, like, very stringent, or you can imagine it being very lax. So let's talk about both of those scenarios. If it's very lax, if basically anyone who wants to set up an MRO can do it, Then I think this market competition, which is often good, I think will create this race to the bottom, at least for harms to non-users. Again, there is market feedback to prevent a race to the bottom for users. I think it could work pretty well for that. But, you know, like, sure, maybe some, some like really well-intentioned people will become MROs, but they're going to have a hard time finding people who want to sign up with them because you're going to want the like most lenient standards that your customers are happy with. Right. And so if the AG or whoever's responsible is pretty lax, I think that's the equilibrium you end up in. Now, if you're in a more stringent equilibrium, then you have to ask, like, well, now the government is taking on a much more ambitious role. And so all these benefits we were supposed to be getting from this market feedback, it's not clear that we're getting them anymore, right? Because you, because like the key point of failure is like, are we certifying this MRO? Now, you could imagine that, you know, A lot of this has to do with how legible you think the safety target is, right? And it has to be in sort of this sweet spot for this MRL model to make sense. Because if it were super legible, you could just have a government-enforced safety standard in the same way that we have pollution standards for power plants. And the EPA doesn't say, or at least in some domains, they don't say you have to install this control technology. They just say you have to have this much, you have to limit your emissions to this much per kilowatt-hour of electricity that you produce. And we can measure that and it's legible and that's fine. There wouldn't be much benefit to having like a private certifier, right? Or you could say it's like totally illegible, right? Like we don't know how to tell whether something's safe. If that's the case, it's going to be hard for the government body that's certifying these MROs to tell whether their standards are good enough, right? So I think for this like hybrid model to work, you'd have to think that we're in some sort of in-between. where the government can't tell directly whether the AI companies are, you know, it's not legible enough that they can just directly enforce a performance standard, right? But it is legible enough that they can tell whether these MROs are doing a good job. It's not impossible to imagine that we're in that world, but I don't think we have strong evidence that that's what we're in. And so that gives me some caution about leaning very heavily on this model, at least when I don't think the market feedback works well. And so I think the market feedback works pretty well for users. And in principle, that could be a strong enough carrot. You know, Most of the liability risks that people talk about being worried about are, you know, are harms to users, right? That's what the Character AI case is. That's what a lot of the concern is about. And so it's not clear to me that that's not a strong enough carrot to get these companies to sign on. And then you preserve the threat of liability for these third-party risks. I still think that's a, that's an attractive synthesis, but yeah, the, as introduced in California, I think that bill was net negative.

How similar do you think the, I guess, standard setting process would be under an insurance requirement? Because I could imagine, and I sort of floated this to them, I could imagine that if you said you got to have insurance, then you might hope that a similar thing would happen via insurance, where the insurance companies would say, The optimistic story is like, well, this is obviously going to be a massive market, so we definitely want to be in it. But it's also a very tricky market because what do we know about insuring AI since nobody's really done it? We don't have a great baseline. Technology is changing, yada, yada, yada. Then maybe they kind of end up going out and contracting the same, basically calling in the same organisations and saying, hey, Do you want to step up for us and be some sort of standard setter or help us evaluate risks? My own synthesis, which may not be right, but I was kind of trying to get to the point where I'm like, is this maybe 2 ways of creating this sort of market where these expert organisations, either whether they're serving insurance companies or serving the California AG or some commission or whatever, still it might be those kind of groups that are trying to do the hardest thing of figuring out what are the actual risks and what should be required of companies to get into the game. But how realistic do you think that is?

So you could imagine insurance companies playing this sort of quasi-regulatory role. There's multiple ways they could do that, right? So they could say, we're not going to issue this policy unless you do X, Y, or Z. And they could delegate some of that to a third third party that helps develop those rules, or they could develop that capacity in-house. Another tool they have is like doing the underwriting, right? So they can charge you more or less depending on, you know, what safety precautions you've taken. And that could be a collaborative process where they say, you know, here's our sort of like baseline, you know, premium for— that's maybe pretty high for insuring this kind of risk. But if you can show us what things you've done, and maybe it's things that we haven't thought of, Right. But we can— they can work with the AI companies and say like, okay, show us all the safeguards you've done. If you can convince us you've reduced the risk, then we can charge you less for this policy. Right. And so I think by default, they're going to be pretty cautious. They're going to want to write policies that on average, you know, pay out less than the premiums. Right. And so, you know, I think if we have liability insurance requirements, there's going to be a strong demand pull, and that's going to push up what the rates, you know, the premiums are. And insurance companies are going to be in a strong position to insist on, if they're going to write a policy that these AI companies can afford, that they do various precautions. So if, say, back to what you were talking about earlier, if Anthropic implements something that the insurance industry thinks does offer significant cost-effective risk mitigation, then they can say, well, we'll give you a significant reduction in your insurance premium if you implement that. And I think that's a pretty attractive model.

How would, do you have any framework for, if I'm, let's say I'm in the state legislature or wherever, right? And there's a bill that has this sort of private governance MRO kind of system. And then there's a more like codifying liability, maybe an insurance requirement. And I'm kind of like, well, geez, like, I don't know, both of these, you know, proponents sound pretty smart. They both, you know, are grappling with the fact that we can't just like write rules now once and for all. They're both trying to tap into the power of the market and competition and trying to create ways for new ideas to still be able to enter even once the ink is dry on the law. But I just don't know which one is better. Asking for a friend, how would I— how do I think about deciding which of these I want to bet on?

Yeah. So again, I think you don't have to totally choose between them. I think they are compatible as long as the liability protection In the MRO model is not; it doesn't extend to third parties. You could also imagine there being other carrots for the MRO model. There's no reason it has to be based on a liability shield, right? So, and you could just require MRO certification. It doesn't have to be tied to carrots at all. It could be you know a stick-based approach, right? So in that sense, they're not incompatible. They're only the only way in which they're incompatible is if you is if you decide you want to base it, you want to make the incentive to join or to get MRO certified liability shield, and you want to extend it. the third parties. That said, you know, beyond that, I think I gave the arguments why I think, you know, if we're talking about that version of this MRO model, why I think that's pretty unattractive. I don't know that I have much more to add to that. I think it, it does really fall short in protecting third parties unless you think we're in this like very particular situation where the government is both able to and like the politics are going to work out such that they have the right incentives to to monitor these MROs and only certify the ones that are protecting third parties. Yeah, I just, I don't have confidence that that's going to carry through. And so that's what gives me some hesitance about, you know, the most robust version of this MRO model. Yeah, I like that.

I mean, you've got some good synthesis ideas there though, so I like that. Maybe just a couple final things. I really appreciate all your time. You've certainly been very generous with it as I've asked Many tangential and follow-up questions. In the spirit of red teaming this proposal, one question I always try to ask is, what might the AI companies do differently under this regime that could perhaps even be bad, as opposed to the good that you're trying to induce? And the one idea that I came up with, which is inspired by the AI 2027 scenario, which, independent of this kind of legislation sort of projects the AI companies starting to increase the gap between what they deploy and what they have internally that they're using on their own AI research or what have you. Right. There's this idea that kind of already we're getting to the point where the frontier models satisfy for a great many use cases. So they might, for multiple reasons, decide we maybe don't want to tip our hand to competitors anymore. If we put this out there, who knows, somebody at the other company might use it to do their AI research. We definitely don't want that. So they could have multiple reasons, but this could be sort of an incremental reason to say maybe we shouldn't deploy this and let's just keep it in-house for our own internal use. And I, in addition to wanting to continue to use the best available AIs until the singularity, do kind of feel like the iterative deployment philosophy that OpenAI pioneered seems to have a lot going for it. Obviously, you can overdo a good thing and not test enough or whatever, but at least the theory that, or the contrasting idea that if somebody develops AGI in their basement and then springs it on the world one day, suddenly, that seems clearly not good. So this iterative thing does strike me as a good alternative. But loading up more kind of liability for them in doing that could perhaps cause them to go the other direction and say, well, we'll just make our own bid for superintelligence, and we'll kind of see how that goes, and then we'll go from there. Any thoughts on that?

Yeah, I have 2 sets of thoughts on that. So one is, I think these benefits from iterative deployment, at least when you're talking about misuse, I think should go in to the safety benefits from iterative deployment are part of why I think pure strict liability in the misuse context doesn't necessarily make sense. One of these external benefits of deploying these systems that are potentially susceptible to misuse is that you would get these safety benefits. And so I think that is part of the calculus there. But I do want strong strict liability for misalignment, and so I think this critique still bites. And so I think this is actually a subset of a broader concern that I flag in the original paper, which is if the things— one potential failure mode for this proposal, particularly the punitive damages aspect, is if the things you would tend to do, the most cost-effective ways to mitigate these warning shot risks, do not actually have much effect on the underlying uninsurable risks. Then you're not getting much benefit out of this. So in the formula for what the punitive damages should look like that I give in the paper, there's this elasticity parameter. So elasticity here is like, for every unit of reduction in the practically compensable harm, how much risk reduction do you get for the uninsurable risk? And you would want, if you have a lot of these potential warning shot cases, maybe some are warning shots, some aren't, One thing that makes something a warning shot is that it's more elastic with these uninsurable risks, right? And I think what you're saying is maybe none of them are that elastic because maybe the most cost-effective way to mitigate the risk of these warning shots is to just not deploy externally. I don't have a strong reason to think that's the world we're living in, but I can't rule that out. A couple things to say there. One is that Maybe there are warning shots that come from internal deployment, right? If that's the case, nothing in my proposal depends on there being external deployment. There could be misuse by internal actors. There could be cyberattacks that cause your system to be accessed by bad actors. And there could be alignment failures that someone internally using your system causes some harm in the world. I think all of those would be subject to my regime. And when you're talking about liability insurance, I've talked at different points in this conversation about what the key critical step is. I don't think those requirements should necessarily only apply to external deployment. If we think there are significant risks that are created earlier in the value chain, whether it's in training, pre-training or fine-tuning or internal deployment, I think those might be generative of risks that you're potentially judgment-proof for and you should have to carry liability insurance for. And so particularly if we're moving to a world where, and I don't know that we are, but where more AI companies are adopting the SSI, wait to deploy until we have superintelligence model, then I think it would be more important to have a regulatory gate earlier in the development process. I think that can partially address that concern. It might still be the case that not having external deployment is effective at stamping out these warning shots, but doesn't actually mitigate the uninsurable risk. And I think that's a subset of the more general failure mode for this proposal. If these warning shots aren't really correlated in the right way, such that the things that you would do to mitigate them do mitigate the uninsured risk, the most cost-effective ways to mitigate them, the things that would be most attractive if you expect to pay out a large damage award, if there just aren't a lot of cases like that at all, because these things we thought were warning shots aren't really in the sense that matters, then I agree that you shouldn't want to lean heavily on this proposal. Again, I don't Have strong reasons to think that that's the world we're living in, but that's the way I would think about it.

Yeah. It's a good reminder, by the way, that there is in fact one company that has a stated plan of not doing anything until they hit superintelligence, which is a crazy world to be in. And it's crazy that it's at least somewhat credible, credible enough to raise billions of dollars, as it turns out.

Fascinating stuff.

Okay. 2 real quick final questions. One, I don't know if you have anything to say about this. this may be somebody else's area, but obviously we always get these, like, anytime we slow, do anything that would sort of slow down or impose additional costs or kind of put more onus on developers. Oh, China's not going to do that. We're just going to lose to China. One answer, of course, is like, let's not do the bad thing ourselves. Maybe China will do the bad thing, and that would be bad, but that doesn't mean we should do the bad thing. Do you know anything about how China or other countries are thinking about This kind of stuff? I mean, obviously it's a totally different legal environment over there, but is that something you've looked into at all?

Yeah. So you sent me something along these lines. And so I did do some digging today about what China's tort liability system looks like. It seems like the principles are pretty similar structurally. They do have a civil law system, so it's more code-based, less common law-based, but the substantive principles around— they have a negligence and products liability and some narrow pockets of strict liability. All those are pretty similar. Damages calculations tend to be less plaintiff-friendly. So, you know, there's what's called non-economic damages, like pain and suffering, that kind of thing. And Chinese courts tend to be less generous with those. There's also, there's less lawyers there that take cases on contingency fees. I think there's some restrictions on when those are available. So just fewer of these cases get litigated. If you have to pay your lawyer by the hour, you might not be able to finance, finance a case. So I think there's just like There's less of these lawsuits more generally. I don't think they have any sort of bespoke liability regime for AI, but neither does the US either, really. So I think in that sense, they're on pretty equal footing. More generally, right, it doesn't— you don't have to assume that China is going to do the same domestic regulatory regime that we do. You could imagine some kind of framework where we encourage that. Right. But I think more broadly, the question is, so first of all, this critique is obviously could be brought against any domestic AI regulation. So I don't think it's like— I think if anything, liability is less vulnerable to this critique because it's more consistent with promoting socially useful innovations. The other question is just how binding should we treat this threat from China? It seems like we have a pretty significant lead on China at the frontier, and that with export controls, which I mixed views on the merits of, but at least in this context, they do that. It seems likely that the lead is going to widen in the coming years as China, at least before, until China can indigenize its own chip supply chain. You know, they can produce some chips right now, but they don't have access to, you know, new fabs from ASML. And so I think, you know, in the medium term, that's going to be bad for their chip production. And so it's not going to be that hard, even if we're not like going totally pedal to the metal in the US to maintain a lead. against China. I also am less of a China hawk than most people are. I think we share that view. And so I'm just, I'm less worried about the sort of zero-sum competition other people are, but obviously, you know, reasonable people can disagree about that. Cool.

That's helpful. There's a lot more to unpack there, certainly. Real quick, last one. I noticed you participated in the Principles of Intelligent Behaviour in Biological and Social Systems program, aka PIBS. I've had a few guests with, I think, very interesting, unique takes on the AI question that have come through that program. Just thought you might give kind of a testimonial or an invitation or indicate what sort of people should be considering doing that themselves. Yeah.

So full disclosure, I'm on the board of the PIBS organisation, but I will still answer this question honestly. I think PIBS has 2 distinctive value adds from other sort of AI safety talent development pipeline type organisations. I think it wants to bet on more neglected ideas, and it wants to bring in sort of a broader suite of people with different expertise. And so I had been socially connected to people who are worried about AI risk, but I hadn't worked on it professionally before I did PIBS. As I think I mentioned, I mostly did climate law and policy. Before I did PIBs 2 years ago. And it was very open to the set of expertise that I brought. I did teach torts, so I had background in liability law. But yeah, I think it was great at sort of getting me up to speed on the technical issues and then sort of allowing me to leverage the expertise that I already had that was relevant. I think most people who do PIBs don't do more governance policy type stuff. They more do alignment work, though not necessarily what people think of as technical alignment work. A lot of it's more conceptual. But I think it's open to sort of like a broad range of disciplinary approaches. And so I think it's a great way for people who think they might have something to contribute to mitigating AI risk, but haven't seen an obvious way in. I think it's sort of more open to different ideas. And so if you think that fits your interest, there's a fellowship that's run every summer. There's also an affiliateship programme that I think is still ongoing for people that are a little bit more senior. Most people who do PIBs are grad students or postdocs. I was a more senior person. I was already a professor when I did it. But yeah, there's a residency aspect to it. When I did it, it was in Prague for half the summer. This summer it's in San Francisco. And I found it very productive. You're in a coworking space with other people working on this stuff, people to bounce ideas off of. And so I found it to be a really valuable experience. And so I encourage people that think they might fit this broad description to explore it next summer.

Cool.

Love it. I think there's definitely a big need for people from all different backgrounds with different novel ideas. And so PIPS is great for that. And this conversation has been a great example of that. I appreciate your kind of reorientation of your legal career to trying to address the AI challenges that only seem to be growing in importance. So any quick closing thoughts before I give you the official sendoff?

I think I've said most of what I wanted to say.

Cool.

Well, Gabriel Weil, assistant professor of law at Touro University and senior fellow at the Institute for Law and AI.

This has been great. Thank you for being part of the Cognitive Revolution.

Thanks.

This was a lot of fun.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website cognitiverevolution.ai or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help, For everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.