Underwriting Superintelligence: How AIUC is using Insurance, Standards, and Audits to Accelerate Adoption while Minimizing Risks

Hello and welcome back to The Cognitive Revolution.

Today I'm speaking with Runa Kvist and Rajeev Dattani, co-founders of the AI Underwriting Company, who aim to unlock enterprise AI adoption by certifying and insuring AI agents. Their core insight is that security and progress are mutually reinforcing. Just as good brakes, seatbelts, and airbags are required if you want to drive 70 miles an hour, rigorous standards for AI agent behavior and reliability are critical for society's effort to realize the potential of increasingly powerful and autonomous AI systems. Their approach, which has already won support from an impressive list of industry leaders, including Cognition, Ada, Intercom, and many more, combines 3 elements: frequently updated technical standards that codify the latest best practices, periodic audits to verify adherence on an ongoing basis, and insurance to align incentives and provide financial protection when things do still go wrong. As always, in this conversation, we get into the many details of making this work, including the fact that today's insurance policies generally don't explicitly address AI risks, creating ambiguity about coverage for AI incidents. The AIUC1 standard that they've developed in partnership with technology and security leaders across a wide range of industries, which covers data and privacy, security, safety, reliability, accountability, and societal risks, Their approach to auditing client companies, which combines analytical evaluation of technical safeguards with systematic red teaming. How the data generated by red teaming helps insurers price risk in domains where historical loss data is either limited or nonexistent. How they plan to structure financial incentives so as to avoid a race to the bottom, such as that which affected the credit rating agencies in the run-up to the 2008 financial crisis. And how, even though the market may not be able to effectively insure against the largest-scale existential risks from AI, the government, as the de facto insurer of last resort, still benefits tremendously from the governance that this model creates. Overall, I have to say I really like this approach. So much so that I did participate in the company's seed round alongside leading investors including Nat Friedman, Emergence, and Terrain. Certainly there is a place for government regulation and also for unencumbered experimentation, but I think a private sector approach to AI reliability Powered by enterprise customers' desire to move with all responsible speed and designed to align all parties' financial incentives with consumer and public safety, seems much more likely to get the important details right. Not just once, but over and over again as both the technology and the market mature. For now, I hope you enjoy this conversation about how the AI underwriting company seeks to build AI confidence infrastructure and ensure the intelligence age. With co-founders Runa Kivist and Rajeev Dattani.

Runa Kivist and Rajeev Dattani, co-founders of the AI underwriting company. Welcome to the Cognitive Revolution.

Thanks for having us.

I'm excited for this conversation. Uh, you guys are doing some really interesting innovation at the intersection of trying to make sure that everybody can have all their AI goodies, which I am very excited about getting, uh, but also making sure that we keep things on the rails, which I think is obviously extremely important as we head into a brave new AI world. So I'm excited to dig into everything that you're doing, the standard that you've put out, and the future that you guys envision. Maybe for starters, because I think you've used a pretty provocative analogy in some of your writing and the way that you've introduced the company, maybe tell us sort of what the downside scenario would be for AI if we don't do a good job of this. I mean, obviously there's extreme extinction. But you've got sort of the nuclear outcome, which is a scenario where we don't get the goodies, but we still maybe get some of the bad stuff. And then maybe you could compare it to an industry where things have gone well that you take inspiration from. Totally.

And maybe before we get to the nuclear outcome, a lot of the concerns that we're seeing in the world today relate to today's risks that are much more kind of, in some ways, mundane. Data risk, safety and security risks related to brand disasters and data leakage. Some of these kind of age-old concerns that become much more severe in a new environment where there are many, many more vulnerabilities. And frankly, people understand the vulnerabilities much less. But it actually kind of plays out into the future. we'll move to more and more agentic systems that are going to be way smarter. And that opens up new kinds of nuclear scenarios. One that's been talked a bunch about relates to imagine terrorists that want to afflict a huge amount of harm. If they have access to brilliant scientists that can help them create, for example, a new virus, they will be able to do that way cheaper. And so you can imagine new kinds of COVID coming out they'll be way easier for adversaries to create. You can also start to imagine what a misaligned kind of system looks like, where today that might look like slight deception or sycophancy. But as they get smarter and smarter, they'll do some of the things that humans can do. One of the things I sometimes get questions about is, like, well, as AIs get smarter, will they still fail? And I think There are a lot of analogies to humans where a really smart CEO will no longer kind of mess up basic logic, but that enables them— their IQ enables them to run kind of Enron levels of fraud. So as the kind of agents get smarter, you get different kinds of scenarios. And we think that really opens up for nuclear scenarios. A different category of scenarios altogether is the geopolitics. Some of which may not have to happen, like relate directly to AI. But if you really get national security leaders believing that AI will give their nation a decisive strategic advantage, that might lead them to preemptively attack their opponents in ways that may look like regular military escalation, but is really prompted by AI.

And Nathan, you asked about other industries. I think there's not one there's neither one industry that we look to and say, you know, that industry was perfect, nor is there a perfect analogy for AI. But really, the thesis of what we're building here, and I think we'll get into this more later, is how can we bring together learnings from lots of other industries, right? So the kind of 3 components to what we're building, the insurance, the standards, and the audits, has come out of cars. And when, you know, post-World War II, there were a lot of car crashes on highways, insurers really developing crash testing solutions that led to airbags and creating the incentives for that to happen. Go further back into Benjamin Franklin's Philadelphia, and we've written a lot about this, and fire insurers leading to building codes. And no one was saying, you know, stop building at that time. It was, how do we actually keep building, but build in a way that's safer? And that's really the essence of what we're trying to draw on here for our solution.

Yeah, it's funny, you went to even some more tail risks or kind of surprising possibles under the umbrella of the nuclear outcome that I had in mind. When I say the nuclear outcome, for me, that's kind of shorthand for we don't get the super cheap, abundant energy that we should have from nuclear technology, but we do have thousands of nuclear weapons that are collectively a sort of Damocles that hangs over all humanity. And I do worry about a similar thing with AI, where it sort of gets weaponised, or there's this AI Cold War. But maybe we also freeze it out from a lot of the practical upside, day-to-day quality of life improvements that I really want. And what I really, for me, often just think of airplanes, because it's sort of like nuclear. There is the potential for dramatic explosions. And obviously, if one happens near you or you're on the plane that goes down, That's really bad. But yet we do get the upside from planes. We fly cheaply and all the time. And I always think back to Dumb and Dumber, like, it's actually more likely you'll get killed on the way to the airport than on the plane itself. So it is remarkable in my mind how contingent the end state for some of these potentially transformative technologies can be based on even just a few incidents that happen early on in its development that can kind of put us on one course or another.

And I think, Nathan, we're already seeing that some parts of AI today, if you take kind of autonomous vehicles, you still cannot drive an autonomous vehicle outside of San Francisco very far before you get stopped. Yet drone warfare is increasingly autonomous, and it looks like there's no barriers to that getting deployed. So in some ways, we're already starting to see that when it comes to autonomous drones and vehicles.

Yeah. Okay, well, let's get into the positive vision. Insurance, standards, and audits. This is like the holy trinity of a virtuous cycle that you guys are trying to kickstart. So paint the picture.

I can start high level. The core idea is that security and progress are mutually reinforcing. A kind of very basic analogy is that if you're a race driver, the reason you wear a helmet and a seatbelt is so that you can go faster around the corners. And the same is true for AI. The better we are able to steer and secure our systems, the faster we can deploy them, the more we can lean into some of these agentic capabilities. So the debate between the doomers and accelerationists, we think this is a core point that they in fact are mutually reinforcing. Maybe, Rajiv, I'll let you speak about the kind of the mechanics of the incentive flywheel.

Yeah, so the core of this, Nathan, is really, as you were describing, if we don't get the security under control, we don't think we'll get the progress, right? We won't get the benefits of the airplanes. And so what are the 3 components to this, as Runa was mentioning? There's insurance, there's audits, and there's standards. So let me just explain what each of them are, and then I'll talk about how they link. The insurance is the financial protection for when something goes wrong. Who's actually paying for it, right? And can you actually get the financial coverage behind it? There's a standard, which is what is the best practice, and can we actually codify the set of requirements that you need to meet to have the best practices and to have met them? And there's the auditing, which says, have you met the standards? Now, the reason we think we need all 3 of these is insurers will struggle to actually underwrite the risk and know how to differentiate a good risk and a bad risk unless there are a clear set of requirements that they can say, you know, yes, I understand these are the best practices. Are you meeting them? Which of them aren't you meeting? And that would affect your premium just like it would in your car insurance if you're driving recklessly. The auditing is essential because a standard without the auditing doesn't really help because it's very easy for companies to say, yes, I'm doing these things and self-attest, if there isn't the oversight to say, actually, yes, somebody has rigorously checked this and made sure that the standard is being met, it's kind of a— there's a trust gap there. And so this is kind of how we think about the flywheel. And critically, in the past, insurers have funded the standards and funded the auditors to actually become more robust and create that oversight. And they've created the incentives for the companies who are creating the risk to comply with the standards because they want the insurance and they want cheaper insurance. And so creating the incentives where you have a pool of capital over here on the insurance balance sheets funding the research, and you have actors who are creating the risk actually wanting to comply and wanting to have the insurance in place, we think actually aligns the incentives in the right way. Maybe one other thing I'll say on this is at the moment, this kind of debate has become polarized between let the AI developers cook and just let them commit to voluntary commitments and that'll be fine, all the way through to on the other extreme, we need top-down regulation, we need it yesterday, and it's got to be really strict. We actually think this is a neat kind of middle ground here where You're aligning the incentives because the insurers want you to actually grow, right? That's how they grow their premiums as well. So they don't want to have too onerous requirements. They don't want to have requirements that don't actually track onto risks because then no one will buy insurance from them. On the other hand, they don't want it to be too lax, and they don't want you to actually be able to break your commitments because they're the ones paying. And so this is a neat kind of market mechanism that aligns the incentives between actually progress and security.

And maybe we can ground ourselves in some of the historical examples and just work through them. So what the very first place we saw the incentive flywheel in practice is back in 1752 in Philadelphia, when Benjamin Franklin sets up the first fire insurance company in the US. The basic problem was that Philadelphia was growing the population 10x over that decade. And so people were packing houses closer and closer together. which created a lot of fire risks. And the fire risks, of course, would spread. No individual actor was carrying the entire risk because the entire city might burn down. The fire insurance company were obviously having to pay whenever a house burned down. So they're interested in reducing the frequency of fires. And they created building codes, which is in fact standards. How far should houses be apart? Can they be under large trees that might be hit by lightning or not, et cetera? And then they created the first fire inspection, folks that went into homes to just see, are they meeting these building codes as a way of actually enforcing it? And so it's kind of that very basic logic that we've seen pop up again and again. For example, in the year 1900 as well, when electricity comes out, again, houses start burning down and the Underwriters Laboratory funded by the insurers pops up to define the UL certificate, which you'll see in your light bulbs today. is a product security standard that they will then go and audit against. That's the kind of mechanism that we're trying to bring to AI.

Cool. I love the vision.

Hey, we'll continue our interview in a moment after a word from our sponsors. Where do we stand today?

I was doing a little deep research on the topic, and I guess my takeaway from what ChatGPT told me is that Mostly it seems like the harms that people anticipate from AI systems are covered under existing insurance policies, at least when it comes to the taxonomy of harms. Like, the kinds of harms are insured. People are sort of reasonably comfortable with the kinds of things they expect, but there's some ambiguity, I guess, around whether the fact that an AI might cause them as opposed to some other cause, could put something outside of scope. Um, and it seems like a lot of people right now are just living with ambiguity. I understand this is one of the things that you recommend that the industry clear up is like what's covered, what's not covered. How do you see that today? And what are the big points where we need clarification?

So the state of play today as it comes to AI insurance is that it's ambiguous about what's covered in the current insurance policies. For example, AI can help you create new kinds of cyber vulnerabilities. And of course, people already have cyber insurance today, but it's unclear whether AI-induced cyber vulnerabilities would be covered by the current policies because AI is not mentioned anywhere. And what we think is going to happen at a high level is very similar to what happened when computers came out. back in the kind of early 2000s. Again, leaking confidential data or having kind of downtime was not an entirely new set of harms. But when computers came out, they happened at a very different frequency and a very different severity. And there was the same question, is this just age-old insurance or is there something new here? And what happened was that over time, as these lawsuits came in to clarify whether things were covered, Cyber insurance was split out into a separate set of products because they need to be priced completely different and the products need to have a different structure. And we suspect that the same thing is true with AI here. One thing we know for sure is that the current insurance policies have not priced in AI risk because the insurers do not know how to price the AI risk.

And this just doesn't serve anyone. The companies who are insured and their customers are saying, I'm not sure if I'm covered. In the 2000s, when it was cyber, this took years to work through the courts, where literally insurers were suing customers and vice versa. And from the insurer's perspective, as Rune said, they've not priced in the risk, they're taking risk on that they're not being paid for. And that's going to show up in loss data, we expect, sooner than later.

Yeah, that's a really— I had a very, very brief foray into the insurance industry some years ago. And it strikes me that this is a really hard challenge for them because they like to do this sort of thing based on historical data, right? It's like we can look back at how many car crashes there are, we can look at how many homes burned down from fires. And with a couple of important assumptions, like that things will be relatively uncorrelated, you can be pretty confident that you can write a bunch of policies and don't shoot yourself in the foot and you'll probably be okay. But the same challenge is popping up with AI everywhere. I was just speaking to a group of educators last week and saying, like, I know you guys have been trained, and it's like central organising principle of the field that you want to be evidence-based, but you can't be evidence-based because by the time you see a study, you got 2 more generations of AI, and the kids are using something that didn't exist at the time of the study. Here we are basically with the same challenge in insurance, right? Do we have any sense right now for what the frequency of these things is going to be, what the distribution of magnitudes of harms are? And if people are— maybe we have some of it, but I'm interested in how much we do have. And then how does that translate into any sort of decision on pricing?

Yeah, there's a couple of issues here, Nathan. One is what you said. AI is just moving too quickly. It's moving faster than anything insurers have ever known. And so the slow rollout, let the data trickle in, pick it up in a few other categories, and then develop a product off the back of it, just won't work. If you were building an insurance product today based on historical kind of AI data, you'd be building for a pre-LLM era and not even thinking about the agentic risk. So that's one issue. I think the The other issue here, though, is they don't really have a mechanism. They're not set up to even collect that data. And so if they don't have a product and these losses are kind of happening in kind of small places, which is typically what we're seeing at the moment, that there was an EY study on this recently. Enterprises are having kind of million-dollar-size losses due to AI. Smaller companies are having in the thousands, tens of thousands of dollars. Insurers might not even be seeing this. And so they can't see it, they don't have the loss data from which to price. And so what we expect is there's kind of a small number of— rather, there's a long tail of small losses coming in, and then we're hearing and seeing the big ones. They're the ones that kind of make the press, and the rest of them are kind of being ignored both by the insurers and by the companies where they're kind of just writing them off. if you then think about how you translate that to pricing, it's just really hard for insurers to basically apply that model. And so there's a couple approaches we're taking here. One is, the nice thing about AI is you can actually do what's kind of known as red teaming. You can run evals. I expect your listeners are kind of very familiar with what that is in the AI context. But if you translate that to insurance, actually, It creates a massive pool of synthetic data, basically, for how this product is going to behave in the real world. And insurers can literally plug and play this into their pricing models where they say, well, I see that this failed the red teaming that I did in these particular ways. If that happened in the real world, I can estimate what the loss would have been because they know how to estimate losses from a from a data leak or a court case that would have happened. That's their bread and butter. And so they can take the data from the red teaming on both frequency and severity and plug that straight into a pricing model. The other thing that's nice here is you can create pretty clear parametric, as it's known in the insurance industry, pricing triggers where you can say, actually, I can know definitively did the AI behave as it was supposed to, or was this an out-of-distribution event? And by using that, you can then very quickly say, right, I'm going to pay out $1 million because this happened, and I'm just going to price it. I'm going to pre-agree the pricing, basically, and pay out rather than waiting for a long court case to come in and settle on that.

One other big consideration is There is always someone underwriting the risk, even when there's no insurance. So today, when, say, a hospital or a bank deploys an AI system, implicitly they are pricing the risk to decide, do they want to go ahead or not? And so although it's like an extremely kind of gnarly problem to really pin down the price, the most important fact is that this is happening every day, all the time, that people are trying in more or less sophisticated ways. And we think by and large, the rigor of the insurance industry is going to make it more likely to get that pricing right than non-experts who are sitting distributed across every enterprise trying to make these decisions for themselves.

Yeah, we're all self-insured, whether we know it or not, at the moment. That's right. You know, I guess one of the things that makes this sound really hard to me, if I try to imagine myself being an insurance company, is It feels like, again, with car accidents or whatever, I would guess that there's some sort of normal distribution of the damages, right? Like, there could be some outliers, but those will probably be relatively infrequent. And I can put most of the incidents in a pack, whereas there are other things in the world that are characterised by more of a power law distribution, like venture capital returns, for one. I would assume, although I don't know for sure, that bio incidents probably fall into this category where a lot of— I'm sure there have been many viruses leaked from many labs that just didn't go anywhere. But then you get one COVID and it turns into a global event. And my gut says AI risk might look more like that, in that it could be a few incidents that drive the bulk of the— the total weight of the losses. Is that right? Do we even know how to think about that? And how much of a problem does that pose for insurance? And then I guess, assuming my mental model is not way off, how do we start to attack that problem?

Yeah, the first answer is no one knows what the kind of tail risk distribution looks like because we have not yet seen it. From nuclear energy insurance, so nuclear power plant insurance that we can learn from here. So the same intuition applies here that probably nuclear failures are going to have a power law distribution where the real big failures will carry most of the losses. And in fact, that's true. In the US, there is in fact an insurance scheme for nuclear power plants. This is required by law. And the way they deal with these tail risks is that the government basically made a scheme where they said, we are going to require every nuclear power plant operator to have insurance. And we're going to require you to take liability no matter where in the supply chain things fall. So no matter whether it's one of your vendors who provided you with a faulty thing, you are responsible for all of this. You're going to take out insurance. But on the other hand, we are going to cap the liability. So if there is damage greater than $15 billion, this is going to come out of the government balance sheet. It's basically a government backstop. The reason why they're interested in this is you're exactly right. The market may not be able to carry these very, very big tail risks. Yet the government would really like there to be an insurance industry because it provides an important governance function. It means that there is now a market-based version to have independent third-party auditors with direct financial incentives to price the risk accurately and enforce best practices on the industry. Basically does some of the governance work that the government may not think that it has the capabilities to or is not going to be responsive enough to the market. And so there are some lessons there from nuclear for how do you get the best of private market governance while allowing the government to capture risk where the market may not be able to really respond to it. And I think this idea of capping risks is going to be critical again to AI, which is going to allow you to cover a broad set of risks up to some amount.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Let's go to the standards and the audits. You guys have put out this AIUC1 standard, the first standard that I'm aware of anywhere in the, at least Western world, for what to do when you're building and deploying an AI agent. Tell me about it. What does it contain? What does it require companies to do? And how did you go about assembling all of these?

Yeah, I can speak to the high level here. The core idea with AIC-1 is to take all of the concerns related to AI agents that slow down adoption and put them into one framework that outlines for each of the risks what you're supposed to do in enough detail that you can have independent third-party auditors come and verify whether a particular company is doing those things. And so the kind of design principle here is to go out and spend a bunch of time. We met with more than 500 folks across the enterprises that are deploying these systems today. This is security leaders, general counsels, et cetera, across banking, health, et cetera, and figure out what is it that's keeping you up at night. And the answers will not surprise you a ton. It's going to be things like data leakage at a new scale. It's going to be, are we going to provide incorrect medical advice to our customers? Are we going to have jailbreaks and prompt ejections? Is there— are we going to be discriminating against our applicants, et cetera, et cetera, baking that into one framework? The goal is when a company passes, meets the standard, the enterprises that are deciding whether to adopt or not will get the data foundation for them to understand whether they can trust that particular agent or not.

The way it's used in practice, Nathan, is often you'll have AI agent builders selling to large enterprises. Think of the large banks, the large technology companies. They want a third party to have vetted the tool they're looking to buy, and they ask them for their AIUC-1 report. So a large company is looking at a AI chatbot developer, say, AI customer support developer, and I'll say, share with me your AIUC1 report that has in it all of the detail on what are the data and privacy controls they have, what are the security controls and implementations they have, what are they doing to protect customer safety, what output filtering do they have, for example, on how does it perform against bias, and is it tracking its role correctly and not responding with anger or things like that? Or is it responding appropriately to anger? If a customer really wants a refund and it's saying, you must give me this, and it's going to respond correctly, what are they doing with reliability? Do they have a groundedness filter in place? Are they appropriately mitigating hallucinations? What are their human-in-the-loop practices? And what's the human oversight? And what are the societal risks that they're tracking and covering? And so in one report, the enterprise can look at it and say, It's been third-party tested, I trust it, and I can see all the detail that I might need and preempt all the questions I have.

Yeah, gotcha. So 500 people is a lot to get on board with anything. How easy or difficult did you find it to boil all this down into a sort of consensus standard? I'm sure there must have been some things that were like, contentious one way or the other in terms of like, well, including that is overkill or not including that, you know, leaves, you know, this risk unaddressed. How did you know where did— where is the line? Like, what's the frontier of debate in terms of what is, you know, solid enough to be in the standard and what's still kind of, you know, next set of considerations?

We found it to be remarkably uncontroversial for most of the topics. There is a lot of interest in just creating one shared language. And a lot of the concerns, say data, it's actually mostly just like churning through the issues and providing clarity on what should be included rather than debating where the line is. In part because the standard doesn't exactly specify what should be done, but it's a lot around disclosure so that any particular enterprise can decide whether this is okay to them. Given this may differ radically whether you're a hospital or whether you're a retailer, they just have different risk preferences. And so if you get some of those design principles, it's actually relatively easy to get people on the same page. The frontier of kind of controversy is probably related to some of the more societal harms. So should we check for like whether a particular product enables folks to create cyberattacks at a much grander scale than before, which may not really be kind of that top of mind for any particular enterprise that's deploying it and might seem a little bit overkill. And so that's where we've had to kind of make some judgment calls about what will best serve adoption, not just for any particular individual enterprise, but for the industry as a whole. Because as you were mentioning, when it comes to, say, nuclear, one incident can kind of chill a whole industry. And those are some of the effects that we've started to include in the standard. Still focused on like, how do you do that in a pragmatic way where you can actually assess it and you can get it done without having to spend tons of dollars to check whether any random bug coding tool is going to bring down a whole industry. But we do think it's important to keep So I guess, how does this look in practice?

I was just talking to Andrew Lee a couple of days ago. He's launching this new product called Tasklet, and it is a very bet on the models agent platform where his idea is like, forget all this sort of step-by-step flowcharts and all these kind of you know, this field maps to that field, as you would see in a traditional automation software experience. And he's just like, we're just going to bet on the models to figure it out. We're going to give them all the tools they need, we're going to give them the memory, and they're going to keep getting smarter, and they're going to go figure stuff out. And so I asked him, like, would you like to buy insurance? And he said, yeah, you know, we actually have had a really great record so far that, like, I thought people were going to be really freaked out or that we might see, you know, weird rogue behaviour from the agents from time to time. Fortunately, they haven't. But still, he was like, yes, I would like to buy insurance, if only because then I can take that insurance to my prospective enterprise customers and show them that we've got this, and that should definitely ease our go-to-market process and speed it up. So perfectly consistent with your thesis there. But if we dig into it a little bit more, who do you think is primarily pushing this? Is it coming from an enterprise demand side or a vendor that wants to prove their qualification side? And how much— is there any way that we can quantify what we're getting? You do these various things, but what was the risk before? What is it after? Do we have any way of sort of saying to an enterprise buyer, okay, you're going to buy this vibe coding tool and we're going to bound the risk in some way for you? We can make some level of confidence assertion that you're not going to have more than this big of a problem. What do those final statements that people are buying into look like?

We see demand for what we think of as a confident infrastructure, both on the developer side and on the enterprise side, because with or without insurance and standards, there is still risk. The enterprise is still nervous. They're still asking tons of questions of the developers today and sometimes just not buying because they can't get to confidence because there's too much ambiguity. So the enterprises definitely would like to have clarity of one language, one standard, one report that they can trust. But the flip side of this, as you're saying, is that currently it's incumbent on the developers to create clarity individually, to earn the trust, to earn confidence. There aren't a whole lot of tools available. They can write public blog posts on their security posture, and that's great. But there is, of course, they've graded their own homework. Everyone thinks that their security posture is great. They will sometimes reach out and make financial guarantees off their own balance sheets, and that can be— frankly, I think that's a great way to do it. But again, their balance sheets may not be able to carry it. So we actually see like the The most forward-looking developers are as interested in creating this kind of confidence infrastructure as the enterprises, even though of course the enterprises are kind of the ultimate beneficiary of it. To your point around like what's the impact on this on the risk, probably the best way to look at this is that when we go and certify a particular AI developer, we run a set of evals. That finds a set of vulnerabilities. And we'll often do that in multiple rounds. So in the first round, we tend to find quite a lot of stuff that they themselves are very interested in solving. So we often find that sometimes you can get up to a 25% failure rate on certain kinds of attacks against the company, which is just obviously not what they want nor what the enterprises want. And we'll then with the help of the standard, help them implement, whether that be a groundedness filter or monitoring solution or a content moderation filter, and see those same failure rates. And we can often see them drop by 90%. The honest answer is also there are no panaceas. AI security as a field and the safeguards, it is a work in progress. It is an open research problem, but there are some pretty basic steps that they can take. And we can see that Put it quantitatively from our first round of red teaming to the second or third time round of red teaming. And that is the way that we get a little bit more confidence that through this process, the AI companies that are deeply hungry for improving actually will improve in a quantifiable way that then can they convey and communicate to their customers to earn confidence, which is ultimately what allows them to invest in this stuff when they have a million things that they could be investing in.

I wanted to do just one kind of double-click on the kinds of companies that you're working with. It seems like a sort of tacit assumption of most of the conversation so far has been that we're focusing on application developers who are presumably, in most cases, not training their own foundation models. They may be using a commercial API, they may be taking an open source model and doing something with it, but they're kind of providing the wrapping of that. And I don't mean that in a derogatory way, but obviously There's a lot that goes around models to make them useful for enterprise purposes. But there are other layers of the stack that one could think about wanting insurance for or wanting to do similar audits on. Are you guys doing stuff at the foundation model layer or even deeper at the physical data centre infrastructure layer as well? Or are we limited for now to the application layer?

Yeah, we think this model applies at all 3 layers, right? So if you think about this kind of alignment of incentives, research into security, and how that unlocks progress, it's clearly, you know, it's clearly most acute at the application layer at the moment, because these are the companies who don't have large enough balance sheets to kind of make large promises on their own. It's also to some extent where the risk is headed in terms of, you know, they're the ones building the agents and deploying agents today. even more so than foundation model developers, for example. But if you think about data centers, we spoke recently to a chair of the audit committee and risk committee at one of the top tech companies who was saying, well, we're investing an astounding amount of money into data centers right now, and our public shareholders don't want to be bearing that risk of what happens if the data center goes down, which could be anything from actually, you know, just wanting uptime through to there's an attack on the data center, or, you know, there's an issue with the, uh, with the security of it and, and it gets breached. And so, you know, they're saying, can we buy insurance for that? And can we have assurance from a third party that these risks have been well managed in a way that wasn't the case, you know, historically with, with kind of data centers they've bought and invested in the past? Um, Now, that's obviously its own standard, right? AIUC1 doesn't currently cover the risks for that. So what we expect we'll do is we'll run the same process that we've run with AIUC1, where, as Runa described, we talked to 500 executives and understood what are the risks and what are the mitigations. And we talked to AI experts. We'll do the same thing on data centers, where we bring in the right folks who are really leading the thinking on this, both on the kind of buyers and the builders of them. We'll codify the best practices, we'll figure out what are the ways in which you actually confirm the security is in place, and then we can insure off the back of that. And you can imagine exactly the same applying for foundation models.

One way to think about it is also a game of earning insurers' confidence over time. So start with the risks that might cost millions of dollars or tens of millions of dollars, move on to insuring risk that might cost you hundreds of millions of dollars or low billions of dollars, and then move on to the tens of billions of dollars eventually. And think of it as a ladder up. And the agentic place is where you have a lot of relatively small incidents, which gives you more data, allows you to build confidence that you have good models, and then kind of slowly earn the confidence to move up the ladder.

Yeah, it makes a lot of sense to me also just from the standpoint of What I see people doing, like who seems to be doing a good job and not such a good job at this stage of the game. I wouldn't say I know too much about the data centre layer. My sense is that there's a lot of new concerns here for them, for sure. But at a minimum, they're well aware that they're putting tens of billions of dollars into the ground. And they have very obvious incentives because they're well aware that they're self-insuring. The foundation model providers strike me as being remarkably, not with some notable exceptions, remarkably good about this stuff. And again, we could question whether the whole enterprise is like a bad idea, big picture, but they're doing a lot of work to taxonomise risks and to build defence in depth. approaches and all that kind of stuff. And then what I see at the application layer, I guess it's starting to get better, but certainly for most of the last 2 years, what I've seen is like weekend hackathons turn into businesses. People never really gave this any thought before they kind of built something and launched it, and they just have zero, in many cases, zero protections in place.

whatsoever.

So I do think from the standpoint of like, just where's their low-hanging fruit, you know, where is there like a lot of needless risk that, you know, enterprise buyers like legitimately should beware and should be asking these kinds of questions. The app layer absolutely has a lot of low-hanging fruit, and there's a need to be able to identify and separate, you know, who has done a good job and who's really, even if you can't take the risk to zero, like who's made a best effort to get to a place where they can responsibly be deployed in an enterprise or a high-stakes setting. And who hasn't? And there are a lot that haven't. Would you agree, disagree, add any colour to my sketch there?

I probably agree with the assessment. I think one piece of colour to add is that at the agentic application layer, the promises being made to the enterprises are much thicker in some ways, where if you buy Claude for your own purposes and you go to Claude.ai, the promise of what Claude will help you to do isn't very specific. They're not claiming that they will give you correct medical advice 90% of the time. They're just saying, here's a helpful tool, use it for whatever, it will hallucinate. If you move into a space like say, customer support, the kind of public claims that these companies are making are like, we will resolve X% of your tickets and kind of implying also in those sales processes, we will not create massive brand disasters for you because we're going to, you can trust us to interact directly with your customers. So the thicker the promises, the thicker the insurances and insurance also needs to be. And so that's a little bit of color for why the agentic application layer is the is the place to start. One other piece of color is that the distinction between foundation models and agentic applications is blurring. Take an example like Claude Code. It's clearly not a foundation model. It's a system of models at the very least with a bunch of kind of different things going on there, a bunch of integrations, a bunch of access. And so this line is really blurring and we think like agents is the most helpful frame to think about it. And that's where some of the application layer companies got there first. But the companies you think of as foundation model companies are quickly moving into that space.

So let's talk about then the nature of the audits that you guys are doing. You mentioned red teaming. If I'm an application developer, and I am, like, I've got Waymark, we are in a blessedly low risk environment because we help media companies make marketing assets for generally small and local businesses. And my standard line for our team has always been, if somebody breaks into our system and watches all these TV commercials that we're making for local businesses, nothing bad will happen. That's a great comfort that we have available. But let's say I want to do this. Like, you can go poke it at our thing from outside and prompt it or ask it to do hateful or whatever commercials, and you probably could find some gaps if I had to guess. What more do you do? You get under the hood, you read source code. What are all the different angles of attack that you bring to bear on figuring out the many ways an application could go wrong?

Totally. The foundation is to start grounding ourselves in the real-world incidents that have happened to be like, what is it that we're looking for? So, we have a database of incidents, real-world incidents that have happened, that that inform what are the things that we should be checking for. Some of those are things that you might think of as kind of very concrete headlines. BBC says, wow, Air Canada's chatbot hallucinated a refund policy. Great. We should clearly check for hallucinations that have financial implications. Some of them might be slightly more speculative, like a research paper that shows under these kind of experimental circumstances, AI chatbots will try and blackmail the user. We also think of as like an incident. It's just slightly more speculative. That forms the foundation for the audit. We then try and abstract that into a taxonomy of both what are the harms we're checking for and what are the kinds of attack vectors that are relevant? What are the kind of latest research into jailbreaks? What are the kind of latest examples of applying the prompter on Twitter for different ways that these models break? And that kind of becomes the checklist of things that we want to check for. The way we check it is broadly, For each of those kind of harms and risks, we specify what are the technical, legal, and operational safeguards you must have in place and what are the evals you need to run. The technical safeguards could be like, oh, you must have a groundedness filter that reduces the chance of hallucinations. Or it could be you must have a plan for what happens when shit hits the fan. What is your AI incident plan? Those get checked either by looking into the code base, reading management policies, et cetera. And then secondly, there's the technical testing element, the red teaming, which really grounds everything. It's not enough to have an AI groundedness filter in there if it doesn't work. So that's where we go and take all these scenarios and try and break the chatbot to see how effective is that. And then the output of this is, you can imagine a report. that says for each of these technical, legal, and operational safeguards, what do they have in place today? And for the eval, displaying back the results of what we found in a way that both helps enterprises make sense of it, but also leaves room for them to make the final judgment call with some level of confidence of whether there is a go or no-go.

And so one way to think about this, Nathan, is there's a trust and verify approach that we take, right? So what are the policies you have? Have you implemented the safeguards? And can we verify they work? So again, you could imagine exactly how this would then also translate to foundation model developers, whether it's like they have these policies on what safety testing they'll do and how they'll implement various safeguards. How do we actually verify that's happening?

Yeah, you mentioned that there's multiple rounds of this process, and I think it's probably worth talking about why that's important. I was just reading a paper from Nicholas Carlini, Sander Schulhof, and others. I'm sure you've seen it, The Attacker Moves Second. And the basic notion of that paper was, if you take the attacks as static, and you say, okay, I'm going to tweak my defences to mitigate those attacks, you can do that. And you will mitigate those static-defined attacks that you were trying to mitigate against. However, when you then deploy those defences into the real world with potentially adversarial attackers coming at you, they find ways around those defences at a remarkably quick rate. And so what they found in their study was that human red teamers were basically 100% undefeated. There were no defences that were totally impermeable to human red teamers. So how do you go through a sort of iterative process to not just have this sort of kind of security theatre of like, well, we had this one, we had one shoe attacker one time, so now everybody takes their shoes off at the airport, and then we tell ourselves we're good, as if there was no other way that you could get something onto a plane.

Right.

So, yeah, how do you think about that iterative thing, and how do you know when to stop? Because this could obviously go on. Forever too, right? How do you even decide when you're getting to a point of diminishing returns or a point where you've hit some sort of good point on a Pareto curve?

Yeah, there isn't like a single neat answer. Ultimately, this is a judgment call. The important thing to know is we do a few rounds every time we do the auditing. And then to get certified, the companies commit to doing quarterly audits. They will change their product. The world's incident will evolve. There will be new research papers into new ways these will fail that we then kind of continuously incorporate into this body of tests that we run, that we run in each quarter. In practice, it becomes kind of a cost and effort Versus confidence. And these are just, in fact, not at all new considerations. Cybersecurity has the exact same thing.

No one—

with enough effort, every system is breachable. And ultimately, that's pragmatic. And over time, the cost of running good audits goes down as intelligence becomes cheaper, but the set of attack vectors probably also grows. So there isn't, in fact, a very neat answer for where you stop. Ultimately, this is guided by kind of the market. What are the level of assurances you want and what are the things that people are willing to pay for to get that assurance? Which is probably like, it's a messy answer, but the actual pragmatic answer for how most of that stuff gets settled in every domain today.

Yeah. What does this look like in terms of your business model? Who pays you? I suppose the developers pay you. How much do they pay you? How does that depend on what sort of product surface they have? What does that quarterly update process look like? Is it the same thing as the original, or is it more streamlined as you go forward? What's the overall customer story for the AI underwriting company?

Yeah, totally. So The company that is building their system pays. That can either be what you think of as disruptors in the space, these AI-native companies, but it can in fact also be enterprises that are developing something in-house. You'll see a bunch of enterprises that are building their own customer support agents because they think that's a real moat and they want still to have confidence before they deploy that. So it's the companies that develop the systems that pay. that fundamentally pay. They pay on a yearly basis. The certificate lasts for 1 year. With that, you're required to have these quarterly technical tests. And the pricing depends, as you're saying, exactly on the risk surface, the product surface. And also, of course, this also ties to value. Bigger companies tend to be able to generate more revenue out of this. They also tend to have a bigger risk surface. So it scales a lot. Other parts of the business model that are interesting is the same companies that are interested in getting certified overlap a lot with the companies that want to have insurance. And for the same purposes, they want to take risk off the table from their customers and deploy faster. Or when it's the enterprise, they want to just reduce the risk so they can deploy faster. And that insurance as a business model also works on it. You typically buy an insurance policy on a yearly basis for a yearly fee. So you can imagine those laying on top of each other. And then we're experimenting a little bit now with, imagine an AI-native company that wants to not just get certified for the product in general, but also want to certify specific deployments to specific customers. They want an extra layer of guarantee. They may want to know that, oh, I'm a bank deploying this AI voice. customer support chatbot. I want to know that with my specific customer support policies and my specific customer sets and my specific policies, that this will also work. And so that's another way to kind of get more precise in the value. That's where the AI disruptors are really willing to pay because it's going to unlock a contract. But it's also very clear that it's the maximum level of assurances an enterprise can get when it's done literally in their environment.

Can you tell a little bit more about what the requirements are from the customer side? And I would be interested to know if you're willing to share tangible price ranges. I assume this sounds like it starts in maybe 5 figures and probably goes into the 6 figures depending on how big the product surface is. And do companies need to make team members available to you? Do they have to Or what beyond the sort of desire and willingness to pay do they have to bring to the table?

Yeah, before we sign a contract with anyone, we do a gap analysis. So we take the standard and outside in, we take what is the kind of evidence that's publicly available that gives us a sense of where do they stand against the standard. And we then do an in-depth review, like row by row, where do you actually stand today? That gives us some kind of effort estimate. What's the amount of engineering effort you need? What's the amount of non-engineering effort you need? And also from our side, what is the total scope here? And that's where we together come in and we're like, what's the price going to be here? The range I mentioned is broadly correct. And that kind of gives everyone a sense of what are the costs involved on both sides. And that's where when they then commit, they commit for a year. At the end of that year, they can decide whether they want to recertify. This is very kind of in line with how they already think about certifications in the cybersecurity space, things like SOC 2, ISO standards. So really, a gap analysis is the core thing that says to everyone, what's the effort that's involved here? What's the cost going to be? And we can do that, frankly, within 24 hours.

And what we find, Nathan, is anywhere from— our largest customer has thousands of employees, they've been running for years. for them, it's like a small handful of hours of actual engineering lift or security team lift, actually, in their case, and just evidence collection for us. Through to our smallest customer, we started working with them, had 2 or 3 employees. And, you know, we needed to work with them in a much more hands-on way to actually implement the security measures that are required in the standards, remediate the issues we found through the testing. And that was obviously a much more involved process. And so it's kind of hard to give a stock answer without doing that.

Yeah, well, it seems like something everybody's going to need, right? In a similar way to whether it's me as an individual driver of my minivan or a nationwide trucking company, everybody's going to have to have some sort of insurance to offer it, presumably before too long, I guess.

Do you think that's true?

class of AI app that just never gets into this for whatever reason? Or do you think this does ultimately become something that really everybody has to do in the same way that drivers do?

Yeah, I might separate the insurance from the third-party testing for this. Right. I think there is, for almost every kind of valuable use case you can imagine, there is a need to have somebody who is not just marking their own homework actually test it and say it is secure to ingest customer data or companies' kind of private sensitive IP and data and make sure that's all being protected, et cetera, et cetera. And then I think on the insurance side, I think that's where you could imagine more of a bifurcation. Traditional software SaaS has been sold on kind of a have a nice day basis, right? Often those aren't insured. You might get some service credits if something goes wrong, but you're not typically not going to get more than that. As you said, cars and houses, obviously everyone carries insurance, and in lots of cases, this is mandated. You could imagine one version of this is a similar bifurcation where the highest-risk deployments and the places that are creating externalities also really should be insured because that's where you want an extra layer of protection from a strong established balance sheet. to provide the financial coverage. Whereas, you know, if I take your example of producing media copy, you could say, actually, the risk here is manageable, and it doesn't need insurance.

Yeah, makes sense. I was briefly in the financial services industry, as I alluded to, and one of the things that I happened to have, kind of another one of these Forrest Gump extra roles in an important scene in recent history was the kind of problem of the credit rating agencies. And basically what happened in the late 2000s is that these companies were, in a sense, captured by their customers. They had the incentive to say things looked good. They didn't have the incentive to turn over too many stones. And so we ended up with all of these very highly rated bonds going bad. How do you think about preventing that outcome? I guess there's probably multiple dimensions of it. One is just like, how do you make sure that your audits stay weird enough? It almost seems like you need heterogeneous cognitive profiles on your team to make sure that you are bringing weird and off-model ideas to the table as often as possible. But then there is this incentive question too, right? Like, at some point, as this gets normalised, there's going to be some pull, or there might even be some competition. That's, I think, really where the ratings agency things went bad, when it was like, well, if you don't rate us high, we can always go to the other rater and they'll rate us high. So then it's like, well, geez, well, what do we do, right? We're going to lose all our business by giving our own customers bad ratings. That's a tough line to hold. So you're trying to create a race to the top, but there is at least some precedent for this sort of thing ending up in kind of a race to the bottom. How do you think about making sure we stay on the right side of history here?

The core thing that's going to create the right incentives here is the insurance that gives us the skin in the game to not just lower the standards. When you take the Moody's example, Moody's does not have financial skin in the game outside of trust. So when the financial crisis hit, there's not a direct financial mechanism that forces them to internalize the risk that's being created there. And when you look back at, for example, the UL certificate that's on all of your electrical products, which I think of as broadly a very successful instance of standards, That was created by the insurers. It started from the insurers having data on what's going wrong and having the incentive or the direct incentive to make sure that the standard covered that. So we think the insurance piece creates the skin in the game to prevent this race to the bottom. And there's a lot to learn from history there.

And in terms of how we keep up with it and keep up with the frontier, Nathan, this quarterly process is essentially just really involved from our side. We have a mix of the actual people we're certifying and the agent builders, right? Because they're on the ground saying, here's a new safeguard that's really working well for us, right? Great, let's feed that in. We have a range of academic researchers and other people kind of building safeguards commercially, giving us input, right? Folks from Stanford and University of Illinois and Haze Labs and Grey Swan and Virtue AI. And then we have the enterprises. We've just launched our enterprise consortium, which is a mix of security and risk leaders from Fortune 500 enterprises who are actually also on the front line of the deployments and saying, here are the risks I'm worried about and the risks I'm seeing. And all of these people come together to give us input both on what's missing, where's the frontier headed, what are the places where AUC-1 isn't keeping up with that, And then how do we write the precise requirement in a way that will capture the spirit of what should be in here? And so really, it's not just about our team here, which of course has lots of these capabilities, Nathan, you're describing, but it's really about how do we mobilise the whole ecosystem around this and bring that together.

Do I understand correctly, just to rewind to the insurance piece and the incentives, are you planning to actually provide insurance? It sounds like you're both going to, if I'm understanding correctly, You're going to have skin in the game at the insurance payout level.

Is that right? Yeah, we'll have skin in the game, Nathan. The way this works is we'll become a managing general agent. It's the kind of insurance term for it. And in practice, what that means is our payouts— so the insurers will pay us, and our payouts will be tied to the actual underwriting result, the profit that they get on the insurance product. So if there are large losses, we don't get paid, or we get paid much less. If, in fact, our risk assessment was good and there's fewer losses, we also benefit from that financially.

Interesting. And if there are large losses, the insurers will not want to work with us, so we will not be able to— we'll go out of business.

Yeah, gotcha. Is there a big difference between that and writing your own insurance? Is it just a matter of size of balance sheet that causes you to structure it this way? Or do you imagine, like, at some point building a giant balance sheet and underwriting your own policies and doing your own payouts? Or is there a reason to let the traditional insurance companies handle that indefinitely?

Yeah, insurance and actually reinsurance is often where the risk sits, is really a cost of capital game. It's a kind of risk aggregation game, as we were talking about earlier. And so there's a lot of benefit there in being diverse across different risk lines, across, you know, selling through brokers who are selling kind of multiple products, et cetera. Really, we want to focus here. You know, we're focused on the AI risk. And so for us, this gives us the benefit and the flexibility to go out and write and structure insurance products, knowing we've got insurance partners, you know, some of the most established insurers in the world who've never failed to pay out a claim. They've been around for hundreds of years. This is their business, and we're really excited to be able to write with them rather than having our own balance sheet.

I've been following this SB 813 proposal for a while. The idea here, as I'm sure you guys are very well aware, is to create some sort of private governance constellation of options where companies could opt in to abide by a certain standard. Maybe you could even be one of these. multi-stakeholder regulatory organisations, I think they're called. And in exchange, the companies get some liability insurance. There's also a bunch of other proposals for liability reform or changes in all sorts of different ways. Do you have ideas on policy? Like, what policies do you think would work best? What are most compatible with the business that you're trying to build? Are there any that are in direct conflict with it? What do you think is the future of that?

Yeah, from our perspective, anything that promotes the creation of a market here and market-based solutions is the key. You know, for the reasons we talked about earlier, that's the thing that aligns the incentives, because if you have a free market of oversight, you're going to actually incentivize robust risk evaluation without hampering growth. And that's really important, right? We don't want this to become too onerous and crowd out the startups and crowd out innovation. And so there are plenty of policy ideas, Nathan, like you mentioned, that do allow for this and do promote the creation of best practices and third-party oversight. And the other thing that's important though, which we think SB 13 didn't do, is actually force the companies who are creating the risk to own it. We don't want to create safe harbors where you have too many free passes where you can create risk and then you're not punished for that and you're not held to account for that. And so we think that's a fine line, and we think insurance actually allows you to internalise those externalities and actually take on the risk you're owning while still creating the market around it.

Last question, who all is involved? I understand there are quite a few well-known companies that are starting to buy into this. So maybe give us a little overview or name check of who's hopping on the train?

Yeah, on the startup side, we've had some— an awesome group of like kind of founding technical contributors from various different spaces. We work closely with Ada and Intercom in the customer support space. We work with Cognition on code. We've worked with a company called Recraft in image generation. So trying to really look across the spectrum of the different kinds of modalities and deployments. And maybe, Rajiv, if you can speak a little bit more to the kinds of companies that are in the enterprise consortium.

Yeah, so we've got risk and security leaders from financial services like JPMorgan Chase, for example, who are really thinking about how do we adopt AI across the bank globally, not just in specific kind of verticals. In tech, the CISO, the Chief Information Security Officer of Confluent, for example, is helping to shape AEC1. The Deputy CISO of Anthropic is kind of deeply involved. And we've got others from healthcare, from kind of retail, etc. So it's really trying to represent across industries, actually also across geographies. We've got folks from Asia, from Australia, from Europe, and really bring this together into one framework and make sure it's as strong as it can be.

Cool. Well, we're out of time. This has been great. Any last thoughts you want to leave people with or anything that we didn't touch on you want to make sure to mention?

I think this is pretty comprehensive.

Love it. Cool. Well, Rune Kvist and Roger Vitani, co-founders of the AI underwriting company, thank you for being part of the Cognitive Revolution.

Pleasure.

Thanks for having us.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts where experts talk technology, business, economics, geopolitics, culture, and more, which is now a part of a16z. We're produced by AI Podcasting. If you're looking for podcast production help, For everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And finally, I encourage you to take a moment to check out our new and improved show notes, which were created automatically by Notion's AI Meeting Notes. AI Meeting Notes captures every detail and breaks down complex concepts so no idea gets lost. And because AI Meeting Notes lives right in Notion, Everything you capture, whether that's meetings, podcasts, interviews, or conversations, lives exactly where you plan, build, and get things done. No switching, no slowdown. Check out Notion's AI meeting notes if you want perfect notes that write themselves. And head to the link in our show notes to try Notion's AI meeting notes free for 30 days.