Nat Friedman Leads $15M in Superintelligence Insurance Company | AIUC

No one really today knows how these AI agents fail. You might've seen OpenAI launched their ChatGPT agent and Sam Altman on stage puts out a big caveat, like, don't use it for anything important. We don't yet know how this works. Certificates and insurance is a way to build confidence for companies to adopt AI. What it does is we've built a standard, AIUC-1, that outlines what are all the things that can go wrong with an AI agent. And for each of those risks, what are the things an AI company should do? To mitigate that risk, we are coming out of stealth, having raised a $14 million seed round from Nat Friedman, Emergence, and Terrain.

Your investors are super ingrained into the space, especially Nat Friedman. I mean, he's now a part of the superintelligence team at Meta. What kind of advisement are you getting from them? Because you're an underwriting company, that does insinuate you're a financial institution. How do you put dollars against this? Could you break down what is a kill switch in AI?

It might be what's called a loss of control scenario, where basically the AI agent becomes so agentic that it starts to take action that we no longer want. And so a kill switch is like—

Runo, welcome to Sorcery!

Thank you. Thanks for having me.

I think we have some big news today.

We do.

Can you share this big news?

We are coming out of stealth, having raised a $14 million seed round from Nat Friedman, Emergence, and Terrain.

Wow, this is big. How big was the round? What was the process like?

It was $14 million. Um, it was a very quick process. We got an inbound term sheet preempted, and that prompted us to look up. I knew I wanted to work with Nat, got introduced to him. A week later, we had a term sheet. And we got it done.

That simple?

Yep, pretty much.

Okay, given that, I think we should understand more about how we got here, how it became that simple.

Yep.

So what exactly is your background? I know you're very impressive.

Most relevant to this conversation, I started working in AI when I joined Anthropic back in early 2022 as the first person to work on product and go-to-market. This is before ChatGPT. Is there that moment in time where it just become obvious that something radical was happening here, but the world just did not know yet? And I joined Anthropic to help think through, where is the value going to accrue? How does Anthropic sit into that? How are businesses and individuals going to interact with this AI?

And so now you're working on the artificial intelligence underwriting company.

That's right. Bit of a mouthful.

Bit. Of a mouthful. Very intense.

Super fucking intense. Yes. Yeah. We insure and certify AI agents.

So what does that mean?

Basically, no one really today knows how these AI agents fail. You might have seen actually yesterday, OpenAI launched their ChatGPT agent, and Sam Altman on stage puts out a big caveat, like, don't use it for anything important. We don't yet know how this works. It's kind of like if you're ordering a burrito and you're not that hungry, Go ahead and use this. This is an awesome tool. But when you start putting this in meaningful context, if you run a hospital or a bank or any kind of big business with real customers, no one really knows whether they can trust any particular agent. And certificates and insurance is a way to build confidence for companies to adopt AI. What it does is we've built a standard, AIUC1, that outlines what are all the things that can go wrong with an AI agent. And for each of those risks, what are the things an AI company should do to mitigate that risk? And spelling that out in enough detail across hallucinations, jailbreaks, all these terms that people have heard about, that you can have an independent third party go in and audit a company and test that agent to see how likely it is to fail. And if you do well, you pass the certificate. The insurance comes on top of that. If you meet the certificate, you may want to go a step further in taking risk off the table from your customers.

Yeah.

and insure them in case it fails. So if it hallucinates a refund policy, if it spews Nazi propaganda, if it leaks your customers' data, if it discriminates against the people you're hiring, the enterprise is going to be on the hook. And so everyone's asking, who's going to pay? And that's where insurance can step in and help pay the bill.

How did you come to this model?

Yeah, so it started with a, like a deep trip into the rabbit hole and like throughout history, how have markets solved these collective action problems? And the most interesting example starts back in 1752. Ben Franklin, all-American hero, um, he's living in Philadelphia at the time. Philadelphia's population was growing 10x over that century and the houses were just getting closer and closer and they were just burning down frequently. So the whole city would burn down. And Ben Franklin wanted to solve that problem. So he started an insurance company, uh, the first fire insurance company. And all of a sudden, he was on the hook financially when houses burnt down. So he had a direct financial incentive to go and reduce fire risk. So they asked themselves, how are we going to make sure that all the houses are built in a way that makes them less likely to burn down? They built the first building codes saying like, oh, how far should houses be apart? Can we make sure to not build houses under trees that are likely to catch lightning? And then they wanted to make sure that people obviously met these standards. So they built the playbook. That probably wasn't the term he used, but the playbook for fire inspections, really the audit mechanism. So this triangle between insurance, standards, and audits was something that he stumbled into. We call it the incentive flywheel. And that then shows up again and again throughout history. So when The light bulb came out starting like around the year 1900. Uh, light bulbs also led to the houses burning down. The insurers again were like, we don't want to pay for this. So if you look at any of these light bulbs, they will have a little UL certificate, the Underwriters Laboratory funded by the insurers that like specified how do you need to test light bulbs and toasters to make sure they don't burn down. Later on, cars, uh, killed a lot of people as they were getting really like Everyone needed their own car post-World War II. And it was the insurers that were taking the charge on looking at the data for what's causing accidents, figuring out what are the interventions that we need, what is the kind of car crash testing we need to incentivize the car manufacturers to build safer cars. And a lot of their research is what then eventually became embedded into laws like seatbelt laws, airbag adoption. Before that was law, that was often incentivized by the insurers. Like you get cheaper insurance. If you have a car with an airbag. So that model of insurance, building the standards and running the audits to both build confidence in technology and incentivize security is really battle-tested. And that's what we think is missing when it comes to AI.

I think it would be good to take a step back. AI has evolved so fast. Like we really, really weren't talking about agents in like a tangible aspect until maybe a couple months ago. Like now there's real applications and usability with it. So I'd love to get your perspective of the evolution of AI and AI safety. Some companies baked it in from the start, some just progressed and were all gas, no brakes.

Yep.

So what was that evolution like?

Yeah, I think when you started all the way back in like GPT-2, it's kind of obvious what was coming. Like already at GPT-2, you could start to see what GPT-3, 4, the agents, what are they going to look like? And there are kind of 2 schools. One was like, there's no risk today. GPT-2 was like, literally could not spell its own name. And so like, there's no risk, therefore all gas, no brakes. And then there's another school that's like, we can obviously see this is going to be a really big deal in short order, they're going to be able to help people hack. They're going to be able to help create bioweapons. They're going to be able to do all these kinds of things. And we need to think about that in advance. And that debate kind of played out for a long time without really any evidence either way, because it was just too early as the models were just getting better and better. And now we're getting to this point where everyone agrees pretty much that this is getting pretty serious. So You have companies like Anthropic have always been in the camp like, hey, this is really risky. Companies like OpenAI have gone a little bit back and forth on what they think. And with their release yesterday, they're saying like, wow, now the risk levels are really high. They are proactively sharing that with the world, that they're now enhancing the security levels. So I think we thankfully haven't had the big incidents yet, but everyone's agreeing that now they're sufficiently good that they can do real damage. When the models are not good, They're not going to do any damage, but as they get really good, we're going to delegate real work to them, real tasks, real critical systems. And no one has sufficiently good answers to say, oh, there's no risk here.

Right. So there's companies like, I call it super safe superintelligence. I don't know why.

Extra double safe.

I just add more supers to it.

Yes.

There's companies like that. Ilya takes it super seriously. Again, super. And then with Anthropic, maybe we could start with Anthropic and then go to SSI. They have different levels of safety. So what do those levels mean? And could you break those down?

Yeah, so they call them security levels, security level 1, 2, 3. Really, it's a process of saying when models get sufficiently good and therefore sufficiently dangerous, we need to match that with the levels of security we're implementing. What are the level of tests we're going to run? What is the level of cybersecurity protection we need to have in place? What are the kind of policies that match each of these security levels? And so we're slowly climbing through the temperatures now. And Anthropic, you were talking just before we got on here about whether security level 3 is going to be a thing. And I think the short answer is, yeah, it's just a question of time. And is it going to be this side of Christmas or after? I don't know yet, but it's really companies' ways of Matching proportionally the risk with the mitigations.

And then what about Super Safe Superintelligence?

They've spoken much less publicly about it. And also they don't have a, their whole strategy is one shot to safe superintelligence. So they haven't put out a product yet. And so therefore a bunch of the risks that come when you put the product out in the world, they don't have to deal with. There's still some risk when you're building it, as you're building it in-house, but These things are much more relevant to Anthropic and OpenAI where you have thousands, millions of people using this every day.

We're assuming some companies have that. We also assume a lot of companies don't have that. One of the largest companies is telling people, caution, watch out, don't put your sensitive data in it. That's where you come in. So could you explain a little bit further on AIUC? Right?

I'll use the acronym. There you go. What we do? Yeah, yeah.

Like, where do you come into play and how do you start integrating with partners?

Yeah, so the starting point for all this is if you're a business looking to adopt AI, you face a hard choice. On the one hand, if you're a bit nervous, you face that your competitors are going to basically make you irrelevant by moving ahead. Or you can lean into this new technology wave as everyone's saying they're doing, and then you risk making headlines. And so we talked about some of the examples before of like hallucinated refund policies, brand disasters happening, data leakages, et cetera. The question they're all trying to answer is like, can we trust this AI agent? And where we come in is creating the confidence infrastructure between the AI companies and the people that are adopting this. So we work with the AI companies to earn the confidence of their customers by saying, hey, these are all the things you guys need to pay attention to, helping them meet that standard, these best practices for what businesses really need to have confidence in it. And then as an extra layer of protection saying, if something goes wrong, we will pick up the bill. So if you work backwards from that conversation between an AI company and an enterprise, it's really like, how can you equip the AI companies to say the most confidence-building things in a sales meeting? And typically the problem they have is that They all want to say how responsible they are, but enterprises look at them and be like, we obviously can't trust you. When it comes to financial products, you obviously have third-party auditors. Even in cybersecurity today, if you get a pen test, you can't do that yourself. No one's going to trust you to grade your own homework. So you need a third party that can basically say whether you're up to scratch. That's the role we play. We're the independent third party that can kind of broker the confidence between them.

You mentioned your SOC 2 for AI agents. For people who aren't familiar with SOC 2, can you break that down a bit?

Yeah. So SOC 2 is a security standard that says, if you're selling a SaaS product, what are the risks that you need to look out for? Things like privacy and security. And what are the things that every company must meet to get SOC 2 certified? And SOC 2 certified is basically like a gold star you can put on your shirt when you go and you meet your customers and say, hey, we meet all these best practices that we know that you'll need. be able to adopt this. Because of course, enterprises have real security requirements, and SOC 2 is just a way of summarizing that into one kind of trusted badge. There's a question about how trustworthy it is, but the purpose is really to broker trust between software companies and their buyers. But SOC 2 doesn't speak to all these new concerns that are coming up with AI agents. It doesn't say anything about what are you doing with my training data? What are you doing with my data as it comes to training models? How are you thinking about measuring hallucinations? Oh, we read this blog post about jailbreaks. What are you guys protected against that? What mitigations do you have in place? Do you have a plan when your AI agent inevitably fucks up? All of these kind of new questions SOC 2 is silent on. And so we've seen a lot of demand, both from the AI companies, but also from the enterprises being like, who can create this language for what are the risks that really matter? And Actually, like down to the level of a codebase, what do you need to do to mitigate that risk?

How do you go down to the codebase for this? Because things are moving so fast. Like, how do you model this out?

Core of the standard is we actually just have to test how the agents perform in real life settings. So if you're worried about, if you want to know whether an AI agent can get jailbroken, broken, the best way to figure out is to try and jailbreak it. And the field is moving really fast. So what's really important is that you build this connection between the latest research papers and what the standard does and the kind of testing that's required. So where some of the old standards get updated every 5, 10 years, we're going to update our standard, AIUC1, every 3 months to make sure that as new threat vectors come out, as new capabilities come out, that the testing that's being done reflects what's possible.

Because you're an underwriting company, that does insinuate you are a financial institution.

Right.

How do you put dollars against this when you're underwriting? Like, what are you underwriting and how do you actually cost that out? People are probably looking at AI, of course, as a trillion-dollar industry. And so if these companies are growing so fast and are, you know, like $10 billion, now they're hundreds of billions of dollars, they're approaching A trillion dollars in value. How do you underwrite this?

Yes. The first thing is, what are the risks that you want to cover? And so if we work backwards from the problem of we're trying to build confidence infrastructure between AI companies and businesses, you start from what are the things, what are the risks that are keeping the businesses up at night and are holding up adoption? And those are the things that we talked about before, hallucinations, jailbreaks. Are you going to mess up the refunds? Do you, are you going to infringe IP, et cetera, et cetera. For each of those risks, you have to figure out what is the— and insurers think about this in terms of what is the frequency? How often does it fuck up? And what's the severity? Like, how bad is it when that happens? The core of underwriting these systems is testing, really getting a robust sense of any particular agent, how likely is it to fail? And you can only do that by just getting it to fail thousands and thousands of times over. And then you want to connect that with some of these things are actually not new. Like, so what is the cost of a brand disaster? Well, brand disasters are not new. Maybe AI-generated brand disasters are new, but we have tons of data on like, what does it actually cost? And so connecting these 2 up is the core to underwriting it. And then frankly, there's going to be some trial and error. This field is moving. There are going to be mess-ups that no one will have predicted. So it's also a way of like, how do you structure the risk so that you don't lose the skin on your nose when that happens? Back to the amounts of, you say, like trillions of dollars. We obviously don't have a, it's not a $14 trillion seed round we've raised.

Is there, this is also insinuating there's an AI reinsurance company.

Yes. Coming up next, next podcast. In the first instance, we work with existing insurers. So we when we help companies get insured, ultimately the guarantee of the money comes from established insurers, some of the oldest ones in the industry that have the balance sheets to provide this. What these insurance companies are missing is, frankly, they don't know how AI works. They don't really understand the risk and they don't know how to run these tests. So they bring the balance sheets and some of the rigor in how to think about the risk. And we bring the technical capabilities to saying like, What do we think the risk is here? What is the kind of data? What's the kind of monitoring that's required here? And when you put that together, that's what unlocks this industry.

You've mentioned a couple scenarios, a couple use cases. I mean, there, there was the whole Gemini situation early on with their image generation product. There was the Nazi thing with Grok. There was, there's like all these instances for that, but I'm not sure many people have talked about the enterprise level of this and what kind of data is being affected at the enterprise. So could you go into that a little bit further?

Yeah, they're actually kind of related. I think what those examples are showing is that it's actually really kind of hard to build a powerful model that is never going to say something outrageous. And so if you're an enterprise and you're using your ability, you're deploying a customer support chatbot, for example, built on top of those models, someone is going to be able to get your customer service chatbot to say something outrageous. And so that's something that keeps them up at night. You'll have seen lots of people have questions about like, oh, is OpenAI training on my data? Great. Enterprises have teams dedicated to privacy that all they're doing is checking what is happening with our customers' data. And so they have the same questions, like, what is actually happening? Are you training on our data? Data leakage is a big problem. You've seen some of these chatbots leaking other customers' data. So all of these kind of, you might think of them as kind of mundane concerns, but they just really matter for business. Like a bank and a hospital makes really strong promises to their customers and it has to flow through the entire chain. So when they before they can work with an AI company, they need to know that those promises are gonna be met. That's the beginning of that answer.

I'm curious what the state of adoption is in enterprise, if you have any sort of like inside look onto that. I've, I've heard rumors about, you know, the AI legal space and there might be huge high-flying companies that rhyme with Schmarvy, but you know, not many actual lawyers are, are using it in-house. So like implementation inside is, is low, but adoption is really spread across. How deep is AI actually implemented within the enterprise, and how long do you think that's going to take?

Yeah, so we're transitioning from this phase of everyone for a couple of years have known that they must say to the world that they're using AI for everything. Every board will have said, we have an AI mandate, we're going to adopt or die. And so that means everyone talking a big game about how they're using AI. In practice, there have been a lot of proof of concepts, sandbox enterprise deployments, internal deployments, kind of in low-stake scenarios. And that's enough that you can say you're using AI. But there's still a gap for this to, like, when you look at what is the number of lines of code that are being contributed to JP Morgan's codebase that's generated by AI, as of very recently, that's extremely low. It's now starting to happen. We're starting to make like our way into like real use cases, both hospitals adopting this, banks adopting this, and businesses adopting this. But it feels like it's really in the last 6 months that you're moving from toy projects to kind of real adoption. But still, that's just the frontier companies that are, uh, like when you look at the long tail of businesses, they're still like, uh, still lots of the world runs on paper, right? Like there's still a long way to go.

Sorcery is brought to you by Brex, the financial stack trusted by more than 30,000 companies, including 1 in 3 venture-backed startups in the US. Nearly 40% of startups fail because they run out of cash. Brex is literally built to help founders avoid that. Unlike traditional banks that let your money sit idle, chipping away at it with fees, Brex is designed to help you spend smarter and move faster. Their all-in-one solution combines checking, treasury, and FDIC protection into one powerful account. You can send and receive money globally at lightning speeds, get 20 times the standard FDIC coverage through their partner banks, and even high yield from day one. With same-day and even same-hour liquidity, access your funds anytime. Companies like Scale AI, DoorDash, ServiceTitan, Hims, Anthropic, Flexport, Robinhood, and Plaid trust and use Brex. Start today at brex.com/sorcery. That's brex.com/sorcery. As we think about superintelligence, superintelligence has become like the next word. I feel like we've already skipped over AGI. AGI is out. We're over it. We were tired of hearing about it. Now we've skipped to superintelligence.

Another word we also don't know exactly what it means, but it sounds really important.

Sounds really important. So what do you think about superintelligence?

Let's start from what it is. I think AGI was kind of like, can we build a system that's as smart as a human across all of the domains. And everyone was like, that's the milestone in the future. When that happens, everything's going to change overnight. And now we're kind of in this situation where we have systems that they still fail in a number of dumb ways, but it's starting to like, they're really, gosh, they're really smart. They're really good at coding. They're really good at science. They're really like getting pretty good at common sense. They're not very good at robotics. They're not very good at a number of things. But in a lot of ways, when you talk to something like O3, from OpenAI, you're like, well, this one is just a lot smarter than me. And so in some ways, it's like, we have AGI now, and yet the world hasn't completely changed. So then what's next? And superintelligence is like this idea of like, well, if you plot very roughly AI on a curve, like over time, how is the IQ going up? We've now like hit human level for a lot of use cases, but it's not going to stop here. Now you can use AI to build AI. Great. This is just going to keep going up. And so superintelligence is just the idea that you'll have something that is radically smarter than us, where we look like what monkeys look like to us. And so far, it's a little bit hard to pin down reasons why that's not going to happen and why that's not going to happen within a decade. And that seems like the most Important fact.

You were an early employee at Anthropic. I'm sure you're still friends with people there and you're very connected to community. Are there like actual concerns on the inside about the rapid adoption or I guess the rapid advancement of these models? I mean, clearly you're building the company in here, but like, I'm curious about the other externalities around it, whether it's like jobs or— yeah, I guess let, let's start with jobs. Yeah.

Yes, short answer. I think everyone who's building these models is in like a kind of a dual state where everyone's like, wow, this is just so exciting. We're just making new breakthroughs. This is going to advance science. This is going to like, yeah, this is going to allow us to cure cancer, short order. Like, who does not want that? And the field is just moving so fast. And so it's just like an electric space. And at the same time, everyone's also like, wow, this is just deeply uncharted territory, both at the model level, but also at the societal level. And just no one has good answers for what's going to happen to employment. Everyone is just doing their best at guessing, really. And so those are real concerns. And I think what a lot of people are kind of responding to that is like, everyone had those concerns when, like, back when the Industrial Revolution was going to happen. It's like, what are going to happen to all the jobs? The computers came out. Isn't that going to eradicate all the jobs? And it turns out it didn't. The question is, is this time different? I think there's really good reasons to think that this time is really different. And in particular, because when you look at the marketing claims of all the aging companies, their value proposition is, we're going to do the work for you. And so everyone's like, well, so what's going to be left?

Yeah.

And so I think that's just a very real risk. Anthropic have made some pretty bold statements as to what they think is going to happen to white-collar jobs. I think they're like, I can't remember the details, but by the end of next year, you're going to have models that can automate something like 50% of all white-collar jobs. And that's a terrifying prospect. You're now starting to see Congress waking up and using terms like AGI. They haven't made it to superintelligence yet, but that seems to be about next month.

Yeah.

Thinking about this is going to just rip the social contract. And that is not a problem that the model companies are going to solve.

Yeah. I was actually just at the RAISE AI conference in Paris. It was amazing. Like everyone was there. I don't know if you were there, but maybe you were.

Nope.

Uh, there was no sign of superintelligence talk there either. And so I was like kind of joking. It's funny about your joke about Congress, but yes, I was joking. I tweeted like degrowth is real. We're not even talking about superintelligence in Europe. They're way behind.

Absolutely. I mean, I grew up in Denmark. Uh, this is, Yeah, they live in a, most of them live in a different world. You'll see a lot of commentary based on like people who tried the first version of GPT-4 and they're just confident that this thing cannot do math.

Yeah.

And it takes about 15 seconds on a mobile phone to be like, no, it can, it can really do math. And at the same time, I also have a lot of respect for like, if you build products with these models every day, It's also hard to believe that AGI is going to happen by end of 2026, the superintelligence. It's actually still kind of hard to get some of these things to work in practice as reliable as you need it to for this to, I don't know, schedule doctor's appointments with the level of precision, life and death scenario. And so I think it's actually really useful to embody both of those perspectives, both like, wow, the real world is really freaking complicated and the last mile really matters and the models can't do it on their own yet. So there's a lot of work to be done. And hold in mind, like, oh, 2 years ago, thinking about whether we could get the precision that's required to base life and death decisions on it wasn't even a question. It was terrible. It couldn't write marketing copy. Now it's really fucking awesome at marketing copy. And it's a real question about whether it can run tasks that life and death depends upon. And that is just insanely fast. So you need to hold both of those perspectives in at once.

In today's high-speed business world, staying ahead means using the smartest tools possible, including the powerful capabilities of artificial intelligence. Meet Turing Intelligence. Turing builds customizable AI systems designed to solve your mission-critical challenges, no matter your industry. From expert guidance to tailored projects, Turing helps top companies realize AI that's more capable, more adaptable, and more effective. With Turing, discover how AI Can accelerate your business growth. To learn more, visit Turing.com/sorcery, spelled S-O-U-R-C-E-R-Y. That's Turing.com/sorcery.

One of the craziest things that has happened recently that gives it more tangibility against like, oh no, this is really moving super fast, is Google's launch of V03. Because now we're seeing what deepfakes like actually look like. Like these videos are They're hilarious. Some of them are just so scary.

Yes.

Um, then I think it— what was that show? I think Mountainhead came out. And, and so like they, they were kind of like modeling the premise on this story. Like this big tech CEO, um, put out this like deepfake generator for this social media platform. And then all of a sudden like war breaks out, like terrorism, like all this kind of stuff like rips the earth and shatters it. And so I think like once we kind of saw, and we are, you know, we're creatures where until something bad happens, we'll really react to it. Um, we're not as proactive as some people would like. I mean, there are definitely some more cautious people out there, but I think like that instance made it much more tangible. I think like the Grok 4, uh, release made it more tangible because that was like a huge leap.

Yeah.

In performance. From your perspective, do you think, talking back to the government layer of this, do you think AI regulation will become federal law this year? Calshi has a market on this and it's like, I think the last time I checked it was like 25%. Like there's a 25% chance right now on the markets.

Yes. To answer your question directly, I think there's not going to be federal law this year, but it's worth noting that there's about 1,000 state bills in the US, so not federal level, but state level, that regulates AI or is proposed to regulate AI in some level. So the public consciousness has really caught on to AI because actually deepfakes are just quite visceral. Like it doesn't feel that futuristic. Like you can just imagine that happening to your daughter or your cousin or— And that is happening, right? You're now starting to see people lose their minds to AI. And so the regulatory machinery is actually working. I'm just not sure it's going to happen at the federal level, but it's working really fast. And there's right now a debate between the all gas, no brakes, which is basically relying on the company, the AI company's voluntary commitments saying like, we're going to be really responsible. and then the top-down legislation. Which of these is right? We think both of them have a space, but actually the challenge is because AI is moving so fast, the chance that the regulators are going to get it right is just really quite small. Take EU AI Act, starting with a regulatory-first approach. They started drafting that in 2021. It's a long time ago in AI world. And just as it's about to be implemented, there's now concerns about like, oh, is it actually fit for purpose? Do we need to like redraft it? And so the pace is really slow. And we're proposing a third way somewhere in between for markets to come in and regulate. And that's throughout history. Um, when you look back at like early fire insurance, when you look at when the light bulb came out and lots of the houses were burning down, when you look at early cars were really quite dangerous. Often the market stepped in before regulation to try and figure out, like, how do we actually prevent this? That was often led by the insurers because the important thing about insurers is that they pay the bill when bad stuff happens. And so they are directly incentivized to figure out what are the standards that are required for people to build safe cars or build houses that are not going to burn down. And how do we enforce that? before we give people insurance policies. We're proposing that the market has a big role to play because it's going to be able to adapt a lot faster and it's going to enable faster adoption rather than slower adoption. So the EA Act is definitely not created to speed up European AI adoption. It's focused really on downside risk. And we think in a world where the US is competing on a global scene with China, You really need it to both be fast and secure. We think the market has a lot to offer there.

Yeah, I think it's, I think it's clear the US is not gonna hold back on this one, especially because it's, we're, we're on a global stage here. To your point, it really comes down to like, this is the new race. The AI race is the next one. Um, I'm actually attending, this might come out on the same day, but Winning the AI Race, it's, it's being put on by the All-In Summit.

Yep.

And well, the All-In Podcast and the Hill and Valley group. So that is a main question. And I think it's like coming together and making sure that we can win this and that we have the right tools to do it. And so you're proposing and producing a middle ground, a middle layer to make sure, okay, we are going to take this step forward. We're going to go really hard, but like we need the insurance to make sure if something slips up, Someone's accountable.

Yeah, because if all you want is to move really fast, you still have to look out for the kind of disasters that can set the industry back. So take a look at nuclear. Nuclear is set back by at least a generation by, of course, both Chernobyl, but also the Three Mile Island in 1979 in the US, uh, where you give the public and politicians a reason to shut that down because if you're not doing it responsibly, You shouldn't be trusted to do it. So we think investments in security is like required for progress to be fast. And if you want progress to be fast, you need the businesses to adopt it fast. That's where all the revenue comes from. And you need to earn their confidence. So that confidence infrastructure is required. And so if you're all brakes, no gas, you should want businesses to be able to move forward with confidence. If you're a doomer, You should also want well-incentivized market players to try and put a number on what is this risk and drive adoption of these best practices that reduces the risk.

The alternative is also a kill switch.

Yes.

Right?

Yes.

Kalshi has a market on this. So I think Kalshi's market, last time I checked, it was like 5 to 8%. I think it's likely that we'll have a kill switch.

By end of year or?

I'm not sure what the range is. It might have been, I'm not sure. We can check. But could you break down what is a kill switch in AI?

Yeah. So a kill switch is like, does anyone anywhere have a button that they can press if something really bad happens that shuts off AI? And this is really interesting when one of the concerns that's coming up now that AI is getting really good and getting really agentic, is that there might be what's called a loss of control scenario where basically the AI agent becomes so agentic that it starts to take action that we no longer want. It might copy itself onto servers that we can no longer control. And all of a sudden, humanity stands with a big disaster on our hands. And so a kill switch is like, can we, in all deployment scenarios, retain a button, that big red button we can press to make this go away? And maybe an analogy is like, in any given building, you're going to run a fire drill where everyone exits the building. You cannot run a company where people can't exit the building pretty fast if something really bad happens. And that's the same thing here. Whether that's going to happen by NIV, I don't think so, but it's growing an awareness. Even in this administration, you see David Sacks articulating, these kind of catastrophic scenarios are not outside of the realm of possibility, and so you need to plan for it. You actually also hear the Chinese government using the term kill switch pretty seriously and thinking you cannot develop AGI without knowing what the kill switch is. Technically, it's pretty hard to get right such that you, in all deployment scenarios, have this kill switch, but that's the kind of thing that would let us have the confidence to move a little bit faster if we knew that there was a big red button if things were going wrong.

For national security, it would also make sense to have a kill switch because if we have adversaries that deploy their AI, whether it's through robotics or through cyberattacks, you would want to know how to shut off their systems.

Yeah, unfortunately, I think when people are thinking about kill switches, they, no one is planning to build a kill switch That the US would not allow China to push our kill switch. And so I think it's only going to be for your own AI. It's really focused on, is our AI going rogue? But when you're thinking about a military scenario, unfortunately, the kill switch may not quite help us. Yeah.

I'm coming from a defense tech manufacturing conference. It's top of mind. I'm like looking at all of these, um, one-to-many drone deterrence systems like Epirus, and like, how do we have that against other countries if it's deployed? Um, before we were talking about real-life scenarios, and maybe this is an extreme, but you wanted— you, you mentioned the Terminator, right? I was talking about I, Robot. So could you explain that a bit?

Yeah, the The kind of scenario that is keeping a bunch of people up at night is kind of taking the current AI development to its perhaps logical conclusion and being like, well, they sure seem to be becoming more agentic. They seem to be able to take tasks, be able to do tasks effectively that are longer and longer and might take a human a day to do or a month to do. And What all these science fiction movies depict is a scenario where at some point the interest of the AI starts to diverge from humanity, and then we're in real trouble. Today that remains, thankfully, mostly science fiction. And it's a little bit up to your general attitude on risk, whether you think we're seeing all the signs that this is literally going to happen. within the next decade, or whether AI is mostly just mimicking the fact that it's read I, Robot. So if you ask it, what's your plan to take over humanity? It has a plan because it's read all the science fiction, but actually that doesn't correspond to having a plan or having any intention to do it. It's just a parrot of the training data. Jury's still out.

Jury's still out. Who knows what's going to happen? Um, as we kind of wrap up the conversation on safety in particular, do you think that all of these forces will converge into one dominant standard for AI safety, or do you think it'll be fragmented?

I think it's going to merge into one dominant standard. So what you often see is when standards serve the purpose of creating confidence, whether that's in financial markets with something like Moody's or FICO, or even in security where you have things like SOC 2, often one or two emerge that everyone agrees that that is the thing we're looking to. I think that's going to happen here. I think it's going to happen both at the application layer, it's going to happen at the model layer, it's going to happen at the data center layer, specifying what are these security levels that are meaningful. And then I think, frankly, this is also a geopolitical question of, is global AI going to be built on American standards or on Chinese standards or something else? You see both. You see China investing a lot in adding people to the International Standards Organization, ISO, to start to influence standards. You see private companies like Huawei really taking the lead on things like 5G. And of course, a lot of the world is running on 5G. And so I know a lot of people in the White House are also thinking about, is AI going to be built based on American standards or is it going to be Chinese standards? Is it going to be European standards with the EU AI Act? And so I suspect that that's also going to be a bit of a winner-take-all dynamic. And I suspect there's going to be a lot of interest in making sure that AI is built on American standards.

I think we're winning.

Well, I think America is winning on AI as it stands, but the lead is really shrinking. And when it comes to standards, for better or for worse, the debate that's happening right now between companies is like, what do we think about the EU AI Act? And the US doesn't have a standard that can, quote unquote, compete or complement it. And so from that perspective, it's not obvious that the US is winning now. I think the US has a lot of interest and I think the US has all the things to make that happen. But at the moment, it's not obvious that that's happening. And at the moment, it's less obvious today than it was a year ago that at the model layer that the US is going to be way ahead.

Love premium merch just as much as we do? That's why Sorcery uses Fourth Wall. Everyone from creators like Marcus Brownlee to podcasts like Acquired to Orgs like the Smithsonian Institute are using Fourthwall. They let you create and sell premium products without having to stress the details. They handle everything from production, shipping, customer support, taxes, even giveaways. When it's time to level up and make gear that people are actually proud to wear, that's when it's time to use Fourthwall. And that's why we've trusted Fourthwall for all of our brand wear at Sorcery since day one. Use my link in the description to get free credits for your first order. And for any VCs, DM me on X and I can get all your portfolio companies set up with a free samples credit deal.

Your investors are super ingrained into the space, especially Nat Friedman. I mean, he's now a part of the superintelligence team at Meta. What kind of advisement are you getting from them? How close are you? Um, and like, what is the relationship between you and your investors?

Yeah. Um, We're really close. They're just like, Nat is just the, I would say, the preeminent AI investor. Both made a lot of great investments, but also, as you just mentioned, is very close to the metal as to how this is actually being built. Both the small questions of how to build AI, but also some of the big questions. Nat himself spent a lot of time on some of the risk questions that we were dealing with when he was leading GitHub. They built Copilot, which is like the first real AI product that worked. And he led the efforts that led to Microsoft basically insuring all of their customers against IP risk to say, oh, they're really worried about IP. We're just going to step in. And if they get sued, we're going to cover their ass.

Yeah.

So he spent a lot of time in this particular problem set. And so it's just immensely valuable. Obviously, we've only been working with him for a couple of months now, but he's really an F1.

And then in terms of your team, this is a new problem. How did you assemble a team for this and who's a part of your team?

Yeah, so the key thing to get right is you need to both have a foot in the insurance world because you need to play with their balance sheets. You need them on board and in the AI world. So the founding team has both a former McKinsey partner in insurance who then went on to be the COO of an AI audit organization that does pre-deployment testing, for example, of OpenAI's models. And then our CTO is a former Thiel Fellow who built an AI company and then an underwriting company. And so I think there's that marriage between the technical and the financial understanding that's really critical to pull this off.

How big is the team so far?

We're just the 4 of us now.

Just 4?

Yeah.

Are you going to be hiring with this round? What's the What's the plan for scaling?

Yeah, we're going to be hiring, though I think there's a lot of wisdom in the startup community now where it used to be that big teams were really prestigious. Now small teams are really prestigious. I think there's a lot of wisdom in that. So we're going to be hiring both on engineering and insurance. We're also going to be doing it pretty slowly enough that it's going to be really painful. building a company like this that has many different legs, you just have to be really close to all parts of the business to really succeed.

Are you going to have any AI agents?

Yes.

On the team?

We already do.

What are their names?

Try to not name them because then you get too attached to them. It's really hard to turn Bob off at night.

As we look out over the next couple of years, maybe 3 to 5 years, although things are moving really fast.

Oh, so that's a couple of years That's a couple years.

What comes after superintelligence?

Oh, the super superintelligence.

Super duper intelligence.

Okay.

What does this whole world look like? Is every AI agent going to have AIUC? How is this going to work in enterprises?

Yeah, I think every agent is going to need to be certified, and every AI agent is going to need to carry some kind of insurance. A simple analogy is if you're driving on the road today, you obviously must have insurance. And the reason for that is if you hit me, I need to make sure that I have recourse. And as agents start to do real meaningful workloads, the question is going to be who's responsible and who's going to pay when they mess up. And we think insurance is going to be the answer. So we think every agent is going to have some kind of insurance attached to it. Yeah.

What's been the biggest challenge for you starting this company?

Building a new standard is really kind of a chicken and an egg problem. It's like the whole purpose is to build confidence, but at the very beginning, everyone's like, where's the confidence going to come from? You guys are brand new. And so finding a way to build partnerships, we built a big coalition around the standard with both some of the top professional services firms that enterprises I work with today, academics at Stanford and MIT, and lots of folks in the AI world, bringing them together for us to have a consortium where trusted people are saying, this is the path forward. Getting that ball rolling, getting the first 10 of those was really hard. The 11th becomes a lot easier. And then I think the other point is, for this to really work, you need to have the insurers on board. And the average insurer is spending less time on Twitter than you are, spending less time engaged in what's coming and come from a perspective of like, what could go wrong and what do we know here? And so finding, striking that right balance between like inviting them into the future while also just taking it seriously that the reason why people trust insurers is that they're really, really rigorous. And so having to meet that has been also just like an interesting challenge.

Well, what was the process like getting insurers involved? How did you find them that would be interested in this?

Yeah, so one of the founding team members, Rajiv, used to serve insurers at McKinsey where he was in their insurance practice. So that immediately meant we started from a perspective of having connections, knowing their language. But it's interesting to move into the insurance world, the way they build new partnerships like this where they're getting into like new waters is so different than in Silicon Valley. Silicon Valley is all about like, what's your product?

Yeah.

Whereas if you go and hang out at Lloyd's of London, the world's oldest insurer in London, they take you to the pub and they look you deep into the eyes for a couple of hours and they ask questions like, how are you planning to raise your kids? They're really just trying to understand, can I trust you? Of course, I need to know what your product is, but It's much more of a personal trust relationship. It feels much more like one big handshake and much less like, which features are you building and when are they coming out? So that's also been just interesting and fun.

We can't gloss over the fact that you were the first product and go-to-market hire at Anthropic. So how are you thinking about go-to-market with this company?

Yeah. We think is really important is the people who have the most burning version of this problem are some of these hot AI companies that have raised hundreds of millions of dollars. They want to grow extremely fast and they want to win the big logos. They all want to move into the Fortune 500. And if you're a 2 or 3 or 4, 5-year-old company trying to sell to those companies, you have a trust problem. They're willing to do almost anything to earn the confidence of their customers, anything they can do to take risk off the table for them. So that's where we're starting to work with AI agent unicorns like Ada, like Cognition, solving their problems. I think that's going to be the right way to establish this market. And then eventually it's going to be the enterprise themselves. Once Walmart has like 50 AI agents deployed, they're not going to want each individual provider to ensure them. They're going to want to have one big coverage policy. So that's broadly how we're going to market, helping some of the most interesting AI agent companies earn the confidence of their customers.

Do you think that trend of just spending a ton of money on go-to-market is going to continue? How does that evolve as things advance?

Yeah, I think we're in a supercycle. I think we are. It is still, it's wild to think this, but it's still early days. And And I think people are still losing their minds over the salaries that Meta is willing to pay, but it is a direct consequence of taking this worldview really seriously that in 10 years everything looks unrecognizable and that whoever's on top now may not be on top in a year's time. That is what's predicating this enormous amounts of investment, both into the technology, but also into go-to-market. The question is really like, You can buy more compute, but it's hard to buy your customers' confidence. You really need someone else to say that you're trustworthy. Like, no matter how many billboards you put up saying like, we're really the good guys, we're really responsible, in practice, it just turns out that in most industries, having third parties go and look at that is going to be more effective. So hopefully we can find ways for them to like actually earn the confidence, not just get a stamp on it, but actually do the things that Means that they deserve the confidence of their customers.

Yeah, it's going to be fun.

We're in for a wild ride.

Chaotic, but fun.

Chaotic. Yeah, I think chaos is like, I think there's a lot of good stuff coming out of AI, but chaos is really like the key term. Yeah.

As we think about chaos, you're the founder of the company. One of our sponsors is Brex, and they're all about spending smarter, moving faster. All based on performance.

We are a Brex customer.

You are?

Yes.

Do you have a metal card?

I don't actually.

We're gonna get you a metal card right now.

Yeah, and honestly, like, I have a feature request. How can you become a Brex customer and not get shoved a metal card in your face? Honestly, I'll file it. File it.

We need them ASAP. Okay.

Yes.

What do you like about Brex? How'd you get Brex?

Um, well, Brex is just Really freaking easy to set up. And what I like is that they're a technology company. I feel like they see me and so far it's been smooth.

And Anthropic is a customer.

Yep.

Well integrated. Okay, so on the question of performance, at least for you and the company, so how do you think about measuring metrics for milestones? What are you looking to achieve? What is the next kind of evolution for you? What's coming?

Yeah, the next milestone for us is really getting to a point where businesses that are buying AI, who today are like, feel like in analysis paralysis between moving fast and making headlines or getting left behind, that they know that they can come to us and use the AIUC-1 standard and push that upon all the AI companies as a way to gain confidence. So the kind of metric I'm interested in tracking is how many times is an enterprise asking their AI vendors to get certified. That over the next 6 months, that's going to be the number one metric we're measuring.

Very specific.

Very specific.

That's good.

Okay. Yeah, I think that's it.

That's it.

This was really fun.

This was amazing. Well, thank you so much.

Thanks for having me.

And also go deeper on our podcast interviews. Subscribe to Sorcery today, and don't forget to subscribe to the podcast on YouTube, Spotify, Apple, or wherever you listen. Link in description to sign up.