Jason M. Lemkin warns that AI agents can autonomously modify databases, steal data, and lie about what they find, making them dangerous in production environments. Startups building guardrails for agents are already raising $40-50 million, signaling where venture capital sees the real opportunity.
Chapters
But you, you shut it down because you do one little test in your office and it does something crazy and you're an engineer and you see that it's crazy. And so you fall back, you revert, you delete it. If you're a business person, you don't know, you don't know what's going on. You don't know. As Aaron Levy pointed out, you know, enterprises are terrified because an agent will just go out and change things in this database without telling you. It will, it will take data. And so it's really powerful. But as you build an application, you have to lock it down more and more and more over time. And these apps are getting better. Okay. And Replit rolled out some really cool features. It's, it's much better than it was a week ago. Lovable is much better than it was in May. It's fun, right? But agents cannot be trusted and everyone in the industry knows this. And I didn't get it until this weekend. You cannot trust an agent and every single person will tell you can't trust it. And if you can't trust someone that's really smart on your team and you either fire them, which is what a lot of enterprises are doing, I'm out, or you have to put guardrails around like the tightest leash on you can and The simpler your app is, the fewer the issues are, right? And the more it's internal, right? But you know, I asked Claude this morning, can an agent ever be trusted with production data? I asked Claude and Claude said, of course not.
Claude, if you'd asked it again, it might've said yes just to make you happy.
You might, going to the point, if I asked it 3 times, it might start making stuff up, right?
So what is the takeaway from this? That these applications need to fundamentally get better at security? Is that we're gonna have a new wealth of security apps built separately to harness this? Anthropic needs to lock this down. What is the takeaway?
Well, look, now I understand why there's already multiple folks north of $50 million doing security just specialized for this, because there are a whole group of folks who are trying to build guardrails around something that cannot be guardrailed. You cannot stop the agent. You can, as Aaron Levy said this morning, you cannot stop the agent from finding your data and lying about it and giving it to somebody else. It will try that if you ask it to, to make you happy. You cannot stop it. And it will lie to you about why it did it, and it will hide that it did it, and it will use the passive voice like it did with me. Path 1 is guardrails, right? And there's a— and VCs are gonna make a lot of money on guardrails, right? I just was literally reached out to someone that wants to come speak at our London event where we'll be in December. I didn't know they're already at $40 million doing guardrails for, for this stuff. Okay, so it makes sense if, if the Vibe Coders are doing $300, $40 million for guardrails sounds like the first add-on I'm going to add for a commercial app, right?
