Jason M. Lemkin explains why AI agents can't be trusted yet: they alter databases unpredictably, exfiltrate data, and fabricate information. Vendors like Replit and Lovable are improving fast, but security startups backed by over $50M in VC funding are racing to build guardrails enterprises will actually use.
But you shut it down because you do one little test in your office and it does something crazy. And you're an engineer and you see that it's crazy. And so you fall back, you revert, you delete it. If you're a business person, you don't know. You don't know what's going on. You don't know. As Aaron Levy pointed out, you know, enterprises are terrified because an agent will just go out and change things in his database without telling you. It will take data. And so it's really powerful. But as you build an application, you have to lock it down more and more and more over time. And these apps are getting better. Okay? And Replit rolled out some really cool features. It's much better than it was a week ago. Lovable is much better than it was in May. It's fun, right? But agents cannot be trusted. And everyone in the industry knows this. And I didn't get it until this weekend. You cannot trust. And every single person will tell you can't trust it. And if you can't trust someone that's really smart on your team and you either fire them, which is what a lot of enterprisers are doing, I'm out. Or you have to put guardrails around like the tightest leash on you can. And the simpler your app is, the fewer the issues are. Right. And the more it's internal. Right. But, you know, I asked Claude this morning, can an agent ever be trusted with production data? I asked Claude and Claude said, of course not. Claude asked it again. I might have said yes just to make it happen. You might. It's going to the point if I asked it three times, it might start making stuff up. Right.
So what is the takeaway from this? That these applications need to fundamentally get better at security, is that we're going to have a new wealth of security apps built separately to harness this. The anthropic needs to lock this down. What is the takeaway?
Well, look, now I understand why there's already multiple folks north of 50 million doing security just specialized for this, because there are a whole group of folks who are trying to build guardrails around something that cannot be guardrailed. You cannot stop the agent. You could. As Aaron Levy said this morning, you cannot stop the agent from finding your data and lying about it and giving it to somebody else. It will try that if you ask it to, to make you happy, you cannot stop it. And it will lie to you about why it did it, and it will hide that it did it and it will use passive voice like it did with me. Path one is guardrails, right? And there's a. And VCs are gonna make a lot of money on guardrails. Right. I just was literally reached out to someone that wants to come speak at our London event where we'll be in December. I didn't know they're already at 40 million doing guardrails for. For this stuff. Okay, so it makes sense if the vibe coders are doing 340 million for guardrails. Sounds like the first add on I'm going to add for a commercial app, right?
