#348 AI Agents in Your Systems: Speed, Security & New Access Risks with Jeremy Epling, CPO at Vanta

While like 79% of them are using agentic AI, I think only 45% of them said they have the compliance tools to actually go support it. And over half felt like the AI innovation is completely outpacing their ability to keep up with all the new security threats that are coming.

Whenever I run training on AI agents, the first question the audience asks is, is this going to cause me security problems? The answer is yes. AI agents are going to cause a lot of security problems in the near future, but they're also great tools for helping with cybersecurity.

It definitely feels like there's a brand new wave of attacks that are coming and like they're also helping attackers write code that they never could before in more advanced ways. So there's a lot of old kind of smelly data sitting around everywhere that's very easy to kind of like pull up if you don't know what's good and what's not.

I want to explore this double-edged sword. So today we're chatting to Jeremy Effling, the Chief Product Officer at AI-powered compliance platform Vanta. Jeremy has over 2 decades of experience as a tech executive and previously was VP of Product at GitHub.

Hey, my team contributed to like, you know, whatever, 90% of the revenue we got last last year by answering these questionnaires. If we don't feel like we have a high-quality answer, we'd rather just say like, we don't have an answer to this. Time, like, if a human can't understand it or answer it easily, AI is definitely not going to be able to.

So let's find out if agents are going to help or harm security. Hi, Jeremy. Welcome to the show.

Hey, thanks for having me. Excited to be here.

Yeah, great to have you here. So to be honest, I'd just like to know, what are the biggest security issues that are facing businesses at the moment?

Yeah, I mean, I think probably no one will be surprised to hear AI is at the top of the list. You know, a lot of companies are adopting AI. They're seeing a ton of benefit. They're doing it really, really quickly. And there are still a lot of questions that kind of come up, like every product is adding AI capabilities. And then security teams start to wonder, well, what are they doing with all this data that we're giving them and our customer data? Do they have access to our customer data? Are they training models on it? What are they doing with it? There's all these exciting new technologies like MCP servers that allow agents to talk to other agents on behalf of users., and we're still in the early stages of understanding how to secure those. I think there's also a lot of AI security threats, like Anthropic mentioned a recent nation-state attack where cloud code was used a couple of months ago. And so I think there is a lot that's surrounding security teams. And we actually recently did a survey, the Vanta State of Trust Report, and asked a bunch of CISOs and security professionals how they were thinking about AI. And while like 79% of them are using agentic AI, I think only 45% of them said they have the compliance tools to actually go support it. And over half felt like the AI innovation is completely outpacing their ability to keep up with all the new security threats that are coming.

It's one of those things where it's brilliant that there's so much progress going, like, with AI, like, all the time there's something new coming out, but then trying to keep up with it on the security side, this kind of the gap between compliance and security and the actual, like, adoption, that's a little bit terrifying, right?

Oh yeah, definitely. I mean, I think that, uh, I mean, the good news is I think with any of these security things, there's always this cat and mouse game, you know what I mean? Between like the attackers have the same technology that the kind of defenders and good guys do. And it's a matter of like who's ahead there. Um, I think that the good news is with like, you know, products like Vanta and others, like you can then give even more security information to your security tools and to those agents to help defend you faster and better because they should definitely know more than what the attackers can see from the outside. So I think there's a ton of benefit. I think the defensive side has a bunch of tools for them and like a lot of exciting new ways to go defend against it. But it definitely feels like there's a brand new wave of attacks that are coming and like the internet will become like a more dangerous place before it gets back to like a more safer place again soon.

So I guess there's so many like, uh, new kind of threats and problems going, like, where do you begin? Like, where do you start to shore up your defenses?

Yeah, I mean, I think that a lot of it is a lot of the similar security practices that security teams have been doing for a long time. I think they've just been super charged. Like, they're always dealing with threats and attackers coming in. It's just now those attackers can produce more sophisticated threats and do it in a more distributed way, even faster than they have before. And you have like kind of novice wannabe hackers, if you will, who now have pretty advanced coding skills in comparison to what they had before. They didn't have to go build these up over time. Like, Cloud Code and Codex, as much as they're amazing for us improving the quality of our products and shipping products faster, They're also helping attackers write code that they never could before in more advanced ways. So I think security teams are kind of dealing with like all those threats and prioritizing what those are, you know, leveraging endpoint defense, EDR solutions, things like that are still incredibly important. I think data security is more important than it's ever been before as well. Like the thing people are trying to go after, the crown jewels are usually always some data in your company. It could be your trade secrets, it could be secret recipe, your formula for a new like pharmaceutical medicine or something like that. And so being able to protect that, I think, has become a lot harder because not only do you have these attackers coming in from the outside attacking in new novel ways, but you also now have all the internal tools that your employees are using at your company are all pulling on all of this data and sending it to models to ask questions around how do they develop that next breakthrough drug? How do they go through and, you know, go ahead and get an edge against this competitor or, you know, build this new piece of like novel tech?. And so I think there is this new set of like, man, data leaking kind of out of the company and the company's trade secrets is a lot more top of mind. So they're starting to ask a lot harder questions of their vendors of like, hey, when we're sending this data to you and your AI models, like, what are you actually doing with this? Can this be reproduced somewhere else? Like, what is happening? So I think those are some of the top threats that are coming up. And, and then they want that proof back to be like, okay, great. You said you do this, how can I trust it via an audit report or something else?

That's really fascinating. I mean, I think if someone's attacking you and stealing things, then that's one thing. But if it's kind of an unforced error where you are actively leaking your own trade secrets to the world, that's a real, like, that's a proper own goal. That's a really terrible thing. So in terms of stopping this, does it start with the security teams or do they need to be like a C-suite that do this? Like, who needs to solve this problem?

Yeah, I think it's a company-wide effort given how everyone in the company is using these tools. I think a lot of it starts with data governance. I think a lot of companies are pretty lax on data governance and probably have been for a while because they're like, oh, who's going to have the skills to go, you know, mine in this SQL database or this Mongo database or whatever else to like pull information out. But now all of these vendors and tools just reach in and integrate with everything and they're trying to pull out these insights and Even if you have someone that is not malicious internally, they might be sending data that they shouldn't. Like they pulled a bit of customer data because it's so easy to go pull that and they shared it with another model. And your policies for your company say, hey, we don't share customer data like this. And now it's in someone else's model and maybe they're training on that and it can get reproduced in some other way. So I think this isn't just kind of nefarious people from the outside or nefarious people on the inside. People just don't realize how much data AI is constantly scraping up and how much these integrations are kind of pulling in all these data sources. I think there also is a little bit of making sure people aren't outsourcing just good human judgment. You know, there's a lot of things AI is great at. It also makes mistakes. I think everyone knows about hallucination and other problems we've had. And they get like bigger companies with all this pressure to move faster. I think a lot of employees are just looking for that quick answer and sometimes that answer isn't there. So one of the things we do internally with all of our data systems is our actual like product analytics team sits on a lot of our channels where we do have AI tools. So when people are coming in and asking questions, even if an AI tool gives the default response, we're kind of doing our internal evaluations on those responses and being like, oh, yep, this one's good, this one's good, this one's good. Oh, actually that pulled the wrong data from like an old data source that actually isn't trustworthy or an old table that we forgot to go like drop and delete at some point. And like, you shouldn't trust this. So I do think there's people that are just doing the right thing and they have access to all these things they didn't before and they'll just make mistakes because the AI gets it wrong. And so internally making sure all of your functions are thoughtful of like, especially if a piece of data smells wrong, go talk to an analyst or someone else or like do that digging yourself to be like, okay, is this exactly right or not?

I love the idea of having analysts just check the, like, the quality of the output. Then it needs to be someone who understands what the problem is, just be able to say, is this right? Is this wrong? And that's where you're going to get the feedback loop to fix things.

Yeah, totally. And I think that like keeping those clean data sources is even more important than ever and having like a data engineering and analyst team that says like, these are the authoritative things we trust in them owning that pipeline and hooking it all up. Because I don't know, I've worked at a bunch of different companies and I can say like, there's a lot of old kind of smelly data sitting around everywhere. That's very easy to kind of like pull up if you don't know what's good and what's not. And so regular hygiene, I think, is probably even more important than it was before.

That's an amazing analogy out of the era of like old smelly data. That's funny stuff. So in terms of managing all this stuff, you said like managing integrations is kind of a problem. Like, is there a process to fix this or are there tools to help you deal with it? Like, how do you go about solving, making sure that the right integrations have the right access to the right data?

Yeah, I mean, I think this is a big thing where the security team helps a lot. Like pretty much at every company through as they go through the procurement process, we've got someone who wants to go buy a tool. They're super excited. The security team comes in. Sometimes they're like the no people, but they're doing doing like a really important job for the company that are just like, hey, this tool seems exciting, it probably solves your problem, but like, can we actually trust this company with their data? And so I think that is when the security team really comes in and starts asking those deeper questions. You know, there's different compliance frameworks that are out there. You know, just like if an enterprise is buying a piece of software, they'll absolutely say like, hey, do you have SOC 2? Do you have ISO, you know, 27001? If it's like the federal government, they'll ask about like FedRAMP or CMMC or these different standards. There's the same thing for AI as well. ISO 42001, you know, the NIST AI RMF, the EU AI Act. So there's like different ways to go and say, hey, here's a set of compliance frameworks and controls that we support. And then you can share those, that evidence out. And I think that really speeds that process along so that the security team that wants to go buy a tool and is evaluating it, can get all that information really quickly. They can go through it, they can ask detailed questions and try to drive that procurement process through really quickly. And then I think afterward, then the next hard part comes in. It's like now you bought the tool, you trust the company, you think they're going to handle your data right. That's great. But now internally, does everyone have the right access and how is that going forward and how are they using it? What data are they putting in or out? And I think there's a lot of new tools that are allowing people to kind of monitor those data flows to understand what is getting put in, to kind of set off like alarm bells if things seem off. But I do think in general everyone's seeing such a productivity benefit. It's, you know, it is a little bit more like fast and loose now because they want that benefit of AI. But at the same time, we need to be like thoughtful of like not leaking customer data and breaking promises that companies have made in the past.

Definitely. And it sounds like a lot of this stuff is about managing who has specific access to what things, all these kind of rules, it sounds like this kind of, uh, the tedious, uh, manual task that ought to be automatable with AI. Is that something that AI can help with?

Yeah, definitely. I mean, I think that there's a case where, you know, you can look through different people, you can understand, like internally what we do at Vanta, we use Vanta to go do this. Um, we go through and look at everyone's title. We understand where they are in the organization. We can see what other applications other people in this job function have done in the past., and you could kind of build a rough profile around like, oh, engineers on this team usually need access to these things. And this thing that they're asking feels like an anomaly. And like, maybe that's something we should go look at. So I kind of think of it as like anomaly detection and kind of giving, I still think there's a really important human in the loop. And I think security teams especially will be, I think one of the last ones to kind of give full control over to AI to remediate and make all the decisions. So I always think about designing as like, review and approve flows. Like as someone who sells to security teams where it's like, we give you all the context that you need so you can quickly look through it and put in like your expert judgment and then decide like yes or no. And how do we tee it up for that like super fast, easy decision versus, hey, you've got to go access 15 different tools, read all these docs, talk to all these other people like, and you know, now something that took you 5 or 6 hours to go do, now it's all teed up for you to go like quickly read through, make a decision, and then move forward.

I love the idea of just enabling humans to make good decisions faster. But I suppose you mentioned that security teams, they're probably the last people that want to turn over full control to AI. I suppose there's a natural tendency for security people to be a bit paranoid. That ought to be kind of part of their job function. How do you build paranoia into your processes then?

Yeah, I mean, I think a lot of it comes down to like trust and quality. So, You know, one of the things when I think about designing AI features is how much accuracy in the quality of the answer is important. It's, you know, it's kind of like we've experienced for years, like pre-AI with like Google search results. It's like there are plenty of other search engines. There's plenty of other companies that spent like billions of dollars to go make them great and to do it all. But like the quality of like those top links was the most important thing. And I think the really differentiator for a lot of these AI products, and some might have more interesting UI experiences and things in the future, but I think is the the quality of the answer you get. So when I think about designing for a security team, I think about a hot— like, it is good to refuse to answer questions if you don't have high confidence. And so that is something that we have definitely engineered into the Banta AI agent, where when we go through, we look through and say, hey, if we don't feel like we have a high-quality answer, we'd rather just say like, we don't have an answer to this, here's a couple things you can go look at, or hey, we're gonna go work on this and we hope to have an answer for you in like one of our future updates. And I think that just builds trust with security teams that are already more skeptical. And, you know, some of the other things that I think a lot of AI products do, like showing citations. So like where specifically in references, hey, this came from these documents, it came from like this specific paragraph or sentence here. We pulled it out, let them build confidence in it. We have a lot of flows, like I mentioned before, when you're in these procurement workflows, you know, you have a software buyer and a software seller and they're trying to exchange information. I think one of the things we look at a lot in our AI is what the software, you know, seller say in their, in their answers actually match the documents they provided you as the evidence. And obvious, sometimes it doesn't. And sometimes that's just like a human error. They forgot to update some policy or the person misspoke or didn't know, but flagging those things really quickly, I think helps build confidence. So I think just being really focusing on high quality answers that people can depend on over time. Higher than like being able to answer everything. So I think of like coverage rates are really important, but accuracy is even more important. And I'd rather not drop my accuracy bar to have more coverage. So it's like, what can we go do to keep accuracy high while kind of like working up our coverage?

Absolutely. I mean, there's no point in getting the wrong answer really quickly. So I love the idea of like, yeah, you focus on quality, make sure you get the answer right. And citations is like a surprisingly hard challenge with a lot of the generative AI is like, it ought to be an easy thing for you to think, uh, naively, that if it finds some information, it should give you the right citation. But often, yeah, it's, it's total nonsense. Um, now I, I know you've been— you mentioned, uh, Vanta Agent. So this is like an internal agent that, uh, your team's been working on. Talk me through, like, what, what does Vanta Agent do and why did you build it?

Yeah, I mean, I think for us, we just see, especially in the space of, like, trust and GRC, a lot of different security professionals have to do a lot of manual work, and a lot of it is very document heavy. So Bantam kind of started off with a ton of integrations, and then we tried to automate everything we could. Like, hey, you've got your, you know, source control provider like a GitHub or a GitLab or Bitbucket or someone like that, got AWS or Azure, GCP, Oracle, or someone like that, you can automatically pull that down. But still, a lot of these processes like SOC 2 and ISO still require you to write a lot of policy documents, You're sending a bunch of documents and evidence. You're still scanning screenshots because, you know, you can't get API requests for all these tools. So sometimes you need to have a screenshot and go do that. I think AI really unlocked us being able to solve a bunch of these problems we never could before. So when a buyer and a seller are exchanging this big security questionnaire, like if you're a software company and you sell to like a big bank, you're going to get a questionnaire that could be 400 or 500 security questions long. And, you know, you start selling to a lot of enterprise customers, you're getting these all the time. And it's kind of this mind-numbing task to go through and they're all worded slightly differently, but asking kind of the the same things. And it's not like high-value security work, but it's like, it's very important that you do it and you give it accurate, but like the repetitiveness isn't there. So that was one of the areas we tackled with our agent first, was this buyer-seller relationship of automatically answering security questionnaires. You can build up a knowledge base that's curated. We tell you when that, hey, you're getting new questions in that you don't have coverage for, and then we can suggest things like, hey, do you want a new AI policy around this? Or all of your customers are asking you to redline your SLA to something different, maybe you want to change your SLA and maybe you want to go do these and thinking about all that customer feedback coming in. The same thing for like if you're on the buying side, the vendors, you know, you're receiving all these questionnaires, getting there. And then I think as security teams go through, you know, they often just see drift in their policy stack. So they, you know, maybe they want to go update an SLA, maybe someone leaves the company., and then they want to go back through and connect that through and say like, oh great, our CISO left. We should go update all the documents. Manta can just update, detect, and do that all for you. You want a holistic risk program when you're thinking about things. So like you have these high-level business risks, these all map to things that people aren't as familiar to, like frameworks and controls. And then you have individual tests and different security actions everyone's taking. And you want to be able to see that whole program together and ask, maybe we do want to go for ISO 42001 to prove the quality of our AI experience and security, you can just ask the agent like, what do I need to go do? And then we'll look through everything and be like, hey, you should add these security controls. I would update these policies. You should think about this. And we kind of guide you through that process. So I think of it as, you know, along the lines like we refer to as your 24/7 GRC engineer. It's kind of someone that's sitting alongside you the whole time trying to go do a bunch of the work for you. So you can kind of stay in this like review and approve workflow where like you're bringing your expertise to evaluate key decisions and then kind of moving to the next decision and not having to do as much of that kind of manual repetitive, you know, kind of grunt work, if you will, that was like common with, you know, certain GRC teams for a long time.

Certainly filling out like a 500-page questionnaire about like what do you do on security across your whole business for some enterprise customer, then doing almost the same thing but not quite the same thing for your next customer. That sounds like a really, really tedious job. So I love the idea of using AI just to completely, well, outsource as much of that as possible. So I guess talk me through, like, how did you approach building this then? Like, what do you do to make sure that you do have these high-quality answers? And I guess if you're filling out a 500-page survey, that's gonna be like a big customer, that's gonna be like a big account you're trying to close. So I guess there's a lot of money on the line. How do you make sure you get that high-quality answer? Like, what did you do to build something that gets the right answer?

I think a lot of it was, you know, one, probably things that people have heard about or talked about in the past is like having a really good golden dataset. So for us, it was seeing the questionnaires that we've gotten in the past for customers that wanted to share more of that data with us. And then we have a really big team here of GRC subject matter experts, GRC being governance, risk, and compliance, is that function that deals with a lot of the stuff that Banta does as a company within the security team. And so those SMEs, as we call them, the subject matter experts, are all former GRC professionals at large companies, former auditors, you know, have been auditors at like big firms or focused on authorization packages for the government. And we have them as part of our evaluation or like eval process. So we go through, we build a golden dataset, and then we go through and just continue to kind of like tune the prompts and the models to the point to have high quality answers. And then like we talked about before, also pulling in those like citations and quotes and making sure we're getting the right ones to kind of build that trust. So it's basically an always-on, never-ending function. I think that's one of the big differences when you think about building products now and in the past, like ML products were like pretty similar to this, but I would say now that AI has become so ubiquitous, I think there's kind of standard deterministic SaaS where it's like, hey, I built the thing and I tested it and I said one comes in and two comes out the other side, and it's always going to be that way no matter what. But with AI, the evals are like always changing because it is probabilistic and not deterministic, because the models are always updating, because your customers are always pushing and asking to go do more. So I think about when we start a new AI feature, there's a part of the team that's just dedicated a decent side to quality forever, you know what I mean? Because like quality is the feature and it's not just like the new capabilities we add. When we thought about things like questionnaires too, there's all sorts of different formats. So then you have to think about the parsing side, like, did I get a Word document? Did I get a Google document? Did I get a spreadsheet? Are there conditional questions? Is the way that they did the formatting in this, like, Word document, like, different than the way they've done in the past? Some people use portals, so you go to, like, a portal to go answer the questionnaire. So now we're, like, scraping the DOM in the browser or building a browser extension. So I think there's a lot of interesting AI optimization work, and there's different tools and platforms you can use for parts of this to like just get the questions in a, you know, a consistent way and then going through and answering them with high quality. And then the ones that you don't know, we then leverage of like, oh, who's answered questions like this in the past? Let's go ping them on Slack and assign them out. So like, you know, our lawyer should probably go answer this question. He's probably not spending all day sitting around Advanta, uh, but you know, as part of his job. And so let's go to like where he's working so we can ping him in Slack and be like, hey, this question came up. For this, you know, big enterprise customer, can you just give us the answer? And they can respond back in Slack. The agent grabs that and goes ahead and puts it in there and says like, okay, great. Like Ian signed off on this. He said this was the answer. We're good. You know, goes off into the, you know, into the response to the customer and we can go close the deal. So a few different ways I would say AI kind of plugs in at different points from pulling all the data in to answering the questions and the citations. So like and for the things we can't answer, helping people like get those answers in the right way.

That's kind of cool that using AI for like all the sort of pipelines to like build the rest of the AI as well as like actually using the AI in the end state. So yeah, you mentioned the evals and this seems to be like, it's a bit of a tricky issue technically. So you said, 'cause it's probabilistic, sometimes the feature's gonna give the right answer, sometimes it may do something completely stupid. How do you, how do you test it in that case? Is it like running queries, like asking questions, the same question 1,000 times and seeing how many times it gets it right?

Talk me through your testing approach. Yeah, definitely. Right now we use a tool called Langsmith that like helps us run all these evals. A lot of the time product managers are doing it as well as these GRC subject matter experts. Sometimes engineers and designers are in there too, but they're the primary ones. And so yeah, you can go through and kind of see all the different questions that have kind of like come in from customers if they're sharing that data with us. And then we can go through and we establish our criteria and mark them. So we could be like, oh, great. And then you kind of decide what kind of answers you want. So you can see the AI gave this answer. Do we want concise answers or not? You know what I mean? So how well did it do on conciseness? How well did it do on accuracy? And you can kind of create these different ones and then you rank them with just like a thumbs up, thumbs down of how well it did on each of these. And that helps you then go in and tune the prompt more or tune the model to get there. And then you can even have the models. I mean, to some degree, it's like, you know, turtles all the way down, you know what I mean? Where you can have them produce. More questions, and then you see it run through. You can also set them to do automatic evaluations as well. Um, so it will go through and you can be like, okay, these are the criteria I'm looking for. Look at the different ways I've judged it. And now you start coming up with your own questions and your own responses and see that a human goes through that evaluates those. But it is a kind of, you know, longer process. I think humans in the loop are incredibly important, um, especially because it's so easy for models to get confused on small semantic differences. Differences. You know, like we could be reading a big security document and, you know, a security incident versus a network incident, it'll be like, oh, those are the same ones. And they're like, actually, these are like materially different things. You know, if there's like a security breach versus there's an issue like with like the networking, you know, infrastructure or something like that. But sometimes you'll see it kind of get mixed up on some of those. And I think that's why, you know, so many companies are, you know, looking to tune kind of like, even if they are using a big foundational model, why we see such different results from these different companies because they're tuning it uniquely for that domain. You know, like, hey, we are tuning exactly for lawyers that need questions around these types of contracts, or for us, like security professionals that are trying to answer our questionnaire or trying to determine if the evidence they've produced will be accepted by their auditor. So we automatically run an eval on that and say like, hey, we know, you know, tons of auditors in the past have said this is a good type of system description. And so you can go into your audit with confidence that like, hey, I, I've produced the right kind of evidence. Okay.

So it sounds like there's quite a sophisticated system then, because you've got these layers of like, well, maybe the AI is like helping you both write the test and judge whether the tests are correct. Maybe you want humans in there. And, uh, it's probably gonna take a few iterations to get something good in terms of like just the system to test whether or not the rest of your work is any good.

Exactly. And like we have teams that are running these evals, you know, some of them on a daily basis, definitely all of them on like a weekly basis where they're kind of going through with a human-in-the-loop approach for all of them. And so, yeah, and then every time a new model comes out, things generally always get better, but sometimes certain things regress and you kind of have to go tune for that one.

Okay, so it sounds like it's kind of tricky dealing with the moving technology platform, but I'm curious, are you able to predict which things are gonna work well with the agent and which things aren't? It often seems with AI, it's like sometimes, you think it's gonna be like a tricky thing, actually, a simple prompt's gonna get you like the perfect answer. And sometimes it's like, this is really difficult to get it right. So what works well? What doesn't? Like, what do you find easy?

What did you find hard when building Banter Agent? Yeah, definitely. I mean, I think that, you know, I'd say the most success we had early on was with the questionnaires. Answering the questions once we got them, extracting the questions from all those different document types and things took a little bit of while to kind of get to. I think any of the really complicated questions, which are the higher value ones, have also been, has taken like more time. I think that the more recent models, like in the last 6 months, have gotten a lot better and we've gotten a lot better at evaling and our prompt tuning. So, you know, a GRC person will be like, hey, we have this risk around, you know, data governance, like our data governance rules are too lax at the company. And then they want to map that to a set of controls. They want to understand what's going on in the industry. What are the best practices? How they should quantify this business risk. 'Cause a lot of times an executive will be like, how much money would this cost the company? You know what I mean? Help me estimate that. And so I think those more complicated questions are ones that it's easy for the AI to kind of get confused or like not give a super clear answer to. So those are ones where we've had to spend a lot of time making sure our data model's clean. Um, oddly enough, like I find that a lot of the time, like if a human can't understand it or answer it easily, AI's definitely not gonna be able to. So A lot of the time, if we think about our development team, like internally, like our engineers, you know, anything we do to clean up our codebase and make it easier for a human to understand our codebase and write code generally makes all the AI models better as well. So it is just a general like, hey, you know, I mean, they're trained on kind of like how humans think, if you will. Um, and so I think those have been really helpful for us as we go through those like more complicated questions have taken more time to connect more pieces of you know, the platform together and are, and have more like vague outcomes. In those cases, we like to also just kind of suggest multiple options to people and let them know where it's like, hey, there's like 3 or 4 different ways you could go tackle this, which is more important to you. And I think that, you know, Cloud Code is probably the best example of this in my opinion, where if you, I don't know how much you've, you've written code or many people in the audience, but like when I'm using Cloud Code, a lot of times for big hairy changes and things like that, I'll put it in plan mode., and it's really great about coming back, asking me follow-up questions, asking me to clarify different pieces. I can go back and forth with it and like brainstorm together to get to the final output. Requires more from you, but I find that the end result is much better. And I think that's something a lot of agents, including Vanta Agent, are like digging into more around these follow-up questions, helping to clarify so that we can like focus the model on a really great answer.

Ah, yes. I mean, really, there's like, uh, two different approaches. You can either have Agent as collaborator. So it is asking you questions and wants that forward detail. And then there's agent as automator where it's just like, it will go and do the thing and get an answer. Then you're kind of looking at the final result and seeing whether it works. So I suppose, do you have a sense of like when one thing works better than the other?

Yeah. I mean, I think for the discrete focus tasks, like answering a questionnaire or things like that, we try to like, you can think there's basically like multiple, even though we refer to it as the Banta agent for us, you know, there are many agents underneath that kind of like look at the question and decide who they should federate it out to and which tools should get used. So one of the first passes we think about is kind of, hey, is this a simple, easy thing where we can just pull a discrete, highly factual answer? Yes. Oh, is this a really general question that's going to require more time and thought? Okay, great. Let's farm this out to multiple different subagents. Let's do multiple tool calls and kind of pull all that together. And that's when we're more likely to fall into this pattern of going back and forth with the user to like clarify so we can give them a great answer. And so I find those freeform questions that are more, you know, less like, hey, can you go change this thing in this policy from like A to B and just like do all the work for me because there's 16 policies and I need to update the approver to all of them. Just tell us to do it, we'll go do it. Or like, please make all my SLAs the same, or you know, do these things, but you're like, hey, I'm trying to think about in the next year, what are the top AI risks that are coming up? Can you like help look at my risk program, what other top risk programs look like, and determine like what gaps I might have? And I want to go pursue this new regulatory framework, you know, how, how can I do the least amount of work to do all these things together? I think those are the ones where we want to collaborate with, you know, uh, that security person.

Like the whole time it will be asking them more clarifying questions and things like that.

Absolutely.

So yeah, certainly as things get more complicated and the stakes get higher and the project or the problem gets longer to solve, then yeah, that's when you need the human collaboration element.

Okay.

So I'm curious as to what skills you need in order to go about creating agents. Like, I guess, talk me through like what you've been looking for, like with your team, like what makes a good agent creator?

Yeah. I mean, I think that like entrepreneurial skills are like very important. I think someone who kind of has that builder DNA is probably the term we use the most. Internally where you don't really see boundaries and you just want to like solve this customer problem. And I think are excited about this technology and what it can go do. Um, I think if you get a little too framework-based where you want everything to be perfect and deterministic and to fit into nice, neat little boxes, sometimes it can be like infuriating for people like that. They're really important and we need them in other parts of like Vanta and many other companies as well. But like usually being on the bleeding edge of some of the agent stuff is not the thing that gets them the most excited. Excitement. So I think for us, when we're looking at those people, it's someone who will go deep with customers and really understand what their problems are, someone who has kind of an innate sense of like what a high-quality answer is for this area. So we actually hire a lot of people that aren't security subject matter experts but can partner with our security subject matter expert team to really do that like last-mile eval. I think that's like a key part of it, uh, for the agents, understanding the kind of like basic workflows of someone that's living in it every day, you know, especially more on the cutting edge. It's like you don't need to be technical to do all these things. But I do think the most interesting AI experiences generally for me are all like coding experiences. And maybe it's because I come from like an engineering background. But when I think about, you know, Cursor, Claude Code, Codex, a lot of these, I think someone who is spending a lot of time in those experiences and see how people are pushing the edge, you know, whether it be like Lovable, v0, um, you know, Figma Make, things like that. You know, pretty early on we ended up buying licenses for v0 for everyone on our product management team. And then our design team uses Figma Make a lot as well as v0. And we were just like, great, like just start building prototypes. And I felt like our technical people jumped on it really quick. The less technical ones were like, I bet I can't do this. I bet it's hard. And I actually just recorded a Loom video and like sent it out to a bunch of people and I was like, I got this all set up for you. Here's how you do it. It's a lot easier than you think. And now. We have product managers and designers, some of whom have like very little technical, you know, experience in like writing code, now going out to customers and creating full end-to-end workflows and prototypes and validating ideas in like, you know, days or hours where before it would be like, oh, how do we get this on the engineer's backlog? Get them to go build it for you and do all that. So I think that quick cycle time and someone who's excited by how everything is changing all the time and like wants to lean into that. Are the people that are, are really creating these like exciting experiences.

That's very cool. So I was kind of expecting some of the things you said, like, uh, okay, having like a builder mindset and a few technical skills, but the idea of just being entrepreneurial in that you can just sort of dive into a problem where you're not sure what the answer is going to be and just give it a go and try something out, that, that's really interesting that that's the way to go and that's one of the big sort of criteria for success there.

Yeah, definitely. I mean, I feel like there is a lot of, we just still don't know everything that AI is good at. And someone who has that built-in curiosity of just like, can I use it to solve this problem? Let me try. And like the persistence to try it enough to know if it's going to work or not. Because some of the stuff that didn't work, you know, months ago now works better now. Like I think that, you know, at least, you know, for me, when I think about like Opus 4.5, I think it just did 4.6 like last week. That was a kind of watershed moment for me for Claude code getting like significantly better, which was like around November, December timeframe. I think computer use, if people are familiar with that, where it's like you basically can have like the AI run a browser for you and navigate to different sites and do things. It definitely feels like it's probably lagging 6 to maybe 9 months behind like where everything else is, but it's like getting a lot better really quickly. So that's something that we've been investing heavily in internally and just being, I kind of waiting for that moment when we're like, it feels like it's on the trajectory to be like mass adoption and let's make sure we're always investing in it. So when that time comes and it flips over, like we are ready to go ship something. And we have a ton of customers using Computer Use today for a bunch of scenarios of like automating screenshot taking and things like that for evidence gathering if there isn't an API.

Oh man. So, uh, Computer Use is really interesting because it, on the one hand, it's a very cool thing. Uh, if you want to just automate any kind of task, it's like, it's basically like it's the new version of a macro, right? Like I guess you've had Excel macros for decades, but this is like the new cool thing., but it's also a big security risk. So I guess, uh, if you're sort of considering adopting Computer Use or one of these sort of similar tools, uh, like what do you need to do to, to do it safely?

Yeah, I mean, I think that one of the things we did in Vanta is like we started off in the beginning with all of our automation always being read-only, and you have to opt in to write actions. And I think that helps. It doesn't help everything. Obviously there could be data exfiltration and pulling and leaking things, but at least you're just not like it should be extremely rare that like a reading some data will take down production or do something like that in that vein. And so I think having customers opt into these like right actions kind of comes back. And then once they do, I would say, you know, downmarket smaller companies are much more open to this type of thing where they're kind of like AI native. They want to push the boundaries. They're just trying to find product market fit and they're willing to kind of like take more risk there and say like really large enterprises are not willing to take that risk nearly as much. So I think having, understanding who your customers are, what options they want. We definitely have customers where we start read-only, they opt into these kind of write use cases and then giving them that granularity of control where it's like, yeah, I'm good with you like updating my policies or are you updating this evidence or updating these controls? And then I think the highest level is like computer use, probably read-only where they can pull the data in the screenshots. There'll be a lot more comfort with it than logging into your AWS account and like changing some very important configuration setting in AWS without you knowing it. Feels like it's going to be pretty far off from people just kind of like letting the agent YOLO those types of things. And I think what we see now is more like review and approve where we're building to like, hey, we've done all the work for you to tee this thing up. It's now at that moment. Do you want it to go do it for you? So you can quickly kind of get an idea of like what it did to get to this point and like, yes, I feel good. It looks good. Go ahead and do the thing. So maybe you don't have to go log in and like twiddle the knob or remember exactly which page and like the Google console or Azure console the settings on, but you at least know. So it's like, here's the exact thing it's going to change and you can build confidence. So that helps you go faster. But I do think it's these layers of kind of read-only, writing to some simple things, read-only for some of these other things. But computer use, I definitely think is extremely powerful in both like the good and the, and the bad because like it has access to a full like VM or container and can kind of, can do what, whatever you give it. Um, so I think people being able to lock down those, and there's a lot of companies that are providing kind of these isolated. Untrusted, you know, code execution environments for computer use to run in, which is like what we use internally too.

Okay, that sounds sensible, is like, give it a sandbox, like make sure it can't destroy everything, can't delete your whole AWS account. That could be, I mean, there's potential for some real big disasters there if you just let like AI do stuff on enterprise accounts, like stuff in production. But maybe we take a step back, like what are the big success stories or use cases you've seen at the moment? What are the cool things you can do with this sort of automation?

Yeah, I mean, I think some of the big ones that we've seen for people is the questionnaire automation has been huge. We've seen a ton of customers get that and just save hours upon hours and hours of time, which is great. Like, security reviews now, you know, take like 20% of the time that they used to going through these questionnaires. Some— in a bunch of cases, we're getting like 100% coverage rates, which is like phenomenal. And then if we don't cover something, we automatically let you know like what type of answers need or policies or anything, so you can't get those coverage rates up. Um, I think on the vendor assessment side, no one likes sending those nag emails to like a vendor asking them for like all the evidence and all their documentation and pouring through those hundreds of pages of documents you get fast. So I think the Vanta AI agent has been really helpful in like streamlining those reviews. I think since from October to, I think it was like the end of January, we already cut that number in half for all of our customers, the average amount of time a security review takes. And we think we can cut it in half at least one or two more times, which is huge. And that's like the total wall clock time. Obviously the AI is really fast, but sometimes you're waiting on the human to like log in and kind of do that review and approve. I think though, the thing gets me excited is, you know, a lot of security teams don't have that feedback loop that kind of engineering and product teams have had forever. Right. You know, it's like, hey, you ship a product to your customers, you get all this data and telemetry back and like feedback, things they love, things they don't love, like how they're using the product, then you can make it better. For a long time, security teams were just like, all right, we're going to get some certifications or we're going to kind of do some stuff. And they're kind of just stuck. They don't like know what customers' opinions are. They're not connected to revenue. Um, and it makes it hard for them to kind of justify their, their budgets and explain to like management, like what they're doing and like the board, other than just like reducing risk kind of becomes this generic insurance thing of like, oh, I guess we need to spend a bunch of money on this, but I'm not sure the benefit we're getting. So one of the things that Vanta does is really ties all that feedback you get from customers in questionnaires, in conversations, the contracts you sign, the unique commitments you're making in those contracts down to like revenue. So it really changes the job of a CISO where now they can come in and be like, hey, my team contributed to like, you know, whatever, 90% of the revenue we got last year by answering these questionnaires. These customers wanted these security proof points. They're now asking us to do these other things or asking for like changes in our contracts to go do this. These are the changes we're going to make in the security team to help them, like, justify their budget and their headcount requests and, like, better show off the amazing value that they're doing. And so I think the agent being able to attach all these pieces where it can analyze all this data you're getting from your customers, what your controls and frameworks and risks in your program is currently doing, and then how you can turn that back around to, like, help grow the business. I think just fundamentally changes this notion of, like, security just being this, like, nebulous cost center that, like, takes a ton of money. And, you know, a lot of executives don't understand what they're doing. They use these confusing terms and all this jargon. And now we can be like, oh no, they are actually powering the heart of your business. And the agent helps them easily, like, tell that story and do it continuously.

I love the idea of, like, nebulous cost center. Like, a lot of those things you said about security teams, it's the same with, like, data teams as well. It's like, I think, pretty sure they're doing something useful, not sure what. And, uh, yeah, being able to demonstrate that value to executives, to the rest of the business, that's incredibly important. So, uh, yeah, I love the idea. You get rid of the, the, the rote work, outsource that to AI, and then you can focus on showing your value. Uh, fantastic stuff. All right, uh, just to finish up, I always want to learn from more people. So, uh, whose work are you most interested in right now?

Who should I be following? Oh, I think, um, most of what I've been following have a bunch of the engineers, probably from Anthropic, I would say, who are like on the kind of like cutting edge of going through and understanding like where models are headed, what they're doing. Um, I'm always interested in Yann LeCun. Probably a lot of people end up following him as well. He has like a new startup there. And I think even if he can be a little bit more negative on like LLMs, I do think he's right. They will hit a point where it's not like true intelligence, but like we continue to eke more out of them constantly. But I think he has a great perspective of like what the future of AI could be, what the limit of what we've seen today is. And so.

So definitely someone that I always like to follow. Oh yeah, Yann LeCuny, he's been thinking so deeply about these problems for decades now that he's always good for a spicy quote as well. So yeah, definitely someone worth following. All right, thank you so much for your time, Jeremy.

That was a lot of fun. Great. Thanks for having me. It was a great conversation.