EP251 Beyond Fancy Scripts: Can AI Red Teaming Find Truly Novel Attacks?

Security running.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, myself, Tim Peacock, the Senior PM for Detection, UEBA, and Risk in Google SecOps, and of course, Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners all on the Google Cloud Security Community page. Anton, we have a super fun episode today. We're joined by one of my friends from life in the city, early scientist at OpenAI, and somebody creating, as far as I can tell, Terminator, right? Skynet?

No, not Terminator. I think that this was very clearly an augment use case for AI, not a replace. I feel like we've asked him 3 or 4 times during the show, is he looking to replace— to not spoil it— this particular role in security? And he said, no, no, no, no, no. This really helps the good person in that role to do much better rather than it replaces a bad person in that role.

And I believe that, but it still on some level is disconcerting to me that we're teaching the machines to do this.

To me, everything that is alluding to or pointing at superhuman abilities is that part. And you can always counter by saying that a 1960s transistor calculator can multiply 6-digit numbers better than a human. So, we've achieved superhuman abilities in at least one area 50 years ago, and everybody is okay more or less with that, right? Except some school teachers perhaps.

Perhaps.

For some reason, finding the true unknown vulnerability feels more risky than multiplying 6-digit numbers, to me at least.

Yes, it feels much more risky to me.

Thank you. Thank you for agreeing with this.

Yeah.

No, I'm with you on that. Although during the episode, you were a lot more sanguine about things than I was.

I was triggered by that point. And initially, I was kind of impressed but also nervous. And it reminds me— I don't want to go on a rant here, but it reminds me that—

Oh, no. Go on a rant.

Whoever thinks that AI would destroy humanity is ultimately an AI optimist. Because he believes that AI has powers to do it. And this, when I realized it many months ago, I was just shocked by that realization that if somebody says, you know, this can happen, and I'm like, you're an optimist. He's like, no, no, no, no, no, we are talking about destroying humanity. And I'm like, you are overestimating the capabilities of this technology, so you're an optimist. It's odd, but it also kind of points out an odd time we live in.

Okay, well, before we get any more catch-22 AI optimism, AI pessimism, let's turn things over to today's guest. Today we are joined by Ari Herbert-Voss, founder and CEO of RunCivil. Ari, thank you for joining us today. Listeners, I want to give you all a little bit of background about Ari before we get in. Ari, early, early at OpenAI, you dropped out of Harvard, is that right?

Yeah, that's right.

Yeah, so like classic quintessential Silicon Valley AI sort of story here. You've now got a startup which is building AI red teaming. Is it really a good idea to teach the AI how to hack people? Let's start there.

Oh, absolutely. I think it's essential.

Okay, say more.

All right, so let's chat a little bit about what it is that we're actually building. So the company is called RunSibyl. We want people to run Sibyl, which is the name of our set of agents, everywhere. So basically, the goal of the company is we're automating hacker intuition, which we can break out into 3 phases. So your first phase, of course, is discovery, where you want to see, like, what are all the assets available to you. The second phase is the testing, where you then test all those assets that you've discovered. And then the third And then the third section or segment is remediation, where you then want to fix all the stuff that you found.

So 3 phases: intuition, testing, remediation. Are you also doing that third stage, or are you really right now just focused on discovering things and owning them?

Primarily the first 2, but we do have fixing capabilities too. We don't want to leave our customers in the lurch where we find a bunch of stuff and then we're like, cool, we found a bunch of stuff, have fun with it. So we focus on prioritization, we give remediation steps, and we also tell you how you would replicate the bugs that we find.

Oh, that's cool. Anton, I think that connects to kind of the question you wanted to ask here.

Yeah, because it's 2025, right? And a lot of people are building some form of AI for offense, for pentesting. And at this point, I'm just going to offend everybody by roughly equating offense, pentesting, and red teaming, which of course are not the same.

Oh boy.

But the point is that— is the point that I'm making, it's not AI for SOC triage. It's AI for either testing or offense. And this is a very common pursuit for a lot of people to try that. But when they do that, it's always unclear whether they're summarizing large pentest reports, whether they're prioritizing, whether— if they're actually hacking or if they're doing other things. So, which gives me a bit of a so what reaction. Oh yeah, you have AI for pentesting. So what? Give us more about like where the value sits. Like, what are you doing specifically? Are you discovering new issues? Are you discovering unknown unknowns? Or are you just summarizing reports?

So I think we can actually talk about this in the context of whether or not AI is meant to replace a human red teamer, a human pen tester, or something else in between, or if it's meant to be an augmentation. So let's first talk about the distinction between red teaming and pen testing, because it is actually important here. So you can view getting a pen test as essentially the same as hiring a bunch of pirates to see if they can steal your jewels. So they'll test to see if you lock the doors, they'll test the integrity of those locks, and they'll try to drill through your floor all to see if they can figure out where your jewels are hiding. Red teaming is a lot more like hiring a bunch of ninjas to tell you if the laser detection systems that you've set up are actually set up properly. You really only see red teaming at organizations that have a mature blue team function. Otherwise, it's like you're bringing a gun to a knife fight and the security team isn't really going to benefit very much. So we view Sybil as a replacement for pen testing and an augmentation for red teaming.

Huh.

A way to view Sybil is as a solution to the slow and monotonous, but critical work that's done as part of a deep security review. Because in our experience, hackers are like herding cats. They really enjoy the creative parts of the job the most. And it's not like testing every web app to make sure authentication or authorization is working correctly across all user roles and, and anything like that gets them hyped. That's a lot of like spreadsheet work to make sure that like everything is buttoned up properly. So we're focused on really automating that stuff because it is critical. It's just also very mind-numbing.

I'm trying to think of what sort of mythological enemy sits between pirates and ninjas. Like, I appreciate the difference between pirates, which are brutish and loud and full of grog, and ninjas, which are silent and stealthy and will take out, you know, the emperor. I'm not sure who fits in the middle there, but I, I like that. Sort of spectrum to exist on.

Yeah, well, I would say that Sibyl sits in the middle.

Not zombies though.

No, no, I think zombies are even less organized than pirates. But I also totally appreciate that hackers are super interested in playing with a laser pointer and very uninterested in balling up the yarn after they're done playing with the ball of yarn, to mix cat metaphors at the same time.

So you just made an interesting point that your technology can replace a pen tester but augment a red teamer. That sounds profoundly important to me. Can you clarify this?

Yeah. Let's dive into what a red team actually is compared to a pen test. Of course, we mentioned the pirates versus ninjas, but in terms of an actual organization that would buy a pen test or get the services of a pen testing team versus having their own internal red teaming, a lot of that has to do with the scale of the organization. I was the first security person at OpenAI. Around the time that I left was when we became mature enough to actually start building out a red team. And that was about 3 years into my tenure there. Usually you don't build a red team unless you have a mature blue team function, like somebody that can build detections. Because otherwise, as I mentioned, you're bringing a gun to a knife fight and it's not really clear if you're going to get the utility that you want out of that. So we are focused on replacing pen testing because it's a third-party security review function. We don't think that we'll replace all of pen testing. I think that there's still a very strong and healthy market for high-end boutique services, like the sort of places that have— if you have very complex infrastructure and you want to have a team of experts who are external to your organization look at them, I think those Companies are going to be fine. And in fact, I think they'll also benefit a lot from having AI-powered tools. But we're focused on taking care of the check-the-box functions while also being able to deliver more with regards to the kind of complex vulnerabilities we can find, the speed at which we can find them, and then being able to build it. If you think that you're ready for a red team, but you're not quite sure yet, we can provide a solution to that.

That makes a ton of sense to me. One of the things that you mentioned before we got together for this episode is that RunSibyl is particularly good, or Sibyl, I guess Sibyl is the agent. Sibyl happens to be particularly good at finding auth bugs. Why is the agent so good at auth bugs?

I'll say that authentication authorization bugs are important for a couple of reasons. So first, they often are some of the most impactful issues that you can have, and they're also very hard to scan for. Unauthorized access, protected information leaks, privacy incidents, they all stem from some sort of auth issue, and they all have very severe reporting and regulatory and headline impacts on the business and also the end users, which should stand to reason that you would hope that there would be some sort of scanner that would come out that can solve those sort of problems, but we haven't. And I would say that the IAM industry is pretty robust right now. There's a lot of competition in that space. There's a lot of interesting solutions that continue to come out, especially in the age of agents. But I would say that the thing that allows Sybil to find these bugs much better is that you kind of can't find these bugs if you're just going to script for them, which is traditionally how scanners have been built. Being able to tear into the complex infrastructure that makes up how your organization set up authentication in the first place, being able to understand the context in which an asset exists. So if Sybil is able to dig in and find an asset, say like an IDOR, where you look at a URL and you can tell through the URL, like, it looks like I'm this particular user, but I'm not supposed to have access to this asset, that's not something that traditionally you could scan for.

I'm also curious more about what is this so special about the auth bugs? that makes them more AI-findable? I think that you answered the other half, is that why is it finding auth bugs, but it's not finding overflows in shell commands, but it's finding auth bugs. Is there any way to explain why this kind of machine brain is finding more auth bugs than, say, overflows in some other code?

Yeah. In some examples, we can find those types of bugs, but we've focused most of our efforts on authentication because it's, you know, at the top of most bug bounty platforms. They're always the highest reward category. And people are usually generally bad at testing complex apps. I will say, like, as far as limitations for, say, LLM-based approaches, anything that deals with having to remember the state of something is going to be something that an LLM particularly struggles at, unless you've built a good harness for handling those kind of bugs. It's not to say that buffer overflows and things like that aren't important. They're just not what we've been focusing on for our customer segment, because our customers are coming to us being like, hey, I have a giant web surface. I've got cloud. I've got some other interesting things, mobile.

Hmm.

Could you maybe take a look at that? And they don't care so much about the memory overflow bugs. Those you usually find in older infrastructure that we haven't focused our efforts on. Hmm.

That makes sense. And I think that the whole point about if you can just gently slam the front door and you're in, that's the authentication bug. You don't even need to have like a sneaky tunnel under the house, a helicopter, and in. You can just like, not even hit the front door, but you like gently push it and you're in. Then that's like That's really the extent of it, right?

That's huge. Another thing to note about the economics of red teams is that you only find them at large organizations because that's really who is willing to pay for having people come in and just bang around on stuff because it's very expensive. Like, it's traditionally a very human labor-intensive part of our industry. And traditionally, we spend a lot more on blue than we do spend on red, which I think AI is one of the things that is going to help with a bit, not in terms of increasing spend, but being able to increase coverage.

Well, increasing spend as well. Let's be honest here. Okay, maybe that too.

Certainly good at that. So let's talk about the blue team side of this. So as you're finding these issues, what is Sybil generating to help the blue team close things and prioritize things? And where do you think we're going as an industry and as an AI researcher? It's very obvious that AI is getting good at the offensive side of this. How are you helping the blue side of this?

Yeah, that's a great question. It sounds like the question is about purple teaming, and it's a great segue. So I want to dive into a little bit about the economics of a purple team.

Yeah.

So relative to red teams and blue teams, purple teams are actually quite rare in our industry. You really only see them in places where there's both robust red and blue functionality. As I mentioned, blue is something that you usually build up first because you want to see like, ah, it looks like we need a SIEM, looks like we need to look at all the other things that are kind of happening on the network, and we want to see if there's weird stuff and how to keep it out. And then red is like, ah, looks like blue is mature enough, let's test those lasers. And then purple teams are And purple is where usually you have robust functionality between the two of those things. But I would also argue that they often reflect more of a stylistic choice of the organization rather than an industry standard practice. Like, some security leaders really like to have that kind of meshing between the two teams. Others find a lot more value being able to have them be separate. So you really do get to test whether or not your blue team is doing the right things by having red team be entirely separate. Our goal is to close the loop quickly on vulnerabilities, which means taking up the operational load for prioritization, providing clear steps for reproduction and for remediation. So we also go one step further and then track the attack surface over time to then help security leaders prioritize and communicate risks up and down the management chain. This is relevant to purple team insofar as being able to suggest fixes and help drive reproduction and remediation is very similar to what a purple team would serve if we're talking about like the detection side of the house. But we explicitly haven't focused on that so far. We can support purple team functionality for places that traditionally don't have it, much like We can support red team functionality for places that are on the cusp of having it. But our focus has been primarily on the discovery, testing, and remediation rather than hooking into detection right now. But I will say, by virtue of how Sybil works, we have pretty high fidelity attack chains that you can actually translate into detection rules. But so far, that's not something that our customers have asked for yet.

That makes sense.

So my mind is kind of split between 2 things, which are both equally fine, I guess. I really want to ask both, maybe. I'm nervous about the fixing. You kind of covered some of the fixing, but ultimately, in my experience, if you ask a red teamer or a pen tester about the fixing, he says, it's not my job, let the other guy worry, right? But ultimately, this stance has not really helped all that much. So, what can make your findings more fixable, if I may ask that?

I would say that, I mean, your intuition is right. Like, there's not a lot of impetus on the behalf of red teamers to provide any fixes. But there is also plenty of data out there that shows patching information, and those are things that LLMs can absorb and use. But our focus as a company has been a lot more on the first 2 parts of that 3-pronged stool of pen testing, where we've been focusing on discovery, we've been focusing on testing, and then as remediation, we offer that as a treat. I would also think that remediation is one of those things that's very context dependent. So understanding a bit about what are the assets that are there, like historically, why have they been built this way, versus just coming in with a hammer and just being like, this is how you should fix it, hasn't proved to be particularly useful. So we're able to provide several different suggestions of how people can fix things versus just being like, here, we'll open a PR and we'll offer you a fix. Because we have tried this in the past, but we've actually found that customers don't really want to have a prescriptive fix so much. They want to have a little bit more control over the ability to fix things. So being able to offer a couple of different ways through remediation is one of the ways in which we address that.

That actually makes sense. And I think to me, this is This is probably as much help with fixing as it's conceivable or imaginable. So I think I'll be at peace with that.

So we're at this stage where the software is good enough to do pen testing. What happens in the next 3 years when it continues to get better at the rate it's been getting better, or if it continues to get better at the rate it's been getting better? Like, what next?

Well, I think a lot of people are kind of concerned right now about the possibility of there being an AI bubble and whether or not we've hit the end of the road as far as scaling laws go. But I will say that even if all AI progress were to stop today, there's still so much more that you can squeeze out of these models that we're benefiting from every day that I don't think all progress would actually stop here. In the next 3 years, I anticipate that it just gets more accessible for people to access the benefits of using an LLM to do a number of different tasks. But one of the things that we're really seeing is the speed is increasing.

That makes sense.

That part makes sense, and it's part as exciting as well because For some of the heavily manual things, the slow means it won't get done.

Exactly. That's exactly what I find exciting.

Speed is not the quantitative difference, but it's the qualitative difference. If I'm told it would take me a month, but I need results in a week, that means I cannot do it. But with updated tech, I can. So, I want to go and ask you a very strange, very vendory question. I mean, you are a vendor in that sense, right? What do customers like the most about your tech? The reason I'm asking this is you mentioned speed, you mentioned accuracy, you mentioned better prioritization, you mentioned a whole bunch of other things that are quite impressive. But do you have anything where people tell you that you're good because X, and what's the X?

Oh, yes. I have 2 customer profiles I can share. So, we've heard from customers that having Sybil is a pressure reliever for product security teams that wish that they had a permanent external review function but can't justify hiring a red team quite yet. So, those are the folks that are getting a lot out of their pen test, but they haven't quite hit that maturity on the blue team teams out of the house in order to justify having a red team. There is so much more demand from developers and product owners than security teams can ever really provide. And this is me speaking from my experience at OpenAI. Anybody who's been early at a company on the security team understands the pain of devs being super excited to ship things. And then on Friday, they'll just tack on a meeting where they're like, oh yeah, we should have a security review for this thing. Here's a week, and then we're going to go ship it. And then there's this mad scramble to be like, oh man, we have to figure out what even is this thing in the first place? What are some of the possible things that could happen with it? So having Civel enables you to scale out that offensive security capacity basically overnight. So, all of the product questions about what happens when we push this feature to prod, or did this refactor introduce new security issues we haven't caught, those can all get answered without having to slow down development. And that's, like, the number one thing for any sort of security company, or any security function at a company, is you never want to slow down development. Yeah.

That was one of the conversations we were having a couple episodes ago, was actually, how can security speed up development?

Beautiful.

Weirdly enough, this gives you a huge leg up over people who automate the junior pentester. Imagine they replaced a monkey with Nessus to robot with Nessus. Ultimately, the second one makes developers' life worse, not better. So in essence, they are, from the developer point of view, that AI for pentesting is regress, not progress. Yours is progress, but theirs is regress.

That's what we're trying to do.

Ari, can you imagine how much most startups would've paid Anton as a Gartner analyst to say that about their company.

I'm taking it as a very, very high compliment.

You're overspending me.

Like you could not have bought such a positive quote out of Anton ten years ago, and now he's here saying it for free.

I love it.

I'm a best B lister. No, you're overselling me. No, no, no.

That's amazing. So I hate to do this because we're having so much fun, but I have to ask our traditional closing question.

Well, hold on. I have one more thing I wanna share.

Oh, please.

Please.

Yes. Oh, love it.

Listeners, I've been unclosed. I've never been unclosed before.

Oh.

This is exciting.

Please.

I'm getting all sorts of firsts.

This is great.

I also wanted to add about the feedback that we've gotten from mature red teams too. Yeah.

Oh, yeah.

To highlight the quality. So we've heard from these mature red teams that Sybil helps them scale up their operations. So I actually have an anonymized customer service about how we found something that their human team didn't, which is fantastic. So we did some work recently, or actually we started some work recently with a large enterprise tech company in the AI infrastructure space, and along with some other issues, we actually found an RCE, remote code execution, in a part of their attack surface that had been exposed to the internet for a pretty long time. And on top of this, this is in scope for a well-funded and popular bug bounty program. This has been the case for years. And the fact that a high-impact valuable bug could lie in wait and evade heaps of bug bounty researchers really shows the power of what it is that we have to offer. And we found this within a day while the humans whose day job it is to find and then solve these bugs weren't able to. And that's because we're Cybill is more thorough and persistent than what a human could ever be. Mature red teams have enjoyed working with us so far because we can augment them by not only finding these high-quality bugs and assets that they don't have the bandwidth for, but over time, we can also help them prioritize which assets to go after and when. We've been focused not only on finding, but also figuring out how to augment the actual operations of a red team and why it is that people buy pen tests versus red teams. If I had to sum it up, I'd say that the primary benefit of AI-powered red teaming is that a human can't compete If I had to sum it up, I'd say the primary benefit of AI-powered red teaming over humans is that humans can't compete with coverage. Coverage.

That to me is a good argument because while you can debate whether you have the higher depth and the best of the best of the best of ex-NSA, ex-TAO pen testers, red teamers, but they would not compete with coverage. I agree.

Absolutely. And having worked with really high-quality pen tests and red teams, I know that Oftentimes the buyer-seller process involves, you know, bringing somebody out and being like, oh yeah, you should meet this guy. He hacked Oprah and Jay-Z and all these people. And you're like, wow, that's really cool. But that's usually not who you're actually getting. You're usually getting somebody who maybe joined the team 2 weeks ago and is running through a checklist. Well, if we can automate the checklist stuff and then allow you to maybe work, if you choose, with some of these higher-skilled folks, but also be able to continue to find bugs that a higher-skilled person would be able to also find, there's a lot of value there. And being able to just do that all through the amount of coverage that you get. That's the sweet spot.

That makes sense.

So listeners, I hate to do this. It's my least favorite part of every show, but I have to ask our traditional closing questions. Ari, could you give us some advice for people who are thinking about how to start getting more out of, say, their red team budget? And 2, do you have recommended reading for people? And that recommended reading could really be anything.

Wow. Okay. Well, I'd say for the red team budget, make sure that you have good development practices and baseline security hygiene. And you also want to make sure that you have a blue team first. I would say that nothing demoralizes both a red team and the developers faster than having a red team come in, tear things apart, and then hear, oh, yeah, yeah, yeah, we already heard about all that. We already knew, and we don't have time to fix it.

Ugh.

A red team's purpose is to validate that all your controls, processes, and investments into security and dev tooling are actually paying off. And so, if those things aren't in place, then what are you spending the money on? You're spending very expensive, highly paid, very talented people You're paying them a bunch of money to basically find a bunch of stuff that they already found. Very boring.

We had another guest on the show once who described how their CISO was using threat intelligence reports, and it was to produce fun reading material while they were in the bathroom.

Oh my God.

Yeah. So we don't want red team reports to end up that way. And then do you have recommended reading? Perhaps not for in the bathroom.

Yeah, well, anybody can read anything anywhere. I've got like 2 recommendations. One is if you're really curious about red teams, I'd say that Kim Zetter's blog and her book Countdown to Zero Day is actually a fantastic window into not only the wild technical feats of major hacks, but also the geopolitical impacts and human elements that underpin so much of what we might think of as just code and command. It's a bit older, not super old, but just it's not been talked about most recently, but it is one of my favorites. I'd also say that one of the other books I've been enjoying is Empires of AI. I think it's really interesting, especially right now, as people are trying to figure out what's happening in AI, I think it's also interesting to kind of read the history of where these companies come from as a way to kind of predict where they might go. I'm pretty bullish on the ability of AI to make things better for everyone, but I also like to read takes from everybody to get a better sense of different perspectives on how things are.

Awesome. Well, Ari, thank you so much for joining us today.

Thank you. Really had a good time.

And now we are at time. Thank you very much for listening and of course for subscribing. Please subscribe so you can get new episodes piping hot And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcasts. You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us. And if we like or hate what we hear, we can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast by Google.