EP249 Data First: What Really Makes Your SOC 'AI Ready'?

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Tim Peacock, the Senior PM for Detection, UEBA, and Risk in Google SecOps, and of course, Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on the Google Cloud Security Community page. Anton, we are talking with Manasi today about AI, Hulk, and whether or not Terminators are good. Is that right?

Yeah. No, I don't think there was a question at the end. I think it was kind of more of a statement, but it's a really fun episode and also, I feel like if we ever air this episode as a video, we'll probably have to stop making jokes. If you were watching it, he would make a cringe face. You actually have a chance of seeing our faces. We're trying to shift more of this to video, and I think this episode specifically would be really fun as a video.

So, if you want to find the video, you can check us out on YouTube, youtube.com/@cloudsecpodcast. That's how you can find us on YouTube, or search for Cloud Security Podcast by Google. But this episode had a really, really interesting insight in it around the correct number of times to get something wrong. And it turns out, listeners, the correct number of times to get something wrong is greater than zero in this case.

In this case. But it also reminded us quite robustly of a problem that everybody seemed to have, is that they judge robots much more harshly than humans.

That's right.

And I've seen evidence of that in discussions of self-driving, sort of automated driving.

You see this in Waymo conversations all the time.

In Waymo convos. I've seen other places where they say, but AI cannot possibly be good for this task. It can make mistakes. And it's like, and humans don't? Or like, what is your point? Like, yes, sure, it can make mistakes, but surely it makes sense to compare mistake rates as opposed to like make vague judgmental comments about machines making mistakes.

You know what's possible is maybe the people that say that just have a really high opinion of their human colleagues and think their peers never mess up. No, no, that's not the case.

I can do better. They have very high opinions of themselves.

Ah, see, listeners, this might be one of those cases where it takes one to know one, and Anton is seeing himself in other people. Yes. Yes.

Nice.

All right. Well, before we get too into the psychological examination here, maybe let's turn things over to today's guest. Today we are joined by Monzi Mirza, the co-founder and CEO of Crocal. Monzi, thank you so much for joining us today. I want to start us off with kind of a fun topic. Which is we hear about this aspirational SOC Iron Man suit, like Iron Man suit for the SOC, something to make analysts more productive, maybe zoom around and have some flippant jokes when they pull up their visor. That's cool. What stands in the way of us achieving that? And maybe what's different? Surely if we've had problems achieving that for 40 years, we've known for a long time people would like this. What's different now?

Yeah, well, thanks for having me. I mean, the, the main question, the point is like what historically things have been skipped. There's always been this idea of wanting to do more, wanting to do better, wanting to be more complete, whether you roll the clock back all the way when we were looking at router tables or now forward. The opportunity that really exists today, which is also the detriment really to be transparent about it, is that we've seen what large language models can do. We can see what agentic frameworks can do. And security people having these really difficult mission-critical challenges are saying, hey, I have a tool now potentially that I can utilize. So it's very much an aspiration from the security community saying, hey, maybe I can solve a problem for myself here. And when we do some early testing on this as practitioners, we sit on ChatGPT or some other LLM, assistant interface and type something, we get a response back. And so our optimistic brain says, hey, maybe this will actually work for me. But there is a big delta there, and maybe we'll talk about that more. But that's where the opportunity comes from, is these frameworks, is these access to these frameworks and these capabilities now where the security community is looking, going, hey, maybe there is an advantage here for us and we ought to utilize it.

It certainly feels like there ought to be something there, right?

Yeah. But I mean, I actually have a completely different side comment This is that we are talking about the Iron Man rather than the, pardon my metaphors, the Terminator, or the automated machine that does the work.

Yes. Okay.

I'm using Terminator. Imagine Terminator, but good. Terminator is bad.

Yeah. Yeah. The nuance, I think the nuance there is the realization. If we rewind maybe a couple of years ago, right, there was quite a bit of commentary in the security community and other communities. Oh, generative AI, we're only 3 years out or 4 years out. And it's going to replace a whole bunch of other stuff. And even companies that were creating products for the security community were saying things like, oh, we're going to eliminate the analyst, we're going to eliminate this tier or that tier or this kind of activity, right? So that was something that was happening. And so I think the idea behind the concept, like we like to use the Iron Man suit concept because the idea is that it is the human user that puts the suit on and drives the capability that the suit has.

Mm-hmm.

Rather than in the Terminator kind of situation where it is this thing that is completely Completely on its own, and it's going to do whatever it feels like doing is appropriate. And so the Iron Man suit opportunity is there for sure. And I think that is the right frame of reference in order to do that because there's lots of complexity to the security operations world, whether it's in its investigations or threat hunting or compliance. So pick any component of that work discipline. There is a lot there for at least at this point in time just to say. Well, something is just going to go run any one of those pillars together. So, so I think that's been a little bit of a shift. And I would say, I mean, if you want to be controversial, I think part of the reason is because when those people who came out gunslinging saying, you know, this is going to be a complete replacement of Tier 1, Tier N, whatever, are realizing it's not that trivial.

I think it's really interesting, this Iron Man analogy, because for so long, security tooling that I've used has made me feel like Hulk smash. So it's nice that we're going from thing that makes me angry to things that lets me fly. Anton, you've got another—

Yeah, I have another—

Literary analogy for us to shift to. Yeah, please give us the next literary reference.

Exactly. I feel like we are going from Iron Man through Terminator to Hulk, and now we're going to pivot to Jekyll and Hyde because to me, this kind of Jekyll and Hyde dynamic to the SOC automation and to SOC AI in the SOC specifically. So just give us the good news first. And then, of course, oh, wow, I get stuck with good news and Tim gets stuck with the bad news. Well, okay, fine. It happens. So give us the noble and beneficial promise of AI in the SOC and what can possibly turn to the other side.

Yeah, I mean, the promise of it is what some CROWD customers are experiencing. And what they're experiencing is they have thousands and thousands of alerts that they get. And many of those alerts are actually well understood. What to do with those alerts is well understood. And in those kinds of scenarios, despite the dynamism of the data that they may have in their environment or the chaining that's required in order to investigate something beyond just basic enrichment, that opportunity exists to actually resolve those thousands and thousands of alerts. And the ability to do that across multiple data lakes, because you now have this ability to connect through APIs through agentic systems or through more traditional systems, or even SOAR systems, to go in and connect up to these different things, and then chain the investigative plan together, and then render the result out. So I think that's a clear opportunity. We have customers who are taking advantage of that, and that is a really good thing. And for our customers, they're able to eliminate a lot of like boring stuff off of their plate, so they can start to look at things.

Mm-hmm.

So that one message that said, you know, open port on EC2 instance that you see 70,000 times a week, You still go in and check for every one, but a human's not doing that. A machine is doing that, and the human only sees that one when it's important to see. Otherwise, we get desensitized, and we stop paying attention to that kind of an alert. So that's the positive outcome of what's happening.

Okay, that's the Dr. Jekyll, who is— listeners, if you're not familiar, Dr. Jekyll is the good one. What about the Mr. Hyde, who is the bad one side of this? What gets messy, and where is the AI Not yet able to help us.

Can I stick my snout in there for a moment?

Please.

Because I always have this quintessential view that the bad guy is the boss's boss's boss who thinks that in 2 years, all the humans would be gone from SOC, from security, and frankly, from IT as well. Like, it's this— somebody asked me that at our last RSA, RSA 25, if you're listening next year, and he was like, What would be your reaction if you can't use curse words and swear words? What do you respond if they say, well, our CEO thinks in 2-3 years we'll have an IT team of zero? Like IT team, not SOC, not security, but IT team.

Surely nobody thinks that.

Apparently there are people who think that. And this was like a trusted friend from Deloitte said that he met an actual CEO who assumed that. So I always think that maybe that's the Mr. Hyde.

Is that the Mr. Hyde or is there something else here, Monzi? What is Anton crazy?

Okay, fine. Let's see.

I think Anton's crazy.

Probably.

I mean, Anton, we've known each other long enough, likes to draw big barbell kind of analogies to draw the point out, which is actually very instructive.

Barbell.

And so I think there's a couple of things. One, it's the security industry saying, buy my product, because they need to have a way to satisfy the buyer. And the buyer generally gets satisfied very easily when there is a dollar amount attached to it. And the buyer is struggling, right? Or the leader— security leaders are generally struggling today because they're generally convinced that there are not enough security people to go around. That's a whole other point that we can argue. I think there's plenty of security people to go around. There's not enough security unicorns to go around who can talk PCI and email headers and compliance and know how to write queries on 17 different platforms that you just decided to procure and memorize all the schemas at the same time. That's the shortage. It's not the individual practitioner shortage. So now we go to the point like, what's the hide here? One hide is actually really, to some degree, the industry, the security community, the industry, the manufacturers of products, because the manufacturer products are having the buyers believe that there is this super awesome thing that can just simply be acquired and that is going to give them mass benefits. And we are, you know, and so therefore you can just do this and you can project 3 years out this kind of a cost savings and this kind of a flatline or reduction in workforce, and you're going to make money, you're going to be more profitable, it's going to be awesome. So I think that's one hide. The other hide is the basic fundamental sort of a gap in understanding of the technology that we're trying to use to solve this problem. And in this case, I would say, let's be maybe a little bit more precise and just say the investigative or threat hunting problem for SOC teams. Let's just look at that problem. So what's the challenge of using a generative component to that? And I would say that the hide is in the name. It's the generative part of generative AI. Because what we desire as security practitioners is, in addition to speed, we desire consistency and we desire depth in what we're trying to do. But to get consistency from a system that is fundamentally generative requires a lot of effort. And so that's part—

Consistency is overrated, though. Is it?

Oh, it's— yes.

Okay.

It could be. We put out a show every Monday without fail.

I'm just checking. I'm checking. Yes.

You take a pulse. Take a pulse, right? Yes. Consistency is overrated to a degree because at some point, because of the policy structures and the regulatory environment that most security practitioners live in, at some point or another, they have to demonstrate to a compliance officer, whether that's an external person or an internal person, or even a manager for that matter, to say, well, I did it the best way possible, or I did it the way that I was supposed to do it, or I did it following a best practice. And so by that, you can check off your adversary model of auditor, manager, or actual adversary, or a regulator, right? And so that's where the consistency piece comes into play. It is very foundational. Consistency is also important because if you're going to analyze a piece of data and you need to go to the same place for the same thing, to get your answers, you should be going to get your answer in the right place. So that's part of the challenge. So what's happening? So going back to the height piece, what's happened is we observe many people in really well-resourced organizations that we have talked to who say, well, we're trying to build this capability in-house. We're trying to build an AI SOC agent in-house because we believe that we can just do it. What?

No. Is this like— let me express my shock briefly. Shock.

Yeah, yeah, exactly. So they're trying to do this, right? And so we asked them, okay, great, like, so you're trying to do this, but you're still talking to us. Why are you talking to us? And they said, well, it's actually not working. And okay, well, what's not working? And so we sort of dissolve all the what's not working problems, and we can dissect each one of these individually, but the responses are in one of these 3 classes. They're either in some form of a data class, where people don't know where the data is or how to deal with the data. They're in some sort of a process class where, well, we kind of don't really know how we respond to alerts in this organization, so we can't model the security analyst. Or they're in the governance class. Like, we don't really know what data we should or should not have access to and how the user, the security analyst, should they really have access to HR data because now we can. And so, how do we manage that? And lastly, on the governance side, am I okay sending all of my alerts, all of my investigations, and all of the results of those alerts and investigations to a third party in the sky, to a SaaS service, where someday some auditor might say, hey, you knew about that threat in your environment, or you knew about that compliance violation in your environment, and you didn't do anything about it?

Yep.

That's a classic.

And so, those are the 3 things. The hide really is that this is not— it's maybe rediscovery, maybe faster this go-around, that this is a complex thing to resolve. And it's not just a tooling problem. We used to talk about people, process, and products.

I think that sounds like, if I try to summarize this, there is this concept or a model or something, maybe a framework of an AI-ready SOC or AI agents-ready SOC. So, is there a short list of things that you should have, or maybe types of work you should do to have your SOC be AI-ready? And a side question to this, is AI helpful for that work?

That's a great follow-up.

Yeah. I mean, I think there should be a concept of an AI-ready SOC. I'll tell you what we share with our prospects and with our customers. The first one is that they have to at least have some sense of the inventory of where their data is. You may be surprised to hear this, but a lot of organizations actually don't know where the data is.

Yeah, but this is basically just good SOC. I don't think there's anything AI-ready about it. It's just good SOC versus bad SOC, right?

Yeah, but the AI-ready piece about that is that you should have at least some sense of where is the data going to come from that you need to get better. You don't have to be there today. But for some organizations, for example, they may be using all 3 of the major cloud vendors simultaneously, and many people are. In the Fortune 1000, at least many people are. So you can't just say, yeah, I bring all my data into one place, because you think you are doing that, but you're really not doing that. And so that is a realization. The second piece is you want to have some sense of the fact that you have done things in the past, and there is some degree of quality to that prior work, and there's something about that prior work to be learned from. Yeah, I get it. Some analysts might not have enough time, so they take shortcuts. Others write really, really long posts that maybe sometimes don't even matter. Generally speaking, analysts are doing work to move their organization forward. There has to be a way to learn from that history and to train some set of actions on top of it. And then the other piece too is that you have to— this is really a mindset thing more than a technical thing. You have to be okay with the machine saying that that particular alert just needs to be closed and there is nothing to see here because you're doing that already. Without any investigation. I've seen people close literally 100+ tickets in one swing without even a—

For good reasons or for bad? I think that closing to me is not a sign of anything unless you know that they're closing the right tickets or the wrong tickets, right?

And I'm even going one step farther on the crazy scale saying, I'm not making any quality attribution to that. I'm just telling you that that is what's happening. And so, somebody needs to at least make some kind of a quality attribution to say, you know what? And recognize that we are doing this today. And I'm not going to beat my AI SOC vendor. I mean, of course, I'm biased towards Scroggle, but to the rest of the community who's building products in this space, to constantly try to stump the chump who's trying to give you something a little bit better than zero. And a lot of people are at zero right now. They're at absolute zero right now because they're closing without any inspection.

So that's the thing that's been bothering me for a long time is how do we help management teams get okay with the machine being wrong? Because right now humans are wrong all the time and we accept that. We think the machines will be better than humans on average, but they'll be wrong sometimes. So how do we get comfortable with the machine being wrong sometimes?

Yeah, so I'm going to beg the question for a moment and I'll respond to the question. I think we must agree that the machine has to be wrong some of the time. If somebody walks in a machine that is never wrong in this context, we should pitch that machine. It's too overfitted. And so, it has to have it or it's not going to solve the problems for you.

This is really good, by the way. We should just pause for a moment and just basically focus there. Machine must be wrong sometimes because if the machine is never wrong, you are in the same bucket of overpivoting to reducing false positives and missing many of the attacks. So, this standard must apply to bots and humans. So, it's kind of important.

Yes. It's very important.

That pays for the ticket for the show right there.

That pays for the ticket. Thank you.

Awesome.

So not only do we have to accept that the machine is wrong sometimes, the machine must be wrong sometimes.

Yes, it has to be part of it. You have to be able to honestly ask the question of your vendor to say, show me where you're going to make a mistake. And the vendor should be able to be— should be very transparent and say, under these circumstances, I will make a mistake. But guess what? I'm okay with it. And I invite you to be okay with it. And this is the way I made it transparent so that you can actually go and inspect it. and make up your own mind. And this is what we do as a mitigating circumstance when we know that under certain circumstances we will be incorrect. But I think transparency is sort of the core mechanism in how we do this and giving people more access to this set of information. And I wanna pull on that thread to Anton's point about the threats that you just mentioned. Most customers that we're talking to tell us they expect about a 1/3 increase in the number of alerts that they get. Yeah, 130% per year.

So every year is 30% higher than the previous one?

Yeah. And so the question is, well, wait a minute, aren't you doing all this other stuff like doing better detection engineering, reducing your alerts and filtering and all this other stuff? And they say, yeah, we're doing all of that. But the core reason why we expect that increase is our footprints are increasing. The number of customers that we have are increasing. The kinds of tools that we're using are increasing. Before, a particular business user was able to do, let's say, 1 unit of work work. Today, they're doing 3, 4, 5x unit of work because they're using lots of tools that are now AI-capable or automated in other ways or more intelligent. And so therefore, the footprint is bigger. And we are seeing a lot of kinds of alerts that now we need to pay attention to that we didn't pay attention to before. And so when you look at the technological footprint, like what happened when we went from on-prem to cloud, and what happened when we went to cloud to like hypermobility, and then from there now in the AI space, and then we start seeing the kinds of technology that are getting deployed, right? Everything from people trying to implement guardrails on their AI deployments to developers using agentic frameworks, which utilize transitive trust that we're having a hard time inspecting. And so, there's just a whole bunch of these additional things. Like, who imagined in 2001, S3 bucket exposed to public internet alert? Nobody did. It wasn't there.

Yeah.

So, there's a whole bunch of those things that are gonna happen now, right? So, anyway, so I think that's the very long-winded way to answer your question, you know, that leaders are going to have to get accustomed to the fact that the number of alerts are going to increase. Machines are going to make mistakes, but they are going to create the opportunity for transparency and inspection.

I really like that answer.

So, we now have an actually important question, by the way. This is the question we probably—

Oh, now it's important.

You already paid for the show.

The previous ones were not important, listeners.

Newsflash. We now have an important question to ask you. Everything else was kind of like, I guess, to generate hilarity. The point is that I am still nervous when people say adopt AI for this reason, that reason, because it is not always clear what gets better and at what time. So, people have this intuitive view that they would save on labor, or maybe the big bosses think, or save money, or improve quality, or improve coverage. But you deal with this every day. So how do we measure progress towards AI SOC? Specifically, what gets better and how do we know it did?

We measure it on 3 vectors, and there's a 4th vector that I leave for the end that we actually don't necessarily like to measure.

Oh.

Because it gets super fuzzy. So we'll talk about that one at the end. So the first one is the speed of how long does it take for an alert to happen to the time when we tell a human that this is worth paying attention to? So that first stage of contextualization and enrichment, and in Krogel's case, an investigation that is kill chain driven to say, I see kill chain tactic activity across this many phases for this particular alert, so therefore a human should see it. So that's the speed component of it. The second place that we measure is consistency, and that's mostly a ratio than anything else. So it's a little squishy, but it's consistency in that when you get an alert for, let's say, malware detected on a host at 1 o'clock in the morning when there's nobody there in your organization, or your MSP or somebody else is paying attention to it, that alert is going to get investigated the same way as when that alert happens at 4 o'clock in the afternoon on a Tuesday when your A-team is sitting on the bench working the problem. And so at least some minimal baseline of consistency of what is going to get checked and how it's going to get checked and some process is going to get followed. So that is the second piece that is really, really important. And the third piece, which is somewhat of a derivative of the consistency, is depth. Did the alert get analyzed across some set of things? If it was a malware detection, did you look for initial access? Maybe you should have checked USB insertions, or maybe you should check for email. Or did you check for exfil? Because maybe there was communication, right? So maybe you didn't do like a super, super deep investigation, but you at least checked some parts of this thing, and you did it more than what an analyst would do when they're in a rush, where they just look at the alert and go, oh, there's probably nothing, and just close it, right? So, those are the 3 things.

I have a one-side question before we go to closing. You mentioned speed first. And I would do the same, but I would always caveat it to if you just focus on speed, you're just gonna have a script closing all alerts, and it will be really fast. It can close millions of alerts in a second, probably. So, if you just measure speed, you're gonna fail.

Agreed.

So, how do you stop people from doing that, from rushing to this, oh, yeah, it's gonna go make it faster? And I'm like, Like, hold on, it's not AI. I can make it even go even faster. Delete the whole thing.

Yes.

It's very fast.

Yes. So, that's a mental model that we have to use. I mean, that's why it's a trident, right? That's why it's a 3-legged stool. We use whatever analogy you want to use. It doesn't stand— it's not a unary. It doesn't stand on a single parameter. Your question was around value. How do you show some tangible benefit of doing this? And the tangible benefit is not a single parameter outcome. It is this parameter of speed, And depth and consistency. And that's when you can say, okay, I'm really doing something meaningful. Because you can make that same argument, Anton, for all 3 of these. If you were super consistent and just stupid consistent, then it doesn't matter. If you were really, really fast on its own, it doesn't— actually, even if you had 2 of those together, it doesn't matter. You could be super stupid consistent and super fast, and it's no good because you're not deep. Right. And then you can play this game with all 3 combinations. And so it has to be that triad.

Well, I really appreciate that answer, and I appreciate the thoughtful pushback on Anton, and I couldn't agree more. I hate to do this part. This is my least favorite part of the show, but I have to ask our traditional closing questions. And so, one, could you give us a recommended reading, something that you think would inspire listeners or encourage them to think more about why it's actually valuable to have a fraud loss rate greater than zero and SOC error rate greater than zero? And then 2, do you have one tip to help people get started with their, you know, AI adoption in SOC, or maybe a tip to improve their efficiency?

So I will, uh, I will give you 2 on the things to read. These might be sort of adjacent, but 2 books that changed my life, which is why I embarked on this whole journey and left my job as an executive. The first one was Nassim Taleb's Antifragile.

Nice.

Which to me was essentially the solution to Kai-Fu Lee's AI Superpowers. When I read AI Superpowers, it blew my brains out because it was— as a person who served in the government agencies for, you know, within the weapons complex for a dozen years, I read that and it just— I really got angry. I was like, you know, we have a major problem. And then I read Antifragile. I was like, well, how do we get over this? And I didn't, didn't have that intentionality, but the outcome after reading the book was, well, here's an approach that you treat things that are challenging as resources rather than as detriments, which is why I am now on this thing is like alerts are actually signals. You should utilize them. More alerts are good. You should not filter too early, right? So that's where some of these kinds of ideas are sort of coming from. The next thing that I would say is, like, where— to answer your second question, like, what can people do to get started in this space? The security community has done really well when we've dived deep into something and not assumed that whatever magic somebody was selling us was useful or not useful. This is the same. There is no shortcut.

Yeah.

We have to get better. I mean, I would say start at the basics. This is maybe another one. Go read the Google Transformers paper, Attention Is All You Need. Read that paper. To me, it's analogous to reading Aleph-1 back in the day, Smashing the Stack for Fun and Profit.

Mm-hmm.

That was our foundational paper for many, many years in this community. This one from the Attention Is All You Need paper is our next foundational paper leading us into the future. It's very important. And because it'll just eliminate a lot of kind of stump the chump kinds of things like, well, what if it hallucinates? It's like, well, let's talk about that. And so it's those kinds of— so yeah, so I actually gave you 3 books instead of something otherwise, you know, something wise. But yeah, those 3.

That's okay. We'll take all 3. They're all great recommendations.

Yes.

And then do you have a tip? We love a tip too, other than read those 3 things.

Yeah. I mean, I think the biggest tip that I have for anybody in this space right now is just pay attention to what's going on. The space is moving really, really fast. And what I would suggest is a lot of people can do a lot of vibecoding now. Things are very accessible. The tactical tip I would give somebody is grab a document, any document, grab a CISA advisory, grab, grab anything, and just run an embedding model through it. And then just be able to query that document for things that you care about. Just do that. That'll take you maybe a few hours. There's a lot of resources to help you move really fast in that, but just that will open up a whole nother world. This stuff is actually not that mysterious. This community has worked with far more crazy things. We just need to give ourselves the opportunity to learn.

Well, Manasi, I can't thank you enough for the thoughtful answers blowing our mind about why zero errors is a bad thing. This has been a really fun conversation, and I just want to thank you for joining us today.

Yeah, thanks, Tim. Thanks, Alexander. Good to see you guys.

And now we are out of time. Thank you very much for listening, and of course, for subscribing. Please subscribe so you can get new episodes piping hot. And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcast. You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us, and if we like or hate what we hear, we can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast. By Google.