EP261 No More Aspiration: Scaling a Modern SOC with Real AI Agents

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here as ever, myself, Tim Peacock, Senior PM on Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want delivered to you piping hot every Monday, please do hit that subscribe button. If you're following on YouTube, what do they say? Like and subscribe, ring the bell, all that stuff. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on the Google Cloud Security Community page. Anton, we have a fun episode today about agentic SOC and really getting into the weeds of how to make it work and how to manage your costs. What else do we talk about here?

I mean, people, I think of this episode as kind of the opposite of rambling marketing about agentic SOC and how AI would save you and how it beautifully replaces all the level 1s, 2s, and 3s. No, this is like none of that.

None of that.

It's actually how people did the work in a SOC with AI and agents all the way down, how they run it, how they manage it, how they watch the costs. Watching the costs comes up for people who actually do the work. But people who read the glossaries, they only see high level, oh yeah, you'll save money, you'll fire the people. Not the reality. But what the reality is, nobody knows. And this episode reveals some of that, actually much of that.

Much of that. And I thought it was very interesting, the specific ROI that he was able to point to and say, listen, this is how much it's costing us, this is how much it's performing. I was really impressed with that part in particular. So maybe listeners, without any further ado, Let's turn things over to today's guest. Today we're joined by Dennis Chao, Director of Detection Engineering at UKG. Dennis, thank you so much for joining us today. This is— listeners, if we seem a little rusty, it's our first recording that we're actually doing in 2026. So let's hope we remember how to podcast. Dennis, we ended last season, which seems a long time ago now, talking about the AI apocalypse, the idea that we're living in this post-AI world where LLMs are out there autonomously, agentically hacking companies. Do you think that AI-powered attacks are here as far as what you're seeing? And if so, or if they're not here yet but you think they're coming, what's your plan to sort of live in that world and be ready to defend effectively against that? Is it better patching, better DNR? What's the plan for that?

I think AI-powered, AI-generated attacks are already here. So AI reinfused the agent side, so retries, Scale, things of that nature, generated payloads. I don't think AI— there was an instance in DEF CON where we had AI exploitation and they got the first set of attempts there. I don't think we're at the research level yet. I do think we— it gives trials and errors just like everything else, but I think— do you think we're at generative AI? Maybe to do patching in response to some of that. It's really about more of the suggested detection side and testing. So if you think about detectionless code, you really need to think about, do you have your pipelines, unit tests, integration tests? And so if you can do an AI agent that generates some of that and does automatic testing for you, you're already ahead of the game too. Now, it does increase costs and latency and stuff like that. So I think that's where we're at in the industry today. I don't think anyone's really done a super zero-day based on an agentic solution that I've seen so far, unless you guys have seen it.

I haven't seen that, but I wanna ask, Maybe a more concrete question, which Anton always encourages me to do, which is your team, I've heard, is now using a hybrid agent workflow. What does that mean? Because that's what I've heard, but I want to know what that means and how that's working for you.

By the way, before you answer this, I want to stick my nose in there for a second. I've noticed recently that when anybody asks somebody else about agents, it's usually polite to say, and how you define agents. By this I mean, and the reason why I want to add this is I've met people who think that LLM with tools is an agent. And that's all they have. And I am a little nervous when I hear this definition. So perhaps define agent before you answer Tim's question.

Sure. So our agent definition is an agentic system, mostly. So we have multiple agents doing something together, reasoning, utilizing tools. Now, those tools could be wrapped and used for something. For us, it means synthesizing what a human can do using those tools, whether you're MCP server-based or you're using just a standard API Gateway wrapper or Cloud Functions to do some of those things. Those are what we consider as agents. Whether they do a good job or not depends on a lot of evals too.

Tell us about your evals in a sec, but go on. I want to hear about your evals later.

Yeah, sure. I guess back to the hybrid agent side, we started off way before ADK was even a concept or even published out there. We're doing C++ like everyone else. Put an agent attempt flow, deterministic code, add LLM as a function, call tools, let it come back out. That's your request to the agent. We moved over to full ADK where we went full on letting it decide on its own fate and its own factor, and our costs sort of spiraled out of control. Our context windows kept blowing up. It was kind of crazy. Of course, we were in the early days of ADK as well, some of the earlier versions of that. Now that we've kind of updated ourselves to this and we have additional there's now premium post-hook, you know, support, things of that nature in the ADK. We moved to what we call hybrid agents. So I coined that term from Ana and LAL on our team that's done that. So Ana Marines, LAL Garcia, they kind of went back to the drawing board and went, okay, what do we do about blowing this up? How do we get our cost controls down? How do we actually get it into not spinning in an infinite loop on itself, going into a 4-hour investigation of the SOC? In our case, we want it down to about 5 to 15 minutes, which is our sweet spot. So we went back down and said, let's do something that's best, which is control the master flow. So the master flow being, for us, being a 7-stage pipeline step. We have a ticket operator, a decision agent, we have conditional branching using deterministic code, and we use hybrid in terms of using its own reasoning processes to create new instructions to itself. And then of course, using ADK's Think and Reflect and Retry plugin to go ahead and use some of those semi-deterministic approaches there. We also let it go down its own rabbit path of utilizing what we call agents as tools, just like using the ADK. And so we say one function of the SOC, you need to go down and drill into the investigation piece, endpoints, networks. Do you pivot besides using the same indicator? Those are things that we let it decide on its own and then bring it right back out to the master drawing board or a blackboard type of approach, shared memory there. From there, we actually go back into the Conditional model, you know, did it try and fail? How many failures would it go? So we have semi-deterministic items going on there. We don't use agents as those loops for those purposes because it kind of spirals out of control.

Mm.

We do have another triage loop by itself as one last item. And then for a lot of our evals, we use LM as a judge. And before it gets back to the investigation process and back, we have a final disposition agent finally making that last call. We had it against a set of post-calls. Almost callback to the ADK in particular. We do basic hallucinations and perplexity judgments on that, which adds to our cost too, but it actually creates a really nice disposition. And so for us, the highest level of precision for an analyst equivalent or heavy augmented analyst for automation is the final disposition. Did you make the right call for benign positive, false positive, true positive, and why? Those are our 2 major issues that we look at for damage control and— issues around that. So when we do our sampling, we use basic z-score model, and then we bring that back and go, okay, of 100 use cases, you might say for 95% confidence level, can you say, well, let's just, just sample like 30% of them, rough estimate. And then that's— that actually adds a lot to our cost. We saw our original use case of it just doing a really decent job without the evals at around $4,000, $5,000 a month. Now we're closer to $10,000 a month, but had 120K analysts looking at 10 alerts per hour, 24/7. And when we looked at it from the last week, you know, granted it's still seasonal holiday time, 427 alerts within that period that we sampled versus a human analyst doing the same thing. We still have 2 different SIEMs as we're migrating right now. And what we found was that this is still 4.5 times faster at a 1-person analyst doing only level 1, somewhat level 2 work, full investigation, and that's for a cost of 1 analyst versus having like 10. So we're working through GRC right now to see if they'll allow us to do eyes on glass, knock on wood, because that 24/7 statement. But yeah, that's what we have so far. We actually deployed this 2 weeks ago, actually.

Okay, so a ton of follow-ups out of that, but I think Anton gets first bite at that apple.

Yeah, I kind of wanted to, before I follow up and ask about the use cases, I wanted to basically reconfirm your answer to hybrid agent workflow means it has agentic, aka non-deterministic, and traditional coding if-then-else workflow. Like, hybrid means a bunch of LLM, a bunch of stuff is LLM, a bunch of stuff is deterministic. Is that fair?

Yeah, that's fair. I would say the most important part that we found that controls cost and quality for us in a more easy-to-guess fashion from unit testing is having the deterministic part on the outer shell of the master control flow.

That master control deterministic outer shell makes a ton of sense, especially if you think about, Anton, some of the stuff that Heather was saying during her last appearance on the show about the weird directions that the LLM will go down when you don't constrain it. We had Heather Adkins come on last year and she was talking about how in one case they tried to get an LLM to red team and it ended up researching CPU vulnerabilities instead of using obvious stuff.

Right.

So controlling the What's the right term for this? Like, harebrained schemes of the LLM seems to be important.

Yeah. I kind of want to, in light of that, what I want to ask is kind of more of a use case catalog because a lot of people, when they imagine AI and SOC, as you recall, I kind of make jokes about how I don't like AI SOC, but I like AI in my SOC. The use cases that people see is just obsessed to, they go to triage right away. They think, oh yeah, the machine would get all this pile of stuff flowing through the pipe, and then you crank a handle and it gives you good data. But apart from that use case, what are your AI in a SOC use cases?

Sure. So right now, we just got the triage and investigation portion done, and it creates an escalation to our SOC so far to like the level 2 queue. Now, what we have down in the pipeline, we've already got the containment portion of it. We even had it before 80K. that we want for anything that has a true positive, high fidelity, high confidence interval, that we want it to say, hey, high confidence means this. We have the last known true positives, at least in the last 30 days or so. And we've limited it to other things, including hard guardrails for do not contain a server in production, do not contain your VP of Ops or something like that. And so we have some of those things that are en route and planned. Now, those are extremely limited. So if you look at other, like, solutions commercially like a 7th AI or something, they are not going to want to do that. Or if they do do it, they do it on very specific use cases. We plan to do the same thing in this case. Now, what's a little bit different for us, we also have payload extraction. So as the student's reasoning at some of the sub-workloads, we've actually pulled PCAPs from our NDR, and we've actually pulled it through and let it explode and explore itself. We also have an MCP server that we had to really struggle to get Keep it running. It doesn't do the best job as a full malware analysis, but we have some of that going on. So we wanted to actually do detonation and deobfuscation. And then we need to figure out— we haven't really determined what we want automated for this containment part first. Contain first and then send it back to the analyst, or send it to the analyst and hope that they get back online fast enough to actually do something about it. For instance, let's say— let's case a highly reliable, like ransomware So lots of I/O going on with this, a bunch of entropy being measured. What do you do from this point forward when the analyst is like trying to get back to their pager duty? So we're making those decisions right now. I guess the last use case that we have, and we had it for the end of the year, and we— this is going to be determined based on our cost control— is the hunting side. So we want to take the same model and pattern that we want to do here and then move it to the threat and formal side. So we're going to take a bucket or SharePoint, whatever we decide to do. Let it ingest, determine the hypotheses there. We're going to use it loosely based on Mandiant's, you know, workflows, and then let it go through at least 3 different iterations or variations of a total TTP attack map. And then we'll have it do a tandem item. So we'll create the attack map. The second one, we plan on trying to bring enumeration and context to it. So we have to have a very targeted one. So we want to target something in production in our submodule, this type of product that we have. In our case, we're a SaaS company, and we're going to target this specific set of items. We wanted to enumerate first and bring it back in for that context management, and then of course do multiple compressions and removing extra characters that it doesn't need for its context window. And then we slap on the— I want to slap on— we use Arise, slap on the instrument, the calls, to look for hallucination patterns that we know of, and then we move forward with that. Now, how do we control that cost and variations of that? Today, we're not quite sure yet, to be honest with you. But we do know it's gonna model based on what we do and optimize on the SOC AI agent traditional analyst case. And I guess the third part, if you include DE as part of SOC, we plan on doing an agent that has uses Git runners as well.

Sorry, you include what as part of SOC?

Detection engineering. Sorry.

DE. Oh, DE. Of course.

Yeah. Yeah.

There's a usual debate about is it a peer org or a part of SOC? Yeah. That makes sense. Correct.

And so We're debating on— we know it's easy enough, in quotes, to create indicator-based detections, but we don't know how it will do with TTP-based ones that will stand the test of time. Can it create the right statistics without hallucinating some random thing? Does it know how to read the data and truly understand what we have in a lake when we give it to it and let it test overnight using Git Runner? That's kind of what we have in our planned maturity pipeline.

That's a really interesting roadmap you're laying out there. Maybe take a step back and uplevel this a little bit. So for your overall SOC, what are the metrics you're reporting up to your leadership? And then how do you think that the agent and all the agentic work you're doing is going to shift those metrics?

Sure. So we have 2 different types, just like everyone else does. We have the one that the fancy drawings for senior leadership that they want to see. The return on investment obviously is like, hey, how much produce or throughput are we getting from this, you know, AI agent today? And generally we were seeing a 4-to-1 cost ratio. So that's It's not bad, but we could do better. And then of course, we also see the meantime to detect in terms of not just developing a detection later on, but in the meantime to triage, contain, and basically close the entire ticket in response. We're measuring against that which is AI-labeled and which ones aren't. Now we are fortunate to have 2 SIEMs today that we are funneling 2 different alerts to and same use cases as reporting and migrating them to. So we have the benefit of seeing the analyst versus the AI agent. Now, on the sampling that we are doing, we were trying to figure out how to create the— no one knows how to really— well, everyone tells you how to do it. No one gives you a free pattern to do it and a free code set to do it. So the reinforcement learning and human feedback, how do you automate that? So for now, it seems kind of terrible is that we'll still mark yes or no on labels that we have for our case management system. Now, we don't have the distinct issues around that evaluation. It's like, Which error, which degradation of functionality? And because we have a multi-agent system, roughly about 10 active, potentially up to 20 in use depending on what kind of rabbit holes it goes to, you have to have the evaluations across—

So it does go into rabbit holes, right? Right, yeah.

We're going to use it on every agent, and the agent interacts itself, and it gets kind of crazy. So we're not going to tell management about that for now. But we are going to show them that, yes, based on our rough guesstimate, we're seeing like for every 10 alerts and across a spread of good use cases, that it is getting about 80% accuracy or 70% accuracy. So really, we're looking at the standard, we're teaching them the F1 score, what it really means. We're teaching them some, a bit about perplexity, about just like analysts get confused, there's model confusion itself. And then we do give them the caveat, LLM as a judge is one of the few ways we can get to it in a practical mean while we're instrumenting it. Oh, and by the way, the costs are gonna keep going up because we have to monitoring, it's runtime. So that's what we're reporting upwards. On the backend of that, we have dynamic and static-based evals, static being something always changes. Hey, it's going to go above 10,000 a month or something, right? Or it's going to be longer than a 20-minute total runtime for its entire session management. So those are things. Those are the other dynamic ones are about more about the prompt window sizes and the context around those. So when we talk about hallucinations, we have fixed patterns. We also try to include feedback from the analysts in terms of, we have a QC/QA review, and we actually have comments notes in our case management. We will take those out, let LLM transform them and wash them into LLM-type friendly behavior, and we'll re-inject them back into the hallucinations eval and these other evals around there to see if we see noticeable differences. For instance, there is a pretty good case where you see increased latency, usually Results in increased hallucinations and tool order malfunction calls. And for anyone listening, our data scientists have said about 50% of our context window utilization, even in the frontier models, have seen huge hallucination increases and spikes. So that's what we're using on our end. And then we're instrumenting with Arise. We are using Vertex as well, what you guys call trajectory-based items. We're using our own framework. And I want to remind everyone, you can't just use one framework. framework, precision means something different and recall means something totally different in every single agent and every single agent's interaction with the parent agent in the entire system. Your evals get kind of crazy, so make sure you focus on the inputs and the outputs, at least at the very edges of your use case.

I like that. A lot of depth there for listeners to digest. I think that the cost thing you keep bringing up is so interesting to me. How are you talking to your leadership about that, and how are you thinking about that changing over time as the cost for frontier models continues to come down?

Well, we give them the idea that the frontier models come down not because they're less complex and they offer just like the fast version versus the reasoning version of it. We also talk about in terms of efficiency and token utilization. Can you use less shots in your retry attempts? Those are part of it too. We're mostly looking at also how we do compression right now. There's the different versions that everyone uses, but they all have a variation of using another ML model or an LLM to do that. So that adds back to the cost. So we go back into what we think is traditional compression, which is how many whitespaces can we take away? How many extras can we take away and trim that? For that, we can just regular expression that pipeline and then reduce our cost right there. We're also looking at in terms of telling our leadership, look, even though you may reduce costs in the operational runtime for just using the analysis, You still have security monitoring that I think everyone is really weak at still. So we might talk about ModelGuard, and then Meta has their Purple Llama, right? And you look, but these are all focused on OWASP 10 LLMs and injections, but that's not enough, right? So there's always gonna be a way around there. And so I think what leadership's not focused on right now in terms of the larger industry is how do you measure your runtime and how much do you sample of each size, just like a probe for Close before you— it's just not worth the return on investment. So we're telling leadership right now, here's our preventive controls, here's the hard guardrails, saying cause an exception handler and make it kill itself when it needs to. We have a kill switch as well, by the way, for SOC analysts. So we use a stored playbook to let it kill it. But also we tell them, look, for what we plan to have the cost maintain where it is, roughly at the cost of— we hope to try to get down to one analyst for a factor of 10. But in that 1 ounce, we're also going to— even though we're going to increase our cost over operations, never completely reduce it to like $40K a year or something. In exchange for that, we have multiple ways of monitoring, including the actual famous word guardrails that everyone keeps saying. And we can finally say, based on the security use case product, you can't just slap model guard on there. There's policies, there's latency involved in there, and there's false positives that we have less tuning capabilities of. why don't we focus on having the monitoring portions of it and having those outliers become the SOAR automated playbooks to decide whether or not the AI agent system should continue running or skip this batch and go to the next one? Those are what we're working on today to fill in that gap. We're going to stay roughly about a 6-figure analyst, in our opinion, but we're promising more value-add in the backend for operational monitoring and due diligence. We hope to bring that pattern back to our SREs to say, hey, you can also do this too. It's not just a security thing, it's performance at the same time.

By the way, the cost bit is interesting. Sorry for interrupting, Tim. No, no, you're right. It's just like people who show up with, like, absolutely naive, I'll just deal with some AI SOC vendor, they sort of automatically, completely brainlessly assume that it's cheaper than what they have. But done wrong, it's really not. And it's just like, I've seen people who come with examples of it would be 10% worse and 5% cheaper, or it would be 30% worse and 10% cheaper. And I'm like, wait a second, that's not saving money. That's— or occasionally 10% worse and 10% more expensive. And I'm like, wait, what? Why? I mean, you need to watch the costs for some of the things. I mean, ultimately, here at Google, maybe we are not as obsessed about some of the stuff. But we are still obsessed a little bit. Tim is making the PM face. Yeah.

Yeah. I'm gonna smack Anton with the PM stick in a second here.

Yeah.

I hate to do this because we're having so much fun. Anton, I think we have time for maybe one more question before we close. Do you wanna pick the final question, Anton?

Yeah. I wanna hit the one about the use cases and rule writing because you mentioned, of course, detection engineering hates in your title. And again, I've met some people who show up and think that We can just shovel threat intel into some kind of big funnel, crank an LLM handle, and beautiful rules that are performant, no false positives, no false negatives, just beautiful come out. I mean, that's obviously completely insane. But what is the reality of DE using AI? And what's the reality of the next couple of years, if you can predict?

You know, I think right now, at least with the LLMs that are today, we're just going to use it for suggestive. We're going to still merge our PRs. We're going to look at our testing. We hope that you have a BA solution to help you with your integrated testing, but we're not going to hope for more than decent indicators and decent indicator-driven approaches with a human still approving those and pushing those to prod. We have no idea if it's truly testing the unit tests in terms of agentically, unless you're letting it get results directly from a pipeline, which we hope that you're doing, and then retrying those overnight. But we don't know how good that's ever going to be. The only thing we can see is that it's probably going to use its existing dataset. It might use grounding or search for something that it sees online. We don't know how it's going to pull that back and rank it file. Like, something from VX Underground or something from Malware Unicorn is something different than Joe Blow on Reddit. So that's something to keep in mind too. Our data scientist, Austin Nowak, also says that the evaluations are very different from TE over SOC use cases as well. So our evaluations, although they may seem simple, We have to evaluate more performance-based items. So how do we evaluate the runtime and instrument that? And so when we look at use cases and how we inject it, what does integration even look like? Does it match a TTP in there? Doesn't— do you want it to not match an existing TTP based on coverage because it's net new on, you know, some random zero-day that you see from the news, but you have no additional details about the exploit research and what POC looks like? Those are things that scare me. I think for the next 2 years, we're going to be still handholding that a little bit. But in the next 2 years, I see a lot of excitement where maybe someone's going to have a breakthrough in a research base, especially with these new models like Gemini 3, Claude Opus, and stuff like that. And so if they can get the research portion of it down, learn to prove things, and then build its own POC, we can take that back to DE and do that. Now, I also see A2A, the protocol that you guys made, or Google specifically made it. And I finally can see that there's going to be more examples of that. Because right now everyone's trying to create the context manager window override and compress, compress, compress, and then create new sessions to kind of get around the 1 million token limit. I think in the future we're going to see more micro-ATWs going on with single-purpose AI agents and a simpler ATW talking together. And so it's easier to say, just extract this one key-value pair of this JSON payload or whatever the protocol may call for. And just give me that as my context window instead of trying to do multiple iterations in that same connection session. So that, that's my take on it.

Well, this has been a super deep dive in a lot of ways, and I hope our listeners were listening at 1x speed because this would have been a tough one to keep up with at a higher speed than that. I have to do my least favorite part of the show, Dennis, which is ask you our traditional closing questions. One, do you have a tip to help people on their agentic SOC adoption journey? And 2, Do you have recommended reading? And that reading could be anything, just not Anton's blog.

Not even a single post. Sorry. Yes, I know.

I'm—

we are blocklisted myself because—

Definitely Anton's blog about the, the scene in the state of STEM. But honestly, so at the architectural level, Chip Wynn's books for the AI engineering side, it's probably a good starter pack for everyone. You guys in GCP land have really done a really good job with Skill Boost. I would say take a look at that. Me personally, I'm not going to lie, my team uses ADK because there's a lot of powerful stuff going on there. For you starting out in the agentic world, build the agent using the most crude main deterministic code as possible, Python, Go, whatever your case may be. Get it to work using flat files, then figure out how you want to port it to the next items like Vertex memory agents and stores and stuff of that nature. Figure the part out first, get it to work. I have some code in my GitHub somewhere to take a look at if you wanna look at what one could be for local enumeration and local tool runtime. And then once you have that, everything else becomes an experiment. So, keep trying, and don't vibe code everything, just vibe code portions.

I don't know, somehow it's just advice for the times. It just reminds me of 2026, right? Don't vibe code everything.

just like a function or something you really can't figure out. And then don't forget the exception handling. I know some of you, including OPSEC, including me, we hate writing exception handlers. Start your guardrails off in the most basic exception handling. If it can be bypassed by regular expressions, you didn't do a good enough job. Use LLM as a judge much later when you finally figured out how to keep your costs under control and get it to work.

Well, Dennis, I can't thank you enough for joining us today. This has been an incredibly technical deep dive into the world of agents for SOC. It's rare that we get this level of technical depth in an episode while remaining accessible. So thank you so much for joining us today.

Awesome. Thanks, fellas.

And now we are at time. Thank you very much for listening and, of course, for subscribing. Please subscribe so you can get new episodes piping hot. And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcasts. You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us. And if we like or hate what we hear, we can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast by Google.

Security. Security.