EP231 Beyond the Buzzword: Practical Detection as Code in the Enterprise

Security running. Hi there. Welcome to Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Timothy Peacock, the Senior PM for Risk Analytics and Detection Platform in Google CycOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloud-security/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners all on the Google Cloud Security Community page. Anton, we are talking about a perennial topic and one we've wanted to do for a very long time, detection as code.

But I want to focus on a different moment and a somewhat surreal one. This is the first banter session we are doing with guests actually listening in the background.

That's not true.

Which is kind of like, at this point, I'm not sure. Like, I'm going to dedicate 2 minutes just whining about that.

That's almost true.

Forget detection. Let's obsess about this. No, no, no, no, no.

Because do you remember in the very early days of the show, we actually recorded that whole thing with the guest online and we did it in a straight just intro into the guest. It wasn't quite banter, but Nelly was there for our very first intro.

Yeah, I think she looked kind of— she wasn't very bored because it was literally the first episode of the podcast. And now this is like roughly 2,022nd, if I'm counting correctly.

222nd.

222nd, yes, not 2,000. Yes. But detection episodes just sell tickets. And I think this episode will just be really good. I mean—

Oh, for sure.

I'm not just saying it to make David happy, who is listening, but it is a— It's just a solid detection episode. And we know that detection response episodes are perennial tops, top 3s, top 5s, top 10 is chock-full of them. You know, they sell tickets.

Yeah. Listeners, why are you all so paranoid and all so interested in detection episodes? Why is our listening group not more interested in good things like governance and, I don't know, compliance?

Okay.

Cue the compliance jokes.

Yeah. Data security episodes. We've done, I think, 2. And I think one of them kind of sucked, and the other one was just like kind of submediocre.

Oh, I wouldn't say that about any of our episodes. I love all our episodes.

Okay, fine. One of them was mediocre, the other one mediocre plus.

Yes.

Okay, maybe.

I think all of our episodes are great, listeners. Anton, more judgmental than me. But maybe without further ado, let's turn things over to today's guest. I'm delighted to introduce today's guest. We have finally joining us here on the show David French, a staff adoption engineer here David, I can't tell you how stoked I am to have you here. Detection as code, that's one of those things, it feels almost like a meme at this point because we hear it all the time, but I don't think everybody means the same thing when they say it. And so I think if I surveyed 20 people on LinkedIn, I'd get probably about 22 different definitions for it. Could you tell us what you mean by it and what the upside is for organizations in your version of it?

Yeah, absolutely. Yeah, my definition, pretty similar to Anton's blog that he put out 4 or 5 years ago now. Essentially, I like to think of it as you're applying software engineering principles to the creation and management of detection content, and that's it. So when we think about DevSecOps-style practices that have been around for a while, you imagine you're working in a software development platform like GitHub, you've got maybe CI/CD tools like GitHub Actions or GitLab CI/CD, and essentially you're going through this kind of software development methodology to create your detection content or your rules as code artifacts. And then you've got your pipeline to ensure that changes are kind of tested. You've got a kind of review and approval process before your content or your rules is finally pushed out to your security tools. So in a nutshell, right, in simple terms, that's The way I like to think of it.

Okay, and what gets better when I adopt detection engineering approaches here, or detection as code approaches? Sorry.

Yeah, no worries. Yeah, a few of my favorite reasons for doing this, right? If we think of traditional methods of managing our detection content in our security tools, right? Us 3 are on a security team. Anton logs into our SIEM, EDR, firewall, DLP, whatever the tool may be. And typically, he can make changes. He can create a rule, he can update an existing one. And what happens there is there's no input or review from anyone else on the team, or testing. And people make mistakes. That could lead to false negatives, you missing behavior, people coming to you, why didn't we detect this? The team feels horrible. So really, it's about kind of treating your detection content or your detective controls as important as preventative controls. There's usually a lot more change control when people want to make a change to something that's implementing prevention. With detection as code, we're making sure that we've got that testing, review, approval process before things are pushed out. It also makes it, when you've got your detection content in one place, one single source of truth, say in GitHub, It makes it a lot easier to determine your detection coverage, document knowledge around detections, right? How do we triage this, respond, blind spots, false positives, all that stuff? And when I think of looking at a rule in a traditional SIEM or an EDR, sometimes you only get 255 characters to document what that detection is about and how to triage alerts from it, right?

Ugh!

So I guess it just boils down to, Any mature security team wants everything as code, right? And that change process around it, whether that's documentation, infrastructure, detection content, playbooks, you name it.

So funny enough, you mentioned that blog and in a few discussions that happened since that time, I did get pushback from people, even from somewhat robust security teams, where security threat detection teams and SOCs, where they say, yeah, but we are not coders. We're kind of like a little nervous about this whole whole as-code thing. We don't want to write detection in Python because it's like too much work. And sometimes I point out to them that the primary thinking here is not necessarily that you must code, but that you would use software engineering principles and practices to your detection. And it ends up being a bit of a minor contentious point. Some people obsess about detection as code, and code just looms large in their minds, and they assume they have to Python.

Yeah.

And they can barely do PowerShell, if at all, or they just philosophically object to coding. They just want to click buttons. And I tell them that, hey, it's more about integration of test routines with unit tests and that test. And it's like, sweet thinking, not necessarily code. What is your take? Do you think these are— which one's more important? And by the way, you have a choice of saying equal or both.

Yeah. Yeah, but both is the cowardly answer.

Except for the cases it may be true sometimes.

Yeah, if we could go back 5 years, I wonder if we could pick a more descriptive name for it. Essentially, yeah, when you're thinking about a rule that runs in a detection engine in a tool, like let's pick a SIEM, and you're feeding it logs and it fires an alert, some people do write their detection logic in, say, Python, and that's great for them. But detection as code, in my mind, is more about As you said, that orchestration of your detection content, like testing it, rolling it out, all that kind of stuff. So with Google SecOps, for example, we make it— my team in particular has released some tooling that helps people get up and running with detection as code really easily. And the reason we did that is because, like you said, you might be a SOC analyst that's awesome at doing searches in the SIEM, doing investigations, responding to incidents, but you might never have written code. So we try and make it easy for you to get up and running with, say, GitHub and your CI/CD pipeline, pulling rules from SecOps, updating them, testing them, all that kind of stuff. So yeah, I like to think of it more about the discipline of creating, managing them, DevSecOps-style practices, rather than, let's write all of our detections in Python or Go, right?

That makes sense. So it's a primacy of practice over Actual code, right?

Yeah, yeah, that's why I like to think of it, at least.

I really like this because it's more of a mindset thing than it is, like, a code thing in many ways. And that's, I think, not obvious to a lot of people when they talk about this. A lot of SIEMs don't have great API support for this. What do you do in that case? And does SecOps have good API support for this?

Yeah, if you've got tools that don't— actually, I'm talking to a product manager, right?

So, You are, unfortunately.

So as you know, security people or users of security tools pretty much expect parity between what you can do in the UI of a tool these days and what you can do via the API. If I can create a rule in the API, I want to be able to do that via the API. And this is even more obvious now with the advent of MCP taking the world by storm. You want to be able to use natural language, ask a question, If you're dealing with a tool that doesn't have good APIs and documentation, you can't do much. So I would say if you've got a security tool that doesn't have a great API with these methods for managing your content and testing it, then I don't think you'll get very far. Yeah, and Google SecOps, like I mentioned earlier, we've got— I would say, I don't know what the percentage is with parity, but at least with rules, I know that you can create rules, retrieve them, update their deployment state, test them, all that good stuff. So I think, yeah, we make it easy to implement detection as code, at least.

But I mean, the '90s are over, right? If your SIEM doesn't have APIs, why are you using it? I'm not talking about— not only '90s are over, 2000s are over. I'm hearing that even 2010s are kind of finished, right? Done. So a SIEM that has no APIs is probably just an absurd concept. But won't there be a side question about what types of APIs? Because, for example, can I have an API for changing the rules, adding rules? Of course. But what about something more advanced, APIs that can enable some kind of closed-loop process with testing, with tweaking, with temporary things? I'm thinking that there may be more to API than add rule, enable rule, delete rule, right? Or is that enough? I mean, maybe that is enough. Maybe I'm overthinking it.

Yeah, I've never been asked that question before. I think for what I'm doing today, that is enough. I was working on a proof of concept yesterday with Google SecOps and some MCP servers. I had my detection content in SecOps, and I had my CI/CD pipeline in GitHub. And asking Gemini, Grab recent cases from my SOAR that have been worked and closed out with a verdict of not malicious or false positive. So it goes and grabs those for me. And then I said, look at the comments in each of these cases and determine from the analyst comments what tuning needs to happen to these rules. And then Gemini determined that and opened a pull request for me in GitHub. with the proposed changes to the logic. And all I needed to do then was go in. It looked okay in that instance, but I could tweak that, and then merge it, and push it out to my tools. So that kind of workflow— well, we're living in the age where this is happening now, right? Are we going to end up with— so people are doing automated rule generation. Are we going to live in a world where our rules are self-maintaining, testing, tuning? Do they tell us when they need to be retired because they're not relevant anymore? So yeah, I think the APIs I'm looking at today are enough, but maybe I'm not seeing something that's going to come in the near future.

No, that makes sense. I think that's— it just comes up sometimes in EDR conversations when people have APIs for where they have to have APIs for action, not just for detection. But I think for SIEM, it does make sense. So I want to switch gears a tiny bit and focus on one key area of well, as code thinking, namely testing. Because it's one thing to just write something, shove it in, and hope for the best. But there's also testing and retesting and testing based on changing conditions in the future. So, how do we do that? Like, does it involve buying expensive tools, or is it something about, again, practices? So, run us through the Deceitless Code testing aspects.

Yeah. So, we can just break it down into 2 simple types or buckets of testing, right? And then I'll talk about one in the middle that's like a happy medium, or one technique.

Okay.

So let's talk about unit tests and dynamic tests, right? So unit tests, think of this like you've got your rules in your single source of truth, right? Let's just use GitHub as an example again. And unit testing will involve, whenever someone is proposing a change to a rule, You want to have your tests parse that rule to make sure that the syntax is valid. Let's say we're using SecOps, right? It uses YARA L for its rule logic. And then what you can do is, let's imagine that Tim developed a detection last week, and as part of that, he simulated— yeah, don't look so surprised, Tim. As part of that, right?

It's not impossible he did.

As part of that, right, he executed an attacker technique or simulated some And the result of that was some log events that he could write his detection logic for. So what you can do is save those log events off, and on some schedule, or whenever that rule is being worked on, you can load up a rules engine, load up your rules, feed those log events into that rules engine, and make sure your alert fires. So those are kind of unit tests, basic offline tests that don't involve your SIEM or EDR. For example, right? It gives you some confidence that your rules are still going to fire when they get fed those logs, right? Dynamic testing, you can manually simulate the attacker behavior, make sure your rules fire alerts. Very time-consuming and expensive. There are tools out there, breach and attack simulation, or continuous automated red teaming tools, which you can program up, right? And they execute tests on a regular basis, make sure your rules fire alerts. That lets you know that your logging, rules engine, alerting, the detection pipeline is working end-to-end. But those can be quite heavyweight. Generally, you might need one person, that might be half of their job or their full-time job to maintain that. So when I think about a happy medium, if you've got 500 detections, realistically, you're not going to have an end-to-end test for all of them. You probably won't want to. Imagine you've got a destructive action, or something where, okay, I've got a detection where an attacker logs into an Okta admin console, creates an API key, carries out other actions. Realistically, you're not going to want to do that in production. A happy medium, I think, is absolutely have an end-to-end test for each data source.

Hmm.

For your EDR, firewall, Okta, Google Workspace, have one. And then, My team is about to release a tool, an open-source tool, and we're going to release a blog about it, where we see a lot of success internally and with customers where, like I said, you've collected those log events previously when you're actually developing detection. On a regular basis, you could just replay those logs to your SIEM.

Hmm.

And that'll trigger your rules, and you'll get alerts. That doesn't give you 100% coverage, right? Because if a vendor changes their logging schema on you, your rule might break. But it gives you—

I've never seen that happen.

No, no, no, no, no. No logs ever changed since 1998, as I recall, right?

Yeah, exactly.

So I do want to ask another question, which is about rule schema. You talk about building out a rule schema. You think about detection schemas as a mechanism to standardize the interface between detection objects. That's such a different way of thinking about detection, as if it were like this object-oriented world. Could you help your security engineer off the street understand what you're talking about there?

Yeah. So if you think about a rule in Google SecOps, Splunk, CrowdStrike, they'll use different logic, rule logic, to identify a behavior and fire an alert. So when you talk about a schema, You can pull all that detection content back into a centralized platform. And when you define a schema, you say, okay, this vendor— let's talk about a rule name. This vendor has this name for it, the field name. This vendor has a different one, and so on. You can normalize that. So essentially, you're abstracting away from all the individual schemas for their rules, and you have your own. So this lets you— it makes your detection content easier to scale and manage and test. If you and I are looking at a rule and it's all in the same format that our team's used to looking at, it makes it easier to understand and work with the detection. So yeah, it kind of abstracts away and makes it easier to deal with detection content across tools. In a perfect world, every company, every security team in the world will be on one platform, but it's just not realistic. Yeah. And we still haven't agreed on, you know, a common schema for logs, detection rules, and—

No, of course not. Of course not.

But what about those sad people who write mostly regex matches in rules? Like, I don't have any snide remarks to make here because admittedly I have pity for those people, but there are many of them. And so if there's no schema and they mostly do detection by means of raw log matching, Do they have hope, or should they just quit and go home?

I don't have the answer to this problem. It's something we've been struggling with as an industry for decades. But I don't know. Is there an application for AI there to help us move towards something that's more normalized as an industry?

Right. But bad data is bad data. And I mean, AI may turn something unstructured into something structured if there is an underlying structure. But some of the stuff, some of the raw logs, genuinely don't have structure. I mean, I don't know. I think that, yeah, you're right. This thought is too depressing, so I should probably quit this line of thinking. Let's talk about response as code.

That is quite depressing. Let's move along.

Yeah, let's move on. We're in denial. We admit to that. So, I mean, detection as code is great. We all love it. We obviously are huge fans and all that. But what about the response part? Some people say response as code is just writing SOAR playbooks. which is, I guess, a reasonable take. But how do you bring more as-code thinking to response machinery? I think it's very much needed too, right?

Yeah, I think so. And I haven't seen an amazing solution to this yet. So generally, if we think about a security team has automated some workflows, right? Like you said, if they're using a SOAR, they're typically creating playbooks, And when you export those via API or some other means, you're looking at essentially JSON blobs, which is not fun to hand-edit if you've ever tried it. So that is one option. I mean, one thing I thought about is you could definitely have a production SOAR environment. You could have all those playbooks backed up in, say, GitHub. And when you're testing changes, you could use the UI of your tool, because I think When you're talking SOAR, the UI just makes sense, right? You've got no-code, low-code solutions out there that just make it really easy to automate stuff without writing code.

Sorry for interrupting, but that's exactly— that is the essence of the paradox, at least in my head. Vendors are screaming about low-code, no-code SOAR, while at the same time, other vendors are screaming about detection as code. And then we here talk about response as code as possibly a good idea, but then we just supposedly had this huge victory of achieving response without code. So that's the befuddling part, at least. And I think I know the answer, but I think that it's just interesting to discuss it a bit more.

Yeah, I think, so SORA aside, if you're a security team who's automating workflows using Cloud Run functions or AWS Lambda functions, totally makes sense to do that as code, right? You're probably using Terraform, or you're backing it up in GitHub and working on it there and pushing it out. Yeah, with SOAR, I mean, I think a happy medium, at least today, would be you've got your test environment where you're writing playbooks, testing them, making sure they work, and then you could potentially put that in your codebase and push it out, right? Because then you've got that review and approval process, at least, without someone going into my SOAR playbook and changing it so it isolates all of the endpoints in a network segment or something, right?

That makes sense.

Yes.

One more thing I want to pick on from your presentation that stood out to me was the call for sharing detection content. Could you expand more on where you want to see this shared? Is it shared internally across platforms? Is it shared between companies? What are you thinking about there?

And by the way, here, the '90s are not over because, of course, Snort rule sharing is from the literal '90s, and people have been discussing this probably since the freaking '90s and never reached a At some point, we're going to have to rename this podcast, like, Security Portlandia, because so often, it feels like the dream of the '90s is alive on our podcast.

Yeah, I think when I spoke about the value in sharing detection content, I was talking about the community, right? So, I think it was around 2020 when I worked for Elastic on their SIEM and EDR product, developing content for their products. we just opened up all of our logic, and some people were like, yeah, this is great. Some people were like, you guys are crazy. But other vendors followed suit— Google, Splunk, Microsoft. So what you have is libraries of detections out there that are available for you to just pick up, load up into your tool, or convert for your tool. And I'll get back to that in a sec. And then usually they aren't great out of the box, there's going to be some tuning and customization to fit them to your environment. You don't want to be overwhelmed with false positives. But where we are— so in my opinion, this has built a better collective defense against threats. But every rule is still in its own schema and works for its own tool. I don't know how we get to a point where we've got a standardized schema where you can load it up. The value in sharing as well is if all 3 of us work for different companies and we're detection engineers, threat of the week comes out, and we spend a day or 2 writing our own detection. If you multiply that by thousands of companies, the amount of time people must be spending in private writing the same detection and coming to the same conclusion is kind of mind-boggling.

Yeah, that is kind of mind-boggling, isn't it?

But doesn't it mean that sharing of that sort between community members, not necessarily between vendors, is kind of a hopeless pursuit. I mean, hopeless maybe is too harsh, but yeah, it kind of sounds like that because, well, even with LLMs, reliable conversion of my crazy buggy rule for vendor X to a beautiful polished rule for vendor Y is not exactly going to work that well. It works a little bit, but not well.

Yeah.

That's only going to get better. Like, I think probably 18 months ago when I saw people touting that conversion between platforms and conversion between logic. I was very skeptical, but I've seen it get a lot better in a short amount of time.

That makes sense. And I think that people who have not used LLMs all that much believe that LLMs can do it really well. Sorry, yeah, I probably should say it again. People who haven't used LLMs all that much believe that LLMs are great at it.

It's so funny. One of my favorite things about doing this podcast is I have this mental tracker of me and Anton's current hype state about LLMs. And sometimes Anton and I are super hyped up on LLMs, and other times Anton and I return to a lower level of LLM excitement and, dare I say, LLM skepticism, which you just saw there.

Yeah. And I've had my ups and downs there as well. I think so. When I built that POC that I was talking about yesterday, like the auto-tuning of rules, and it worked, and I got it up and running in about 5 minutes, my wife, who was in the next room and who's been listening to me talk about cybersecurity for 10 years, she's very supportive, probably bored.

Must be a saint.

Yeah, she is. But I walked out, My office is in the basement. I walked out of the room and I told her what happened. And I said, come and have a look at this. And she was like, oh my God, what are you going to do for work in a couple of years? But on a serious note, I just sat there.

But it's an honest reaction though. That's a notable thing, that it's genuinely really impressive. So, we are already going on the upswing about LLMs right there, right? There are magical use cases. You see it and you say, holy cow, that really happened.

Yeah, definitely. And one of the things I think about is, like, what does— so when I started in security 10 years ago in a SOC, I'd been in IT for 10 years, right? So it was kind of like I naturally slid across to that. But what does the entry-level cybersecurity job look like now? Like, it's, uh, do you have to go and do something else like cloud security, IT, before you go into security? Because I feel like the bar is, I don't know, the height of the bar is changing, right? With AI helping us with certain things, taking care of some of the lower-level tasks.

That is a topic that I think deserves its own podcast and probably a whole series over the next 4 years of what does the entry to cybersecurity look like in an LLM world? But David, before we dive into that, I have to do the thing I hate doing the most on our podcast, which is Ask our traditional closing questions. One, do you have a tip to help people get started on the detection as code journey? And 2, other than Anton's blog, do you have recommended reading?

So one tip getting started with detection as code, don't over-engineer it. Ah, all we're talking about is you work at a company, you've got a SIEM, an EDR, email gateway, you've got rules in all those tools. Figure out how to get them into a single location like GitHub, and then experiment with, like, how do I push updates out, and then go from there? How do I prevent people from making changes when I want to flip that switch in the UI? How do I go through my, like, review, testing, approval, and pushing out to my tools? And you can just start from there. Don't over-engineer it.

That makes a ton of sense to me. And then recommended reading. It could be anything from Mary Oliver to Mary Shelley to not Anton's blog.

To your own blog. I mean, to your own blog. To your own blog, sure. We have a lot of good writing at GitHub. We'll probably link to it anyway, even if you don't want to.

Yeah, yeah, my blog has been a bit quiet lately. I'm blogging on Google's blog. Now, this book, Be So Good They Can't Ignore You. Has anyone recommended this before?

No, that's a new one for the show. What is it?

So, it is about why skills trump passion. And people always say, follow your passion, right? Follow your passion. But I was working at a company a few years ago. Everyone was getting laid off. Including leadership. There wasn't much work to do, there wasn't much direction, and I just thought, you know, like, I could sit here twiddling my thumbs, or I could work on projects and, like, build new skills and a portfolio of work to point to for my next job hunt. And, like, that was a job hunt in the toughest market I've ever been in. So that book, if you're going through a job hunt or you're thinking about upskilling, especially with everyone's talking about AI, right? Velocity of everything that's changing and coming out is crazy. I highly recommend that.

Yeah.

That's awesome. I love that personal story and recommendation.

Yeah.

Thank you. By the way, at the risk of me going on a rant at the very end of the podcast, I remember my own fights with people who are really well-meaning, really motivated, but have absolutely no clue. And so, I almost like now I have to go look for the book because I think that passionate, well-meaning, intelligent people who lack subject matter expertise can sometimes be quite dangerous.

Yeah.

Yeah.

I think it touches on that.

Yeah. Yeah. Well, David, thank you so much for joining us today. I was so looking forward to having you on the show. You are, to my knowledge, one of our maybe second guests to appear on both this podcast and the other podcast by the same name.

Yeah. The other guy is Sunil, right?

Sunil.

Exactly.

Sunil. Yeah. So you're in good company.

So you're in great company with Sunil Potti.

I'll take that.

Yeah. Good job.

Yeah, thanks for having me on. I've been listening to you guys for years. So yeah, it's an honor.

Aw, shucks.

Thank you.

Thank you so much.

And now we are at time. Thank you very much for listening and, of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton underscore Chuvakin and underscore Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.