EP123 The Good, the Bad, and the Epic of Threat Detection at Scale with Panther

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a Reformed Analyst and Senior Staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit your subscribe button. You can follow the show and argue with us and the rest of the Cloud Security Podcast community on LinkedIn. Anton, we have a guest today that I've been so looking forward to having on the show. What did you end up thinking about this episode? Because I obviously just found it delightful.

Well, when Tim says this episode was awesome, Anton says, Actually, it was awesome plus. So instead of me saying something sad and theoretically funny and a little bit Russian, I say, holy cow, it's even better than what you thought. And you get the jumping from joy enthusiastic Anton and not like a cynical, almost curmudgeonly Anton. So this was that episode when I said, what do you mean awesome?

No, it's awesome plus.

Well, listeners, it's a good day when we don't get cynical Anton. Maybe without any further ado other than He's not cynical about it, folks. Let's turn things over to today's guest. Listeners, I'm delighted to introduce today's guest. Today we're joined by Jack Naglieri, founder and CEO of Panther. Jack, I want to start out by saying, when I saw your company the first time, I looked at it and I thought, damn, that's the company I wish I had built. That guy, he's doing something so cool. And so I want to start out with that preface and then maybe dive into something very in the weeds, but very relevant to the life that I live here at Google and ask you, you know, what's good detection? What makes something good as a module or as a rule or as a, just as a piece of detection content? What's good look like? How do we know when it's good?

I love that intro and really appreciate you guys having me on today. It's a pleasure to come on the podcast. To answer your questions in reverse order, or I guess your first one wasn't really a question, but to answer your question, I think good detections start with intent. Like most things in life, you have to have a good vision for where you're going if you want to get there. And I think in detections, and especially in security, it's very true. You have to know what's the end state you want to get to at the end of the day. So to me, good detections have that intent of what's the use case or what's the angle of our threat model, or what's this particular type of attack that we want to detect, you know, starting with that first.

Mm-hmm.

And then working backwards from there to say, okay, well, where can I get this from? And I'm speaking, of course, to the context of a SIEM, right? Where a SIEM is watching all of the things happening in your environment and stitching them together, gluing them together in meaningful ways for us to get good signal at a high level, but also at a low level. So to me, good detections obviously are ones that have a 100% true positive rate, which we know that Wait, really?

No false positives ever?

No, I'm just kidding. I wrote a blog about that.

I'm making a cringe face. In case you use the video, you'll see the cringe face.

No, I mean, I think obviously everyone wants high fidelity, and we could spend the whole hour talking about that. But I think for me, the most important thing is just intent. It's like, what are you trying to get out of this?

Right?

Because if you don't have that in mind, you're probably not going to create a good detection. So that's probably the most important piece to start with. And then aside from that, it's like, well, how creative can we get with our detection? How can we make sure we're covering the right types of use cases that will exist for this scenario to fire? Right. And then is the alert clear? Right. Is detection is, is really one element of it, but what you do with the result of the detection is another thing. So making sure that when it either goes into our automation systems or it talks to a human, that the human or the automation is both enabled to contextualize and actually understand the intent behind it. So that's how I would explain it at a very high level.

So wait, to pull on a thread here very briefly, sorry for interrupting you. Intent in this case just means clarity with what you want to detect, right?

100%. Yeah.

Okay.

Because I think some people can get caught up sometimes. They're like, oh, I want to look at every login. It's like, Okay. Why?

Why?

That's not—

yeah, why valuable?

Like, that's security relevant, but why would you want that as a detection? And I think there's a lot of rules out there that are kind of like that. It's just like, cool, hashtag cool security thing that we could look for, but doesn't really have applicability. And I've been reading a book on essentialism, and it's like really changed my mindset in so many ways about my life. And I think in security, it's especially important. Because we as practitioners can get so caught up with like, I need 1,000 tools to do hashtag good security. But I think like most things, there are things that are essential and there's things that are just taking up headspace. So with detections, it's important to make sure we're not just writing detections to write detections, but we're creating really great inputs into either Some type of correlative system, or we're creating high-fidelity alerts that can stand on their own, or we're creating alerts that are super relevant for compliance. So that's what I would hope people approach it with. But I think we all know in reality, without that guidance, people are just going to do what they think makes sense because, oh, there's a MITRE ATT&CK tactic and technique about this thing, so I should have a detection that flags it. But then is that really going to help you with being either better at your controls, your posture, or actually understanding the things that are relevant for you. Because I think relevance gets lost a lot of the times, which is why I lead with you have to have intent, right? Like, what is the intent? Because the thing that Google would look at is probably something not that I would look at, right? All our environments are so different. So again, this is such a meaty topic. We could talk about this probably for an hour, which I guess that's the intention of this, right?

That is the goal. I think what you said there at the end is really interesting too. I've had users say, hey, how come you don't cover every MITRE ATT&CK tactic? And they lose the relevance on some of them. And it's trivial to see examples like inserting removable media, not a threat you have to worry about in the cloud, but it's on the matrix. And so if you're taking a somewhat simplistic approach of like the MITRE bingo card, you would suffer as a result. So intent is a really That's a nice lens on it.

But this also brings me to the other question of good. So, again, I don't want to torture the audience with another example of Anton's 3 AM test, but if somebody wakes me up at 3 AM and says, Anton, what's a good detection? My mind may not go to the micro level, like how to write a good rule. My mind may go to a good detection program at a company, like, do I have a good detection program? This is— or maybe at 3 AM, I would get confused because ultimately, these 2 choices are very reasonable. Good detection means I wrote a solid rule, or good detection means I have a function at the company that detects threats well. So, Jack, what's your answer on what is good detection when you apply the question to a program at a company or an organizational capability?

A good detection is one that produces a contextualized alert That allows you to understand the initial vector that you were monitoring for and the surrounding reasons as to why that happened. In a nutshell, right? Very hard to achieve.

But you said allows you. It implies there's a human. So good detection implies there's an expert or at least some human who is charged with that.

Well, I don't know. You can mean, you could mean a service account. It could mean the automation can then have enough context to either make a decision or go get the information to make a decision.

Fair.

Yeah, but let's just assume that it's a human because at the end of the day, humans are still a really important factor today until the AI takes over, of course. But having the analyst or someone who's versed give the final say before you go do something that's potentially harmful or destructive. And when I say destructive, I mean you're going to destroy an instance or you're going to take someone's laptop offline or something kind of like that, they should have all the context about that behavior to understand it. So an example of that is, who is Anton? What does he do at the company? What does he typically do? Or what did he do an hour before and after? Like, those types of things are super important because otherwise those are the number 1, 2, and 3 questions that an analyst is going to go figure out. So there's certainly an automation aspect of this that has to take hold. And then As we were also alluding to, automations can ping people. And this is why SOARs exist, which we could get into a whole thing about. I personally think that SOARs exist half for the wrong reasons, which is that SIEMs were really bad, so SOARs existed.

The SIEM vendors slept at the wheel for a while. And that's how YUBA emerged and then how SOAR emerged. You're right. That's a popular explanation, is that SOARs are a sign that SIEM vendors slept at the wheel.

For sure. And I think that it makes sense, but I think that SOAR is also, aside from that, have a good purpose, which is to orchestrate the set of steps that would be required to do a remediation. Like, that's their intentional, like, that is their intent of being creative, right? That's, that is literally what SOAR stands for, you know, Automated Security Orchestration. Like, that's really what they were going for. So it of course gets lost when you're feeding 10,000 things through it a day, right? That's not really the purpose of it, in my opinion. So anyways, I think that good detections should result in nicely contextualized alerts to where you, proverbial you, meaning AI/automation/human, can look at their phone and be like, okay, this is really bad. And that's ideally what you'd want, right? I always break down internally, I always break down alerts into 3 categories. I start with things we know are for sure bad, things that we know are maybe bad, and things that just happen that we need to keep track of. And another thing, I mean, I don't try not to speak too much about like plans that we have, but we'll announce some stuff later in the year that sort of changes this paradigm as a whole. But I think the 3-tier understanding of alerts is typically correct, at least from my experience, because—

So then wait, wait, bad, Probably bad, suspicious, or am I oversimplifying it? Bad, likely bad, suspicious? No?

So there's certain things that happen that have such a high fidelity that they're absolutely undeniably bad.

Right. Wake them up at 3:00 AM kind of a thing. Yeah, it's really— you go escalate, it's an incident.

Exactly.

Okay, that's 1. What about the 2?

The middle tier is something that's suspicious that could be bad that we need additional context to understand.

Okay, so then that third bucket then confuses me. If I assumed number 2 is number 3—

No, so the third bucket to me is a security relevant, a security or compliance relevant event. So an example of that is like you could say a login is a security relevant event, which it is. It doesn't mean that you need to alert on it, but in the essence of a SIEM, so if you think about what a SIEM is doing, or any really like data pipelines where you're doing streaming analysis or any type of analysis, you're taking a gigantic corpus of data and you're whittling it down. You're whittling it down to 1% or less than that. And within that 1%, you have these 3 tiers. You have the stuff that's security relevant, you have the stuff that's kind of suspicious that could be bad, but probably half of the time isn't. And then you have the things that you know for sure are really bad, like hits on prior threat actors that confirmed breached you, certain types of commands that were run. So A great example of that is someone running wget to a known bad domain, piping it into Bash in a production host.

You're like, okay. Yes, yes, yes.

That's bad.

That's just bad. Yeah, it's just terrible. It's not bad, it's terrible.

Yeah, but those things happen, right? And there's the— we try to get, going back to what a good detection is, we try to get more confidence and enrichment and contextualization on everything. So in that example I just gave, You can say like, well, wget is bad. Cool. Like that definitely shouldn't happen. But sometimes developers are doing that because they're testing something. Well, you can contextualize where that event came from, right? Did it come from production or did it come from a development account? Cool. Now, what is the thing that they've went and fetched? Oh, well, it has a domain and IP or an IP associated. Let's look that up. Oh, that's actually known bad by VirusTotal. Why is it known bad? Oh, well, because of XYZ reasons, right? So you can follow this chain further and further and further, and the more depth you have, the more it can fall into that like far right category that I would say is like known bad.

Mm-hmm.

And then the less confidence that we have, I would sort of shift it into this like middle gray area, which to me in my career is really a lot of where we had operated in with security. Detection response is just super hard because it is so gray. It's one of the areas of security that is so non-binary that it makes it really hard to do your job. But it can be more black and white if you add enough context to have an understanding of that. So we want to be one or the other, right? We want this like, yes, this is bad. No, this is fine. Right. But it's starting to get really blurry. And the things that we don't deem as really bad or bad at all can sometimes become relevant.

Yes.

Uh, when you look at things in chunks. So an example of that is like, Oh, well, Jack logged in.

Fine.

Sure, that's fine. But then later he did some other stuff, and then he did some other stuff, and then the collection of those things now becomes relevant. But because data is so huge now, you need to be able to siphon those things off for use in these risk-based methodologies, which is what we're seeing a lot of people really just doing in general, not only SIEMs, but other types of platforms as well. And cloud security, for example, So yeah, it's a really fascinating time, especially with a lot of the innovations happening with AI and like other ways that we can be in classifying data. But regardless, security is, is very black and white when it comes to detection. And the things that we're trying to do as practitioners is to increase the confidence of these signals more and more and more and more. And there's so many different ways to do that. AI is certainly not the silver bullet that's going to save us, but it is one element that can be extremely useful. For things like just understanding trends or understanding similarities. Like, I think similarities is probably going to be the biggest thing that we get as practitioners. So I'm really excited about that. But the core of this question is really good, which is really orienting people around how should I even think about writing a detection? Because I think in the past it's just been very, you know, hand-wavy. You're like, just like, well, I think this could be good signal, but it's like, well, why is it good signal? So leaning into the why, leaning into the intent, and then leaning into contextualization, that makes it less binary or more binary. And that's actually what we do want in security.

So I really like this contextualization direction, and I'm not surprised at what you said about not talking about things you're doing later this year. I too, in this area, don't want to talk about things I'm doing later this year, but it's obviously an interesting, and I think for many organizations, very ripe area for a vendor or a partner to provide good work. So that's, once again, the similarities are at least amusing, if not validating that we're thinking about problems in a similar way. So I think Anton wanted to ask a question that I think is pretty interesting, so I'm going to ask it. How do you design an organization to produce detection content at scale?

Yeah.

Like, how do you go from, okay, I've got a couple of security engineers and I've brainwashed them about this intent thing, and now they can think through that, To, I'm a factory that produces really good content for a variety of substrates that my users appreciate.

But I think that also I want to strengthen it that it's good detection and at scale. Because I think that we really care about both because ultimately you can hire somebody who's really good who would write 3 detections for you and leave. And then what?

Then what?

It's kind of not at scale and it's good in some micro level that wouldn't change. So to me, this is the key part of the question is, How do you build the capability to produce, to reliably produce good detections at scale?

So the question has a few different angles to it. Do you mean as an organization that's doing an internal security program, or do you mean for me as a SIEM vendor to other companies?

Are you a SIEM vendor?

There are enough similarities.

Wait, wait, wait, wait. Hang on.

Hang on.

Did you just say you're a SIEM vendor?

Well, he's a SIEM vendor. What's contradictory about that?

I mean, I don't think of Panther as the SIEM.

Yes.

Uh, what, what are the choices he has? He's not an XDR, otherwise we'll boot him off the podcast. Yeah.

Only our one guest would do that.

Yeah, I, I, I'd boot myself off the podcast if, if I called myself an XDR.

Yeah, that's fair. I could see it being a SIEM vendor.

Yeah, I mean, we are. I mean, SIEM is written into regulation. Like, it's not an acronym we're going to lose anytime soon. I think we have so much heartburn as a security industry about it. Because of the, the comment we made about SOAR, right? Like, well, SIEMs are so bad that we need this other thing to make them less bad. All right. I always explain this as well internally. I always say SIEM is a gigantic industry with extremely low satisfaction. And when you look at markets to go attack and approach, those are the ones you want to go after, right? Those, that's the biggest opportunity.

Yes.

So however, Also, because of what we were talking about with contextualization and these other things, it makes it extremely, extremely hard to be successful as a sim. And that's a challenge that I think we all have. As you know, I'll consider Panther as this latest wave of sim companies where the past wave were the ones that were more popular in 2015 to 2018, right? And then from 2018 onward, you have us and this other cohort of people Google is included in that, obviously with Chronicle and a handful of others. And yeah, it's always very challenging. I think we need to really just understand like what the purpose of sim is, because I think a lot of people lose that.

What is the purpose of sim?

Oh no, he's going for it.

You said it. I had to ask. I would be a terrible podcast host if I let you get away with just saying, oh, the purpose of sim, and then didn't try to pin you to the wall on what it is.

Yeah, I mean, this is my opinion.

I don't know if I have a blog written in the last 2 weeks about that. So I think, I mean, let's see if he read the blog or not. Okay, I'm joking.

You wrote a blog?

I did write a blog about what's the purpose of—

He always writes a blog. Don't, don't listen to him about his blog. He's always writing blogs.

Yeah, I mean, if I take off my like security vendor hat and I just put on my like I'm a CEO hat, the thing I would want out of a SIEM is to watch over my controls and make sure that they're working. But I'd also want to make sure that we're checking and we're sort of auditing the controls.

Mm-hmm.

So if I think about at a very high level, I want something that's stitching all of the things together in order for me to see movement of my employees and movement of my users in my systems to ensure that, one, my controls are working, and two, that We're just generally looking at things holistically. So if I think of the general purpose of SIEM is to look at all of these security and logging and systems holistically and find the connections across systems, that to me is what the purpose is. The purpose, in my opinion, is also not to quote unquote be the best detection capability in the world because there are verticalized pieces of security that are designed and they're better for that. So an example of that is like Snyk, right? Let's use like developer security as an example, right? We could absolutely ingest logs about repos and we could do these types of things, but why would we? Like, if you don't specialize in a thing, then you're not going to be the best at that thing. And that's just a well-known principle. You know, you have to focus, right? What is your wedge? What are you, what are you going to be great at? If you guys are familiar with the hedgehog concept, right, from Good to Great. It's the same thing. It's like, and as humans we have that as well. It's like, what are you going to do in your career that's like satisfying to you, that like helps you support your family? Like there's this like intersection. There's so many concepts about this, right? Same thing applies in business. What are you going to do that's like you're going to be the best at, right? Mm-hmm. I think sim naturally, because it has this wide surface area, it's like, oh, well, we're going to be great at search or we're going to be great at X, Y, Z, right? You have to be good at one particular thing. And in my opinion, the thing that SIEMs have really failed at and probably is the reason why they haven't really had this like rapid adoption and displaced Splunk is that no one has really thought about it like, well, like what's happening holistically? Like how do we get really good at that? How do we get really good at like looking at patterns and doing these things that allow us to get the benefit of having all this signal in one place? Right. That to me is the purpose, right? The purpose is to see everything, right? And if you're not able to see everything in a way that makes sense and is conducive to good security monitoring, then what are you doing, right? But hence we've had all these products in SIEM and everyone's always like, oh, well, SIEM is such a crowded space. And that's a true statement, but there's so much variety in how people have approached this.

And mediocrity too.

For sure.

Yeah.

But I think this is really, really interesting because it's that connections across systems that you alluded to there. that I think makes security so hard. As a domain, as defenders, we don't just have to understand one system that we're not responsible for building or maintaining. We have to understand the intersection of N systems where the complexity emerges not from one of them, not from two of them, but from their multiplicative complexity. And so, yeah, a SIEM has to be good at that.

But now I feel like an old guy, by the way, briefly. And the reason I feel like an old guy is that I vaguely remember writing a paper, not a blog, mind you, that sort of highlighted what many years ago used to be called cross-device correlation, where the point was not to detect, hey, Windows log event ID 1152 is bad. It happened 5 times. Wake up. But more about organizing things across devices, which was briefly a purpose of SIEM. Please don't cry. 20 years ago. it sort of went somewhere else. So it's interesting how your point about this type of comprehensive visibility that makes sense reminds me of the very first few days of SIEM in a strange way.

I just don't know what other purpose there would be, right? And I also don't know what other tool would satisfy that because you could think, well, CSPM, but it's like, well, CSPM is looking at the attributes of your systems in the cloud. That's super, super important. That's actually one of the most important things that we could do well that would prevent breaches. But we also know that that can be a gray area as well, where not everyone's going to make the most hardened systems in the world. So how do we make sure that when things go wrong there, we can answer questions about everything, about what happened across these systems? Because when breaches happen, they don't happen in one place, they happen in many places.

No.

You get in on one and then you sort of expand, right? That's why we talk about blast radius when we talk about incident response is like, what's the blast radius of when the bomb went off? It's like, well, what did the attacker achieve? How were they able to achieve it? It's like, well, they came in through the WAF because using the Capital One breach, coming through the WAF, going onto the system, and then running proxy commands through that that did a bunch of other things, right? So it's like the initial vector, and then it's like rapid expansion. So the SIEM has to be the thing that's saying, okay, we saw that one request come in, and hopefully you have that, that level of fidelity, but not everybody does. But regardless, you're probably looking at CloudTrail and S3 and these other AWS things. But again, you need to be able to see across all of them in a holistic way, and no one's truly achieved that, probably for an important reason, which is that the data engineering is so hard with that.

And so hard.

Yes.

That's such a dumb reason for us to not be able to do this, obviously. But I think Amazon and who else is part of that? I think Splunk's part of that too. OCSF.

OCSF.

I think this idea of like, we're going to like normalize everything. But yeah, there's still challenges even with that, right? Like, is that going to be the industry standard? Like, don't really know. I know that it's like progressing and it's getting better. And I think I applaud them for doing it. And you guys could probably keep me honest, but like, it feels very similar to like what Chronicle did for Your guys' data model, which I also really like. I think it's very well thought out.

Right. And the years before, there was Common Event Expression, and before that was CEF at ArcSight. It's the history of log standards. I've been burned by this many times personally, so that's why I'm kind of a little bitter. But OCSF is an effort today.

But it's also like the XKCD cartoon on standards, right?

Right. Correct. But in this case, we don't really have 14 standards people care about. There are 14 standards people don't care about, and you should create number 15 because Because obviously number 15, people would start caring about. Okay, fine.

Jack, we got Anton fired up about that. You found something he loves.

Yeah, for sure. Log standards. I spent years of my life trying to push it ahead and ultimately failed. So now I'm a little bitter and envious at the same time. But let me briefly hijack the conversation. And given the timing, I wanted to use Jack to resolve the argument that Tim and me are having for a year about custom versus canned detections.

Oh, no.

And it's a bit of a surprise question, and I've been dragging it every time to ambush guests with it. So, Jack is obviously a vendor, so he makes detections, right? But your customers also make detections. So, when somebody wakes you up at 3:00 a.m. and says, Jack, can you, as a vendor, build all the detections I would ever need so that I won't need to do anything custom? What's your reaction? And I'm cheating a little bit and asking the question that makes it very Anton-serving and makes Tim lose an argument, but still.

Our job is not to make every perfect detection for every organization because it's just not possible. And I think the approach that we're taking, which is how other people have taken, is at a baseline, you have the 80% of MITRE ATT&CK or whatever you want to call it. 80% of the detections that we've all agreed as a security industry are Hashtag bad, you know, and hopefully it ends in that last column that I was talking about, which is things that we know are bad. But I think most SIEMs end up doing things and we can like pull up examples as we're talking about it, right? A lot of them are open source, like Splunks are open source. I think Elastics are open source. There's the, what's that one project where you can convert all of them into different formats? It's Sigma.

Sigma.

Sigma. Right. Like they're all pretty much the same, right? So I think for me, My goal is to think about in building blocks because there's a lot of patterns that exist when it comes to writing detections. And if you give people a way to customize and mold it for their organization, that's success. But no one wants to start from zero. That's just a known thing, right? I wouldn't want to start from zero if I was a CISO and I was bootstrapping a SIEM program. I would want the baseline of like, well, I want to at least understand, again, going back to connecting the dots across everything, I want to understand that there's a system that's flagging the security-relevant things as they happen across my infrastructure. So yeah, the SIEM should get really good at bringing those to the surface and then having the capability to, one, determine when do you actually want to get alerted on these things. I think alerting by default should probably start dying.

Yes.

Just given the fact that, again, going back to essentialism, it's like, what do you actually need to get alerted on?

What? Alerting by default should start dying?

Yeah, why not?

Of course.

Why not?

Why would you alert on some garbage rule like a UEBA, users in a new location? You should alert on a richly contextualized set that together are interesting and have enough for somebody to respond to.

100%.

This is what we do at Google, Anton.

But wait, wait, wait. Why are you confusing alerting on some shitty rules should start dying versus you should write better rules and alert on them? And I would assume that if you improve the quality of detection content, make it more applicable, then you should alert on that because it would obviously be good. Otherwise, go back to the lab, write a better one.

No, I think maybe the challenge here is there's probably a disconnect between what you're thinking of as a rule, Anton, and what's achievable for an engineer developing something to make as a rule. And then how, as an industry, we need to shift from individual rules to composites of rules.

Exactly. That's exactly right.

Yes.

And there's been a lot of trends like this. So there was a project around graph-based detection that I thought was really cool, right?

Very interesting there.

It's really interesting, but I should ask him, another guy who's— and I don't know if we're talking about the same project, but I know a guy who started this project and I should ask him this question, which is, what if there are multiple ways of traversing the graph? Does it fall off? Right? That's the thing I always wonder with graph-based detection. I'm like, it's cool, like sequential types of detections, and there's a lot of ways of slicing and dicing it. But I think, Timothy, exactly what you said is correct of thinking of things in composites. And you can kind of do this today with search-based SIEMs, and I'm calling that like Splunk, basically like Splunk Elastic, where they can do a historical search on logs and they can kind of correlate things together. But again, like those languages are so limited, which is why You know, both us, Panther, and Chronicle adopted like the detections as code idea, right? And in different, slightly different ways where you guys are doing it with YARA and we're doing it with Python, right? And we actually just introduced like a low-code way of doing it as well. So you don't have to always write Python for it. So, because a lot of detections are really basic, it's like field equals value, field is in this list, like whatever, right? It's not anything where you need to go like. Write a Python class and do it. But like, that is the power of having a programming language is that you can do those things. And it's not this like painful, confusing query that you don't know what it does and you can't test it.

Let me materialize a table of all my bad IPs and then join a log against that and see if it's there. Like, that's to me kind of a crazy way to build detection.

But this is the challenge.

I'm making this hand waving and cringing at the same time.

But Anton, that's what YARA L and reference lists kind of give you. There's challenges there, and we see that internally at Google as well.

Yeah, well, so it's a constant challenge between the bar of the community, the technical bar of the community, and the capability, right? Yes, because graphs are a perfect example of this. Graph-based detection is super cool, and you might be able to express like a lot of really helpful TTPs, but like 0.00001% of security people probably can efficiently do it, right? And It was a problem I even had at Airbnb where when we were doing StreamAlert, we had half of the team were engineers, half of the team were analysts, and the analysts really struggled to wrap their minds around, you know, how do you deploy to a Lambda function? Because that's not their job. That's not their experience. Their experience is responding to breaches and being really good at that. And engineers are really experienced in building serverless infrastructure and building AWS infrastructure or whatever else, right? DevOps. DevOps and software development. So it's a really hard challenge because the thing that we need to solve the problem is highly technical, but most of the industry is not highly technical. So it's like we have to bridge that gap somehow. And I think therein lies the challenge that SIMs have, which is to connect those 2 things together. And we haven't actively done that. And there's a lot of, if I were to build a hierarchy of needs, which I would love, I don't know if it exists, but I'd love to create one of these. Like the base of the hierarchy is like good scalable infrastructure. And that's a thing that just didn't even exist like 3 or 4 years ago. You know, no one was really good at just having the capability of handling a petabyte of data per month. And, you know, outside of the FAANG companies, petabyte of data a month isn't that much. But for other companies, it's a ton. It's like mind-blowing.

That's a ton.

Right? When I was at Airbnb, we were thinking in the order of like terabytes of data per day, which at that time was was really big because we were coming from the Splunk or the Sumo world where everything's priced in gigabytes and we're like, well, money.

Yes.

We can't do that. Like, that's not gonna work. So let's do something that's sophisticated. So that's like the base to me. And then now it's like, cool, then how do you extract signal from it? Cool. Now how do you make use of that signal? How do you contextualize it? Like, it goes up and up and up and up. And then, yes, you know, hopefully we, we get to a point where we can achieve this like nirvana of alerting where it's like a concise amount of things that are very high High signal, which is always what we want. But I think we do need to be extreme and take a different approach of like, no, I literally, I'm not even going to care about certain data sources, or I'm not even going to turn anything on until I have good confidence that my team is getting, you know, a handful of alerts a day. And the beauty of being, you know, I keep saying the phrase like SIEM vendor is that like we can see the patterns of other teams. And that's been the most interesting part of this whole journey for me as a founder or someone who was inside of a company and then made the switch into working with other companies. And that was a big reason why I wanted to start Panther. I'm not talking about that, but like, that's been super fascinating. And then you get these teams who have all these different ways of doing security, which is—

Yes.

Super fascinating. Like, our customer list is fairly public, like most of them. And we've worked with some really cool teams and I wish I'd asked for their permission to talk about some of the stuff they're doing because It's really cool. You see the teams that are very technical that are doing the more sophisticated stuff, and then you see the teams that you think would be more technical that actually aren't, and you wonder why. It's like, well, what's different about them? So for us, it's really interesting because then we have to create the average of the things that they need. And going back to this question you posed, Anton, about the baseline detections versus the custom detections, it's like you should cover the baseline and you should work really hard about the capability for them to create easy custom detections. Right? Like, that's, that's where my mind's at. Detection as code is one of those ways, but it's not super quote unquote scalable for the majority. If you think of the bell curve of tech adoption, right? Like the early and late majority.

Mm-hmm.

Yep. Probably aren't people who are proficient in thinking about software, and people get a little spooked when they see code sometimes. And I'm like, I was Googling like how to write a Python script when I was like, just starting in security. I'm like, why is this so intimidating? Like, we should be thinking about it in a more like growth-based mindset. But, you know, yes, everyone's different, right?

That's something somebody said to me in a SOC one day that's really stuck with me. I was asking them actually about exactly this area of custom detection, and I was like, you know, we're thinking about these different language options. What would you like to use? And they said to me, and I thought this was interesting, I would really like it to be SQL because that's a skill that I could then take and use in another role.

Ooh.

And they had a very growth-based mindset about their own experience in their current role. And now I don't think SQL is the right answer. I think Python is actually a great choice for this. We're nearing our end of time and I want to ask our traditional closing questions. It's clear the 3 of us could talk about this all day, and I think our listeners probably would rather it be multiple episodes than one super long episode. So I'll ask our closing questions. Do you have recommended reading? You've already dropped some. I would love the names and maybe one tip for organizations to improve their detection engineering.

Recommended reading about what? I read a lot.

So anything.

Well, that's the point of the question. It can be the book Dune. It can be your own blog. It can be—

It could be the Essentialism book.

Tanter.com/about.

Whatever you want.

It can't be Anton's blog, though. That's not allowed.

I'm going to put maybe 5 or 6 of my own blogs and resources anyway, because I'm like a bit of a So I'll give a book.

So I just did an audiobook for The 5 AM Club, and I loved that book. It was actually very adjacent to a lot of the things I've been working on just with my own, my own mindset, my own health. They call it like, they have like mindset, healthset, heartset. Like they have all these different elements of like being in that book. And I thought that book was really great. I really enjoyed it.

The 5 AM Club?

5 AM Club. Yeah, that was really good. And then I'm currently reading Essentialism, which has been like mind-blowing to me. So I love, I love both those books.

Who's the author?

It's Greg McKeown. McKeown. I don't know how to say his last name, but that's a great book.

Okay.

And Jack, our other traditional closing question. Do you have one tip to improve people's detection engineering?

Yes. The one tip I'll say off the top of my head is when you're testing your detections, make sure you have a positive and a negative use case. So meaning you have an example that is, this should always fire the alert. And then this other example that should never fire the alert. And I think if you follow that simple rule, like you'll pretty likely reduce the overall volume pretty significantly. And then, you know, the other tip I said in the beginning, which is intent, like have a really clear intent of what you want out of it. And I'm giving you 3 tips actually. So the first one is good rules. The second one is intent. And then the third one is context. You should be able to understand it in a sentence, like what it's doing and why.

So those are my— that's awesome.

Those are my 3 tips. I think about this stuff every single day, so I have a lot of tips, but—

Jack, those are great tips.

Cool.

Thank you. I appreciate you guys.

Thank you so much for joining us today.

Yeah, it was a great conversation. Thanks for having me on. I would love to do it again at some point.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton underscore Chuvakin and underscore Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.