EP282: Debating Coupled or Decoupled SIEM with Alex Hurtado and Christopher Witter

Security. Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here today are myself, Tim Peacock, Group PM on Google CycOps, and of course, Anton Chuvakin, a reformed analyst and now senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as on our YouTube channel. youtube.com/@cloudstackpodcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. While you're at it, you can drop us a review on your podcasting app of choice. Anton, we have 2 guests today, one who we've wanted to have on this show for years now, and one from a company that I'm personally very grateful for because I would not have become a PM without their involvement in my career. Did you know that story?

I did not.

So many years ago, when I wanted to become a PM, I asked my employer at the time, hey, can I be a PM? And they looked at me and they said, well, you're not a PM, so no. I said, that's interesting. How does anyone become a PM then? And so I went out and interviewed at a bunch of places, and they all said, listen, kid, you're not a PM, so get lost. And that was a bit of a bummer. And then I met the security team at Dropbox who wanted to hire me to be their PM, and the PM team said, he's not a PM, get lost. And so the security team wrote me an offer to be a product specialist. I printed out that offer, which is very much not a PM offer, very different. I printed out the offer, I put my thumb over the word specialist, and I showed it to my boss at the time, and I said, look, I'm going to do product at Dropbox unless you let me do it here.

I did not know that story.

And I think it's— I am very grateful to Dropbox. for giving me the opportunity to become a PM.

Without being a PM before.

Without being a PM before. And so listeners, if you've ever wondered how people become a PM, sometimes it's because you're an engineer, sometimes because you get an MBA, and there's a whole bunch of PMs out there like myself who got in by hook or by crook.

Yeah, I mean, we should probably save the story how I became a PM briefly for some really drunk episode. Because there was a moment in my career a good number of years ago when I literally was— I may have been a director of PM. I don't know. I want to not remember it. I want to not remember it. So, I was a really bad PM. And I think maybe my story would be like, how do you become a PM and then prove that if you're a really bad PM, you really shouldn't?

Well, listeners, I think it's very important in your career to figure out what you should and shouldn't do. And there's a lot of things I know I shouldn't do. And I bet there's things that Anton knows he shouldn't do other than just PM. What are we talking about on today's episode other than funny bird stories?

Yes. So this is actually—

So what did you talk about?

This would be actually one of the longer banters because I feel like it's too much fun. But the topic of the show is also very fun. For many— I wouldn't say decades. No. For many months. Long story. In AI years.

Oh, that is decades in AI.

Yes, decades.

Exactly.

People have been debating debating whether these new advances of AI agents are pushing SIEM, you know, SIEM, security information event management, you know, from mid-2000s. Yeah, exactly. Surprising, right? To a more centralized structure and more integrated than like, give me my SIEM, my SOAR, my EDR, my random other crap, maybe my firewall in the same tool. Well, maybe not the firewall. Sorry, guys. But the other one is more about like, hey, I don't actually want the analytics SIEM. Can I have detection from vendor A, storage from vendor B, pipeline from vendor C? And this is sometimes called decoupled. And if you decouple SIEM, I think I made up the term, but if you also do storage that is not centralized, then it's also federated, meaning that you can decouple SIEM capability but keep storage centralized. So, these are a little orthogonal.

Okay.

This would be in the short term. Okay.

Good mentions.

Decoupled, federated. The point is that conventional wisdom, which I define as random guys on LinkedIn, random people on LinkedIn, Seems to hint that the decoupled and decentralized SIEM is kind of the future. It's the cool stuff. Everybody wants this. Well, centralized is kind of like, meh, 2000s are over. But when you look at what people buy—

It's still the 2000s.

They buy mostly centralized. And a relative minority, and there I say a tiny minority, is actually buying decoupled. Even though the noise on LinkedIn implies that it's the thing.

Are you, Anton, trying to tell me that social media is not real life?

No, social media is real life. Who would say such a thing? Obviously.

Oh, sorry, sorry. I'm jumping the gun.

No, no, it's like—

My bad.

My bad.

Of course.

Of course not. So, and I just wanted to have people who show up on the show and say, I'm going to argue position A, I'm going to argue position B, and have them debate it without us, you and me, or just me as things happened, sticking their nose in the debate, being kind of a neutral observer. And this is kind of what this episode was, even though our guests were like super polite. And when— in a few moments, I thought one guest would skewer another, and they just didn't. They missed every chance to skewer.

No shish kebab.

We did not get— I wanted a skewer fight, and I wanted a pillow fight. I just wanted a fight. But it ain't a fight. It's a very gentlemanly— hold on, no. It's a gentlemanly, gentle personally, because one of the guests is a lady, so it can't be a gentlemanly debate, duh. It's a gentle person— what's the term? Gentle persony?

It's a gentle debate.

It's a— thank you, Tim.

Okay, very good.

It's a gentle debate.

Listeners, I think Anton and I have rambled quite enough, and what I hear from him is that this actually turns out to be one of our longer episodes. So let us know in the comments or on Twitter or on LinkedIn whether you like the longer format, whether you like our longer banter, and whether you, of course, well, missed waffles in the opener for this one. So without any further ado, let's turn things over to today's 2 guests.

And today we have with us Alex Hurtado, Director of Detection Engineering at Scanner, and Christopher Witter, DNR Lead at Dropbox. Welcome to the show.

Hi. Thanks so much for having us.

Yeah, thank you.

So, this episode is kind of special, and not only because Tim is not here, but because we've been kind of stuck in this debate, in this metaphysical debate about the future of SIEM or directions of SIEM. And by the way, if you don't like the term SIEM, think of it as a technology to support your detection and response efforts. That isn't about debating the acronym. So, where I'm going with this is that in recent years, there are many people who came up with technologists, as well as positions that this centralized SIEM, one big platform stores data in the cloud or on-prem, is no more. And we have to go with decentralized, federated, decoupled. There are many other terms that imply that it isn't just one product you buy from a vendor. You give them money, they give you storage, security detections, dashboards, everything else. And instead, you need to do something, well, other than that. On the other hand, success of certain large vendors that actually centralized SIEM, EDR, SOAR is kind of the vote, at least to me, for the opposite side, that actually centralized is winning. And however interesting things are, they're actually even more interesting because when people start using AI agents, they actually make the agentic argument to suit their original position. And by the way, this is not Anton's gonna talk forever podcast, but I wanna make a point about agents and then I'll hand it off to you guys. The agent point is interesting because it's so surreal. People show up and say, Anton, obviously, decentralized is winning because now I can just have an agent go and get the data wherever it is and everything's gonna be fine. And then I say, okay, if the agent goes and the data is not there, what is it gonna do? And of course, the answer is it's going to give you a confident answer based on no data. And it literally happened to somebody. So, they then say, okay, so agents are actually a vote for a centralized approach. Then a whole bunch of other people are upset. So, we just want to have this debate here. And Alex would kind of play for the decoupled decentralized side, and Witter would play for the centralized approach.

Yeah.

Centralized side. Fair? And I'll try to be neutral even though I'm kind of a bastard. So, I'm almost never neutral. And I'm gonna try to annoy both sides equally now that Tim's not here. It's fair?

Fair.

Okay.

I'm ready.

Alex, why do you think that this whole decentralized decoupled SIEM is a good idea? Why is it not luck-based SIEM?

So, I think we can probably level set and start maybe with the definition of what decentralized or decoupled, as you coined it a few, several years back, really is. And to me, and also for some context, I've seen the sim sim in probably all the ways that it can be simmed, as a verb. And there's many different ways to sim now. And this I have found to be a very super comforting way for many leaders that in particular are dealing with the two-headed SIEM monster, or they find themselves having to manage multiple SIEMs in their environment.

Mm-hmm.

So, starting off the bat there, but back to the definition, a decoupled SIEM is one— is a SIEM that separates essentially storage from detection and from even now collection. So, all of those kind of 3 parts live across different layers essentially, and it's kind of like you can pick your own, you know, tool to handle each layer, whoever does it the best. Some call it best of breed, per se. And instead, and it's very, very contrary to doing it all under one monolithic architecture where before the SIEM handled the pipeline, right? The log collection, the syslog forwarding, all of that. And it's also the same place where it's stored, and it's also the same place where analytics happens on top. So, that is, in my definition, my words, what I like to call is that decoupled SIEM flavor.

That sounds interesting, but also operationally, at times, sounds absolutely horrible. You have to pay 3 bills, right? So, Witter, do you have anything to add to this? You can support it, you can criticize it, you can redefine it if you feel like it.

I mean, you know, she's not wrong. Like, I like the best-of-breed approach, but also, you gotta realize that a lot of organizations Procurement is gonna drive this. And like, you know, the 3-vendor problem, one throat to choke, or one belly button, if you will.

Maybe.

Like, I gotta go to one person in one place. And it's that, you know, that procurement that typically is gonna get in the way. They're like, oh, well, we're gonna get this really cheap. And so the, like, one vendor that rules them all, it's like, oh, they will throw in pieces and make it a harder decision for organizations. Security's always cash-strapped, not just headcount, but for tools and all that kind of stuff. So it makes it a lot easier to go like, hey, everything's in one place. We have one vendor contract, one MSA. We have one support contract. It's just, it's too easy for most organizations. Now, it'll vary based on size, too, because, you know, your small to medium market, when they first start needing a SIEM, they probably already have Microsoft. They have E5 licenses, and it's going to, like— Easily, oh, well, we'll just throw in this, we'll bolt that on, the data's already there, versus I need a parsing engine or something to parse it and send it into a data lake. Like, you start to eliminate all those pieces and parts because it's just there, which for small organizations, already cash-strapped, human-strapped, it's going to be an easy win, unfortunately. Not the best always, but it's easy to do.

So let me ask you a strange question, and I'm going to just like very temporarily argue for Alex's side.

Okay.

It is easier to buy. Everything you said makes sense. So, and the procurement and the corporate structure and one throat to choke, one face to scream at, all very logical. But if somebody shows up and says, you can do it in a piece-by-piece manner, which would be cheaper by a factor of, if they say 50, and that's a completely hypothetical number, nobody says, nobody would sell you a SIM for 1/50th of the price of a SIM for the same data. But then I'm sure you would overcome those problems, right? If somebody says, sure, there's complexity, sure, there's work involved, but it is 50 times cheaper. Would you change your mind? Would you go back, argue with procurement, and do other things?

Well, it never is.

It never is. But I'm testing. Like, if the decentralized is 50 times cheaper, but just as good, would Witter's hypothetical organization buy it?

I mean, do I have the headcount for it? Like, again, it becomes like, I need a— I need somebody to write detections. I need somebody to write parsing libraries. Most analysts maybe can write detections depending on the platform because your query language is the same as your investigative language. And so you're just like easy to set up. But if I need someone to write parsers, kind of that's where the line in the sand for most detection engineering versus incident response teams will draw to be like, I don't wanna deal with that infrastructure, that nonsense? And do I have the people, right, and the skills and capabilities to do it?

No, they, they become data janitors themselves. Uh, it— I feel like, uh, they— and you have to be very realistic about whether, you know, if you have the time to go into fixing all of— or doing all of the mapping, right, all the parsers, and ensuring that you have not only the data into, into a detection-ready format to then have the, to that, which takes, you know, a ton of time in and of itself to then get to the actual rule building part. So yeah, I do, I do see that. I do really see that, but I would argue, I would argue the one throat to choke argument, uh, which I know like procurement loves to see that coming, right? One MSA, uh, one support queue, but it's also like one roadmap, right? You're, you're, you're kind of fully committing to this vendor's roadmap and that's, and I mean, That's, that's going to take a long time. You're just one of many in a—

in—

I've been in a HA a lot, like many hours in my life. And they're, you know, they're really not going to listen unless you're about to not renew with them. And then maybe they start listening.

Wait a second. So, something unexpected happened because I thought that in my completely fake artificial example of the decoupled being 50 times cheaper, Witter didn't just say, well, if it's 50 times cheaper, I'll just buy it. But there was a nuanced answer. So, that to me is kind of not a strong vote in favor of viability of a decoupled approach because presumably even if the price difference is artificially huge in my made-up example, you would still kind of decide between the 2 choices.

It is—

wait, say that again. It is a huge price difference because it is becoming almost unrealistic to centralize all of the data into one place and still support the visibility that, you know, your organization is generating data every year.

No, no. I mean, I meant to say that the license for software, I think that Alex brought up the cost of storage and that changes the math. But if the tool itself, and that's actually an excellent point to add to the discussion, if the tool license itself, like you're buying 3 pieces from 3 vendors and together they cost much less. What would it take you to overcome the resistance of, no, no, no, I want one vendor, one tool? That's more of a Witter question, maybe.

Yeah, no, I mean, it's simple. It would come down to, like, you know, do I have the people to operate the tooling, right? And some, it's, you know, it would be hard, I would say, because you think about what are the largest data sources for most organizations, right? You know, maybe it's my EDR, a 300,000-employee environment, right? And I don't have a lot of cloud. or I'm moving to the cloud. Like, there are still organizations out there that unbelievably are still like on the fence, still have on-prem, still, you know, doing their own data centers or something. And so, where is the source of all that data and pain? Because to Alex's point, like, if I have someone who can write the parsers and I'm getting it for cheap and free, like, I'm now DevOps tied to that forever.

Right.

When my firewall vendor moves a field, I break the entire pipeline and I have to dedicate people to that. Right. And they have to be aware of it and then they have to fix it in a timely manner and, you know, maybe backfill data or whatever.

Mm-hmm.

So that becomes a problem. But if my main data sources, the ones that are most, my analysts use the most, are the easily, the easiest to access, it becomes much harder, I think, to, to make the, the case for it because even though it is cheaper, there's those hidden costs, but that's the hardest thing to like flush out for a procurement team. Like, oh, but it's gonna cost me. you know, 2.5 people to have them. Well, do you really need those 2.5 people? Like, or is that a full-time role? Well, it's half an FTE, but it takes like time to set up and like maintenance and overhead. So there are hidden costs that it's really easy to like be blind to, which have operational impacts, of course.

Well, on the parser ecosystem note, it may not always— I mean, I think we may be missing out the part that doesn't get said enough, I feel like. Because in a, let's say, platformized, you know, large vendor SIEM, they are building parsers for the greatest common denominator, right? The sources that most of their customers have, right? Not these edge cases, not, you know, that custom app that your team, you know, built and have invested 10 years in. And, you know, that they're not going to have a custom parser for that. And The decoupled model is one that leverages data lakes, right? And they are much easier to consume these one-off, bespoke data sources, if you will.

I buy that. I may buy this argument. I feel like it's probably a wash in terms of a large vendor may ignore a secondary data source. And you do have, in case of— in Witter's argument, you do have one face to scream at. You scream at it and the face says, nah, we're not doing it this year. We're too busy with other parsers. So, you may not get an advantage unless you pick the small vendor and you're a large corporation, in which case you would achieve more. But then on the other hand, in Alex's example, you deal with a specialty vendor who is small, who just does parsers, like a pipeline vendor. And maybe you have more leverage with them. Is that plausible or not? And I don't know which side is winning in this case.

I think it is plausible, unfortunately, from experience. I hate to say it. I hate to give Alex points, but yes, she wins on that one.

So, let me also add one more dimension to the discussion because we sort of threw together decoupled and federated. And in reality, at least as I originally thought about this, there is a decoupled SIEM capability and there's federated storage.

Yeah.

So, there is a way, and I feel like there are now more vendors that are both decoupled and federated in the sense that you can buy detection from one vendor, dashboards from another vendor, but storage you buy from no vendor. You rely on your own S3 buckets, your own storage at SaaS vendor-included storage or something else.

Or like Databricks.

Or Databricks here, and then some stuff was already Snowflake. We have a license, so it's kind of free in that sense. So, the idea of federated was to decrease the cost of storage. Otherwise, you are paying for storage. And when I hear this argument from intelligent, well-intentioned people, like, well, you, Alex, I shudder a little bit. And the reason I shudder is that if my auditor shows up and says, give me 12 months plus 1 day of data that you're storing for PCI DSS, I can't tell them, like, it's over here and some data is over there and some of it is in Snowflake and some of it I think was in an S3 bucket, but I'm not sure if it's still there. Like, that's what makes me kind of nervous in this. Is that fair? Wither, maybe Wither first, and then the edicts.

I mean, having it all in one place, you know, everybody's got hot, cold, warm tiers. So I can archive my data and have it off in another place to rehydrate or access at a later date. I mean, that's not usually a problem. And that centralized approach, then I'm not chasing it down. Now, granted, you can have documentation to say, like, this thing, this source is put here, this source is put there. And some organizations, because of the amount of time they have, will still run into that. Like when they're migrating from one giant platform to another giant platform, they're running them both at the same time because my data is tied up in this awesome platform that I pay one vendor for, and I have to now, in order to meet my obligations, start with a secondary license with another vendor so I can start like getting my historical collection there. Which they're gonna be running 2 systems, like, all new data's here, all old data's there. So you will still run into a similar situation, because unfortunately, we all know we're tied into the platform. Once you're there, it's like lock-in city.

I think that migration, the migration story, to me, is a transient story, not ongoing, right? So you're right, like, when we were younger, presumably, and SIEM was an appliance, What you do is you cut the contract with vendor A, keep an appliance in the shed, you don't pay anybody, data is there, you migrate the data, you buy a new appliance from a new vendor, you use that. In case somebody says, but where is the data from last July? You point at the closet and say, over there. And it kind of mostly worked out. But if you are moving from SaaS to SaaS, yeah, you're paying both contracts. And also, the data is—

Oh, my gosh.

Yeah.

So, it'll cost you to leave.

Yeah. This is, to me, this part of the debate, I think, just straight goes to decoupled or federated. But to me, that's a transition stage. That's not— there are no operational advantages. It's advantages during migration, correct?

Yeah. I like to call that, like, renting a brain. So, with the advantage of the decoupled model, you invested in your data strategy that stays put, and then you could just bring, you know, insert your own, bring your own analytics engine on top or scheduler or orchestrator or what have you. And you can do that ideally with, you know, your EDR as well.

Mm-hmm.

In the case if, you know, your SIEM vendor also, your EDR vendor also wants your SIEM business, like you can't really check out of there. You're kind of locked in, in that case. But the, I even think it's still more of a case for, you know, you were mentioning the shuttering of when audits come around and when compliance comes around, you know, are there, is the data gonna be there in the decoupled model? In the case of a data lake, that's usually the case. You're not really finding, you're not really looking for information in your SIEM, right? It's likely gonna be in your S3. And they're very prepared for that since they've been having to do it since they became a public company. So I haven't found that to be a problem in my experience.

Okay. What about the detection speed? I think like in our prep notes, we kind of mentioned that possibly the detection speed argument should go to the centralized model. Because if I have the data, being close to data is better. Data is structured in a way that supports the detections. Witter, can you make a Strong. Can you make a case whether it's strong or no anymore for that, for centralized in this case?

Witter actually is going to win this argument.

I'm just going to—

okay, fine. That's it.

It's an easy one. It's like you go to an all-you-can-eat buffet and all the food's already there versus like, oh, well, I want burgers. And Jane over here, or Alex is like, oh, I like Chinese. And we're all going out together. Alex is going to go get Chinese. I'm going to go get burgers. We're going to sit down in the in the park and have, you know, have our dinner. All-you-can-eat buffet, it's already all there. So it's a lot easier to pick and choose your events, do your correlations. Again, going back to the centralized use case of, if you're using one vendor, like, we'll use Microsoft as the simplest example and the behemoth in the space. Oh, I have them for identity. Events are already there. They're already parsed. Oh, I have Microsoft Defender EDR. Oh, the events are already there. Oh, I have Office 365. Oh, the events are already there. If you think about investigations in most organizations from the corporate perspective, man, your investigations are done, your detections are easy. It's all plain and simple. Then you don't have to worry about the cloud side of the house to do detections on things, which, I mean, you're going to have— you're probably going to be in Azure and then it's going to be just like that much simpler. It's all like right at your fingertips. So that's a tough one to like argue against, unfortunately.

But it's, I think, the only— I mean, if I were Alex, I'm going to channel my inner Alex. I would argue on just price performance argument. I would just make a price performance argument. I would say that, yes, this approach is better. But imagine that if you're paying $1 million to have it as good as you have, how much less would it be to use a decentralized approach? Things may not be as good, but the bill would be dramatically less. Would you accept that? And I tried that in real life and some people said, actually, no, we won't accept that because the quality of data we have is kind of what we need. If we would get slightly worse data at 1/10 the cost, we'd say no. And other people said, okay, show me the numbers. So any reactions to this? I think that your Azure detections in Sentinel argument is solid.

Yeah.

And of course, one vendor probably parses their own logs at least somewhat well. currently updates it. But what if somebody shows up and says, you can get 10% worse and 5 times cheaper?

I think what's not always brought up here either is the results of these detections. So yeah, you have them all in there. You have all your alerts. But are they good? I have a previous podcast episode, and I literally called it, Are Your Alerts Really Catching? Like, do your alerts suck if we're censoring it? And most of the time, the answer is yes. Actually, almost always it is. And I believe that, sure, not only is it cheaper, but also you're getting a better quality of an alert because you can do much more advanced analytics. You can do a lot more complicated joins with SQL and with data science and data, you know, data bricks or data lakes, it's much better prepared for those kinds of things. And they have been. I mean, we just all know that because, I mean, if you look at departments outside of security, they're kind of— it's— I'm, I'm, I've always been jealous that they're so far beyond us, what we can do, right? Like e-commerce, finance, fintech, because it makes them money, right? And, and so, of course, they're going to use statistical analysis and clustering on all this stuff on how to make more money, whereas We could probably do some of these same things too against WAF logs or DNS, but we haven't advanced as fast as those other business units. And I do think it produces a better quality of an alert, rather. And that gets overseen because it's just because of the cheap pricing of it all.

I mean, I'd have to argue the mean time to detect there, though, is different. And if we're talking about data quality, And data cost, what is the mean time to detect, right? Like, you know, there was one 10/60 a very long time ago, and vendors claim different things, but the longer it takes for those analytics to happen and for all that data and information to get in, especially also too with a decoupled system, now you have like 3 things to break potentially. Or, you know, your ingestion is taking, you know, 5 or 6 minutes to get these bulk logs ingested out of S3. Now I'm like loading them into my detection engine, or I'm running my scheduled SQL queries against them because I can't do real time because I'm not as cool as the centralized model. And so you end up with this problem that it may take 10 minutes or 15 minutes, you know, from soup to nuts, from activity happened, not including, you know, coalescing of logs that vendors do or whatever, to the actual like, here's a detection. Oh, that's like 18 minutes old now by time an analyst sees it, or even longer. Or if you're running on scheduled queries, then you run into like, there, there is a There is a trade-off that we can't just say like money is the root of all evil here, and that's the thing that's most important, unfortunately. While procurement would like us to believe it, and we are a cost center, so of course we're not making money, so they want to keep it as— spend as little as possible.

Another thing I would argue in your favor, Witter, is the detection distance too.

Wait, wait, detection distance? In what sense?

Yeah, so you, you brought this up in our show notes, uh, and it's like that's The number of hops, right, that your data has to make before a detection fires, right?

Mm-hmm.

Clearly, a lot more of that happens, more hops, that is, has to happen in a decoupled model, right? You have to pick it up from, you know, your pipeline player, then you have to put it into the— it has to land in an S3 even, and then maybe Snowflake can pick it up from there. That's already 3 hops right there. And then, by the time a rule gets run, you know, against the disk. So, there is also that part of it, again. But then, the offset is, is that gonna get you a better alert that's worth looking at? Probably, yes.

Actually, I think this side is very clearly in favor of centralized, but there's something else that came up in this discussion that I wanna pick I think the topic of real-time or near real-time. And this, to me, is an interesting question because ultimately, when the SIEM was a big database on-premise, a lot of obsession was about real-time. Can you do it in a second? Are the events delayed? And a lot of this was kind of sacrificed a little bit when SIEM became SaaS. So, real-time started to be understood as maybe seconds, maybe minutes, I don't know.

Mm-hmm.

With the decoupled and federated model, when some APIs are kind of best effort, and my security component of a SIEM calls some storage, and that API goes into a queue, and the SaaS platform that has the data decides to give you the data in an hour, like, how bad is it? Like, can you do real-time in any kind of decoupled and federated model, or is real-time gone?

I would say, like, it all depends on data source, right? You know, VPC flow logs are coalesced in, you know, 10-minute intervals or whatever. So it's not, it's already not real time. So, but if I can ingest it quickly into, you know, my, my, if we're saying that, like, if it was Azure and we're, you know, to my defense, if, you know, it's 10 minutes till it's coalesced and it gets dumped and then I ingest it immediately, that's 12 minutes or something like that. And now I write a detection. It's still not real-time, but it happened in 2 minutes' time. Like, I got the data immediately and I did the thing. The only time you're gonna get anything near real-time is if, one, you're streaming on ingest and you have a streaming detections platform where data's coming in.

Yes.

I immediately compare it against, you know, different libraries and information and kind of like do some analysis there and write some initial detections. But also at the same time, to our defense for scheduled queries and kind of that data lake model, being able to look back over, you know, if you think of adversary activity, well, I want to find like these 3 things that happened in a 10-minute window. Like, that's important. That's not real-time, but that's a solid detection.

Yes.

That I wouldn't want in real-time. Like, don't give me 3 individual alerts because they're meaningless, but together those 3 are like very, very important. And you think of like, you know, recon activity under something.

But if the 3rd one comes tomorrow morning, That's a failure. If the third of the 3 comes tomorrow morning with the correct timestamp, to me, that's a failure. I can't really chalk it as a success because—

Oh, absolutely.

I agree with the real-time, like, the sub-second real-time is probably largely gone. But that type of, like, human-scale real-time depends on the data sources is a good answer. But practically, can you deliver detection value if we want to try to act in minutes? All this recent talk about vulnerability apocalypse implies that maybe you have to respond very quickly. If you have agents, they can also, in theory, respond on the minute time frame. But if the logs show up tomorrow, your 1-minute response has failed. So, Alex, do you have anything that makes the decoupled and federated SIEM deliver in this case?

Well, let's not forget about the Some of the limitations of the, uh, when detection happens at the time of ingest.

Mm-hmm.

Oftentimes it's, uh, it's, it's not that many. I mean, Splunk has a 200 query concurrent limit on how much you can, you know, keep, keep doing and keep stats, keep staggering on a concurrent basis. And, uh, there you have to look at, I mean, and of course that's, that's, that's the detection engineer's job to see how you should be staggering, you know, your searches or orchestrating the scheduling of it all. But the limits really get you, especially for an organization like the size of Dropbox. Like, I'm sure you have thousands of detections that are super, super priority. Are you reaching your limit, your max limit of real-time detection? And that's something that we, you know, you kind of have to stagger and prioritize.

Mm-hmm.

In the case of like thousands of detections running at the same time, that's going to be at the enterprise level. But maybe for a smaller organization, it might be okay. You might be okay under that 200 limit.

You know, I think it's going to depend too what your vendor is providing too sometimes, right? Like, you know, EDR, if I'm only worrying about what the EDR vendor provides, then, you know, there's a lot less detections I'm going to write. I'm only ingesting the alerts. I can act faster because they hopefully find it immediately, right? It's not retroactive. There may be some that are retroactive, but that's going to be a problem. And you will— you will hit limits depending on size. And it goes to your earlier comment about quality of alerts. Like, if there's thousands, are they really working? Are you able to validate that they're good? You know, people have thousands of rules.

Yes.

Did we just not see that thing? Like, you know, there are then— and some of the, the like larger platforms actually have, um, caps, which I have run into before at other places, um, where like during the pandemic when you couldn't get hardware, it was a SaaS platform, they're like, hey, actually you're capped right here right now because we can't get enough hardware to continue to scale our cloud operations. And if you think about that, that's just crazy. But also It's not, it wasn't a pay-per-drink platform. So like, you know, the Sentinel, oh, an alert fires, that's $0.001 every time it fires. Like your warehouse costs are sunk. I buy a warehouse, I use it, I run my alerts or detections in my data lake against it. That's like one operation. And so I don't have to worry so much as long as I keep it like utilized and it doesn't get overutilized.

Hmm.

I think most of it to me is absolutely credible. Yes. I feel like this is, you know, if you pay more, you'd get better performance with a centralized system. It's kind of correlated. Like, if you want faster queries, buy more cloud, buy more hardware, whatever you're doing. So that to me is, that success is probably not obvious to replicate for the decentralized. But let me, given that where we are on time, I wanted to bring up the elephant in the room, which is of course AI.

Yeah.

Hmm. Does either of you have an opinion whether the current use of AI in detection response, security operations, SOCs favors the centralized or decentralized? And this is like an important question, so I should probably do a drum roll. We'll figure out whether our editors can make a drum roll happen. But do you think AI agents in SOC and DNR favor centralized I don't know. Don't look at me.

I would still argue they favor decentralized because the storage layer is abstracted, right? And it's not— and oftentimes, well, versus the monolithic approach, the storage layer behind your SIEM is not often reachable by anything else. Uh, and just coming out of BSides SF, and it was, it was a while ago at this point, but, uh, coming out of BSides, there was an amazing panel about how, uh, well, one, I think we're on the same page about, you know, you should never DIY your SIEM or vibe code your SIEM. I think we all know that that's, that's just not possible. But then the second thing was, uh, now customer, or, you know, prospects, customers, they're looking, they're now only considering products that have An MCP now. If their Claude can communicate with it, if their tools and their systems, the way that things are communicating is now fundamentally changing. And that is not possible in a monolithic model. I mean, unless they're coming out with MCPs, but I don't really see that happening yet.

Hmm. I've, you know, if you think about how AI likes to work, context, and, you know, the less things it has to ask and query, while the centralized model, you know, maybe it is one MCP or you're hitting up a series of APIs, right? All the data is in one place. Uh, it's probably a lot likely to get the data in the same time frames, properly correlated and formatted and served up onto you on a platter. Now with the like decoupled model also, it makes it like one of the problems with decoupled or potentially if you're running, uh, you know, something more like that, where your analysts have more work because they have to touch more tools or things potentially. Like, oh, I need to go here to search for this historic data. I need to go here to search for this, like, you know, other data.

Mm-hmm.

It's all centralized. They don't have to worry about that. But the problem is that the AI takes that problem away from the analyst and like kind of gives, uh, points to the decoupled model where it's like, oh, it's not important anymore because I've abstracted that away from my human beings. Because they're just going to interact with their AI and say, hey, I need all the logs historical, you know, within this time frame. It goes, cool, I'm going here for that date, I'm going there for that date. And then they present it up on a platter like, and here's what I think about this, uh, you know, incident or investigation, and make it super easy. So I'd almost call it a draw, but if you want to go down to token burn and money, I'm winning.

Fair, fair enough. Fair enough.

I, I— you guys are chickening out. of this very critical question. And I think both of you are just committing this— well, I guess Alex was a little more committal, but I didn't really hear a really good argument though, because what if the agent goes to some kind of faraway land for data and comes back with nothing? Somebody would have to engineer around that. And it's a fair bit of engineering. With centralized system, it would be very clear. I need DHCP logs. I go, they aren't there. I tell the system that to make a decision, I need the HTTP logs, they're not there. But you have to engineer the same thing for decoupled. For all sorts of failures, you have to kind of decide what to do when you don't have the data and whether you can still do your job of an agent, in this case, without the data being available. Or maybe it's available in the morning, but not in the afternoon. Maybe now it's available in 3 seconds, and in the afternoon, it's It's very busy. Like, that's what makes me nervous. With a centralized system, I say I need this data, I either get it or I don't get it, and I know what to do. With decoupled, it's more nuanced.

So, you're saying it's more prone to hallucinations if it's decoupled?

More prone to— it's harder to engineer so that it doesn't make conclusions off very incomplete data. That's the one thing. one narrow argument is that—

But isn't that what the MCPs talking to each other are supposed to figure out?

Kinda yes, but also no.

The thinking of the agents is, you know, your overeager intern who may be naive about the impact of that thing.

Yes.

And they just want to please you, Alex. They want you to be so happy about everything that they presented to you that you're just be like, woo-hoo, pat them on the back and send them on their way. Um, and I think like, but that's a, that like engineering case in environments where you're building it, those are guard— those are simple guardrails. I mean, we have them. Um, we're using agents in our environment and we've, we've run into those problems, right? Where it like created a situation or a scenario, but because we have humans involved who are looking and validating, like early on in our experiments, we were able to easily say like, okay, no, you need to validate and go back and do these steps, like, you know, counter your arguments, come back to me and say like, well, I think it's this. Well, prove that it isn't that. And, you know, flip it around, which has been very interesting. But occasionally, like, I'll say, since the English language is so nuanced, it can put the same sentence in, it'll be wrong if you rearrange the same. And it's come down to 7 words in a conversation I was having with the Nouns. Like, no, no. If we take this word and move it here and take this word and move it there, like this sentence is 100% true. If we take it as the agent gave it to us, it's close, but not.

It's a completely different sentence.

Exactly. Or even a different—

yeah.

So, and, and, you know, when you're doing analysis or when you're allowing an agent to make decisions that could cost your organization millions of dollars in breach costs and GDPR fines and violations, like you don't play with that.

Okay. Oddly enough, you both convinced me that it is decentralized versus centralized and agents doesn't actually call to either side. I was secretly believing that agents favor centralized, but now I'm shaken in this belief. So, I think it depends kind of an answer rather than centralized is better. Okay. So, given where we are timewise— Well, I think— I want to get to our favorite traditional closing questions, which again, Tim usually does. So, usually, we ask all guests for one tip and one piece of reading, a book or resource, or as Tim doesn't tire to remind everybody, it can be anything except for my blog. So, I'll repeat the same argument. So, would you care to provide one tip and one piece of reading?

Sure. You can go first, Twitter, because I need to pull mine up so I can have it right.

All right.

Well, as I say, I have the book that everybody should be reading. If you're into threat hunting and Mac, you can't go wrong. It is like, you know, it's sitting here. I bought— everyone in my team got a copy. We're going to go through it one by one, chapter by chapter, and kind of learn because there is no resource like it out there. So Threat Hunting: Immaculate by Jaron Bradley is awesome. Funny enough, I didn't even know it just literally sits on my desk as a reference. One tip would be if you are exploring using AI, to our point, like evaluations are not a joke. Science is important. Changing one word in a prompt can impact the outcomes substantially. And so testing and validating and not just kind of like laissez-faire, oh, we're doing AI, we're using agents, like having a prompt, having the output and having the model and keeping those consistent while you test and validate is super, super important. Like you can experiment and play around, but if you're gonna be serious, you gotta be documenting this stuff and actually testing and retesting and having datasets, you know, and not making changes to get to where you need to be. And then like, okay, we're not quite there.

Yeah.

We get it where we want it. It's getting good, but it needs to be better. And when you're adding different things, you need to go back through, okay, we just changed the rule. Like, we just gave it a new guardrail. Test everything again. Because one simple word— and I'm talking from changing the word from I want to find malicious to I want to find bad— will impact the outcomes. No joke.

Perfect. I love it. I really appreciate it. So this is very useful.

Alex? Well, on the macOS front, I really would love to spotlight a content creator, another one that is writing a lot of research in this space too, especially on the detection engineering front, is Olivia Gallucci's work.

Oh yes, I've seen her speak at Undaunted. She's awesome.

Yeah, she is. But then more of a tactical resource as well that I have up every day, and I'm using as I'm building out the detection engineering library at Scanner. is the— is, it's work from Andrew Van Vliet, uh, library.tired, uh, slash labs. Uh, it's fantastic technique research report library. Uh, and it's, and it's, it goes beyond, I mean, your atomic, you know, point detections. This is like full, deep, like you can't find any, any CTI, you know, content on there. 'Cause a lot of the times, I mean, these CTI blogs out there are really just trying to sell you, you know, their, their product and just really filtering through, through that is, is really tough these days. Especially, and it's like, it's almost like, do I even want to use my tokens for it? So I find, I find these, this, this, this work by Andrew Van Vliet, and it's entirely because of him. And I know how he works, and he does fantastic work and research and building very thorough and resilient detections. So I'll drop that for you guys in the show notes.

Perfect.

Awesome.

That was really fun. I kind of wanted maybe a little more pillow fight energy, That actually worked out really well. So thank you for the nuanced takes. And I am gonna— we were gonna wait, we're gonna have Tim listen to it. And he decided who won.

You wanted a pillow? Well, yeah, I mean, you guys do argue a lot on this podcast.

We do occasionally.

Witter, he seems like— are you a girl dad, Witter? Because it seems like you're a girl dad. No, no, because you're just so kind.

And I'll be honest, I could have argued either side. And so, like, you know, it is, it is to each their own, and everybody is different, every environment's different, and it'll make sense for different, different people. Unfortunately, like, that's, that's just the way it is.

I did want to argue, I did want to argue that this, this take that I saw recently that I haven't yet to talk about anywhere, and I don't know if you saw it, Anton. Some, some person, I won't say the name, but they were, they said that Detection engineering shouldn't even exist, and it only does because SIEM has failed us. Do you remember this by any chance?

I have not seen it, but at the risk of extending the podcast beyond the 45 minutes, which we swore to never do, I feel like I've seen, if the take is the same, that I've met people who say, I don't want to engineer detections. I just want to consume them. You're the vendor. Here's money. Give me detections. That take I've seen. And in a sense, at smaller companies, it's probably correct. Like, they're not going to engineer detections. They don't want to engineer anything. They don't have engineers.

They have people.

No, right. They have a little bit of money, and they're willing to find somebody who would take their money so that the threats would get detected. That's how we have EDR, and that's how we have a lot of other tech. But to me, that's not a detection engineering failure or SIEM failure. It's just that, to me, there is a category in the market where detection engineering is not for them. They don't want to engineer. They don't have engineers.

Yeah. Well, I mean, they're still doing themselves a disservice not building custom detections. And I'm so sad that Tim isn't here because I believe the analogy that was used, and he's like the king, Tom Brady of analogies, was that nobody buys a car expecting it to crash. Uh, and like expecting you to do all of this work that you have on your car, otherwise you're going to crash. In the same way, nobody should, nobody should be expected to build and really fine-tune and customize their car. Otherwise it, you know, it's your fault.

Separate debate. Let's just, I'm just going to abuse the host privilege and say, stop. This is really fun. We got to discuss it at some point later. No, we got to discuss it some other time. Thank you for being on the show. Tim would tear me apart for 45, 47 minutes of recording without him.

So.

Again, thanks for both of you. This is really fun, and I think the energy kind of went to lively, friendly energy, not the debate energy. So thanks a lot.

Thank you.

Thanks a lot.

Security.