EP197 SIEM (Decoupled or Not), and Security Data Lakes: A Google SecOps Perspective

Security.

Hi there. Welcome to Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior PM for Risk Analytics and Cloud Threat in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website. cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on our community page. Anton, we are talking about future technical directions in SIEM architecture. Is that right?

Correct. But it's also— you make it sound kind of dry.

Oh, I couldn't make it sound any drier if I tried.

Well, yes, there is that. But it's also actually a very juicy subject. So it's not dry by nature. It has to do with 2 massive forces clashing in the market of SIEM.

Wait, that sounds exciting.

Yeah, see, see, I made it exciting. Yeah, I made—

Okay, what are the 2 massive forces? Say more.

One massive force is the argument that SIEM should be disassembled or cut into pieces, into maybe in smaller chunks. And the other force is that SIEM should be integrated with more things. It should have not just Yuba, as you well know, but SOAR and possibly even EDR, and SIEM should absorb XDR and also cloud detection. So, there's this disassembled, decoupled SIEM force and the more tightly integrated SIEM. And people on both sides are convinced that they're right and the other side is just completely flat out wrong. It reminds me of some politics in some country I've heard about.

Huh.

But it's real. And that is why a technical dry look at this is very important.

So, this is a battle of less is more versus more is more?

No, I wouldn't quite put it this way. It's the battle for I assemble it for you, you use it.

Specialization versus integration?

Better, better.

Yeah, yeah, yeah. Well, maybe with that and this battle of 2 massive forces, let's turn things over to today's guest. Today we are joined by Travis Lanham, Uber Tech Lead for SecOps Engineering here at Google. Travis, thanks for joining us. It's always a pleasure chatting with you. We recently had a pretty big news week about 3 different SIEMs. And that's had a lot of discussion in people's minds about the future of not just those sims, but the concept of sim. And so we've had sort of two sides to this. We've had people who say sims are going to become disassembled or people who say sims are going to become super assembled. What's your take?

Yeah, thanks for having me. I think it's a great question and I think it kind of starts with what's the value that folks get out of a sim today, right? You know, originally this value started around compliance reporting, right? Put a bunch of your logs in here and we'll get you that checklist of whether you're meeting certain compliance regulations, right? Then I think this kind of evolved into security practitioners jumping into the tools and getting value out of SIEMs for forensics, right? Okay, I know I had a security incident. Maybe my antivirus told me something looked bad and I want to go and do a more thorough investigation. And then I think on that maturity curve, there was kind of the next advancement around actually using SIEMs for detection. And I think this is really where SIEMs are still today, right, in trying to prove out value around detection. And I think if you look at what you need in order to provide value from detection in a SIEM that's not in a control plane, right, and being able to kind of aggregate this global view across all your different control surfaces, whether they be endpoint, network identity, etc., is around how can we bring data together? How can we have that, you know, in a common format, have that in something that detection rules can run over, and have that be something that can be queried and, you know, metrics can be driven out of to be joined against threat intel and kind of all these use cases. And so I think that really comes back to if that's kind of the next value prop for SIEM, right? And some organizations achieve that to an extent today, and a lot don't. Then I think it comes to this question of, okay, where does that data need to live and how does it need to be operationalized to achieve that goal? And I think that brings this into kind of an interesting set of questions where today I think a lot of those use cases require all that data to be in one place. That data today lives in many different places, which is why I think this topic has become very popular. And I think there's this open question on, hey, in the future, Is this kind of going to be a lot of data still needs to be centralized? Can some of it run more at the edge of where your data lives? I think those are kind of the key elements.

So help me understand, you're proposing a world where you can achieve centralized visibility without centralized data.

I think that's the goal of this kind of disaggregation or like shimming on top of other data lakes approach. I don't think that it's very effective today, right? A lot of why folks are trying out this approach is because the number of control points that they have generating logs or generating telemetry or visibility into their environments has grown a lot. And their security teams or their operational management teams haven't been able to scale out their kind of centralized visibility as quickly, right? And so often kind of falling back to that second value add, right, around forensics, right? Often, you know, what we'll see folks trying to do is, Hey, I need to go and do forensics in this cloud environment, and the logs for that cloud environment live there. And that might turn into, okay, I need to run some queries against that data, or I need to pull that data into my SIEM and then run some queries there. But I think it's still kind of stuck in that forensics—

Hmm.

—dirty, right? It's still not getting into that true, you know, vision for SIEM as being this kind of central detection point where you can really get a lot of metrics that'll help you do kind of more threat intel-driven approach to detection. that'll help you, you know, ultimately become what I think that first compliance milestone was around, right? Right. Where compliance started out as this reporting, and then you had the forensics. But then, with the detection, you could also potentially get into this, you know, a future state of the SIEM as being this kind of governance control point, right? Where you can not only have compliance as a reporting capability, but you could have kind of this thing that SIEM is naturally positioned well for because it has all this visibility, which is to actually have Hey, here are the compliance controls that I want to enforce across my environment. I have all the monitoring visibility that's telling me, am I meeting those or not? And then I can hook into kind of the remediation side or the prevention side and actuate out in my environment. Here are the controls that need to be enabled, you know, 2-factor authentication, you know, binary denial listing, whatever it might be, right? And kind of have that, you know, be that value add of having that central view over the data. And so I think it's like today, a lot of that requires, having all that data in one place. And I don't think that, you know, there's really been an effective way to get that central point of visibility without being able to put your data all together.

But wait a second. That to me doesn't automatically mean we have to decouple. I mean, in theory, you can build a platform that combines high-scale storage. Oh, no. I guess I'll cut off myself in the middle of the question because if you're doing it federated, you probably can't really truly couple it, right? So that's the argument.

I think there's one kind of piece that's around things like performance and like, can you really do a lot of these powerful metrics across, you know, all your different visibility points, et cetera, right? I think there's another piece of this which is really around whether if you have different data sources in your SIEM today, and, you know, a lot of people have kind of tried different approaches to normalization, like, is that normalized format able to give you something that I can ask a question and it goes across all my log sources and gives me the answer? Right? Even a simple question like, have I seen IP address X in my environment, right? That requires doing that kind of basic level of normalization that if you have a SIEM today and you have multiple data sources in your SIEM today, a lot of folks aren't even getting to that kind of maturity level of, okay, I have even these most basic parts of those security logs parsed into a consistent manner, right? So I think that's almost— if we're not doing really well at that problem today, in the industry, then trying to do that across a federated approach where all the data isn't even in the same system, I think becomes this exponentially more difficult problem.

So wait a second. So people who insist on decoupling or sort of disconnecting the security part from the data storage part, what are the upsides? Do you have compassion for these people or do you think they're wrong? Or how do you think about— I'm not trying to lead you to an answer. I'm just legitimately curious because I see those industry debates when people say, no, no, no, no, no. Now we have this amazing storage. We can just build security, plug it into the scalable storage, whether it's BQ or something else, and we'll have a SIEM better than anything that came before, even though it's technically decoupled. And these are tempting, but I also have these flashbacks to the old times when, sure, you can buy an Oracle database off the shelf and then build on top of that, and then you would have a decoupled SIEM. That approach we abandoned. So I'm schizophrenic about this, torn, really torn about this.

Yeah. I think there's a lot of compassion and understanding where folks are coming from in this, right? Like a lot of folks are coming in, your attack surface has expanded much faster than you can onboard monitoring of it, right? And so you have all these kind of pools of control points and infrastructure that need to generate visibility, and then you might not be able to onboard all that visibility centrally, right? I think this is also one of the promises of cloud, where as that surface becomes more and more homogeneous, monitoring that surface becomes easier because there's kind of this thin waste for the monitoring layer. So I think there's definite compassion around why people are in this spot today. You have all these different places where you're running infrastructure. You need to get visibility into those places. There's kind of this natural, hey, I want to be able to do a federated query for forensics, or I want to be able to do something.

Mm-hmm.

But I think in order to really move up that maturity curve, right, you have to say, okay, I need to get this into a consistent format. I need to be able to run my queries across all this data. I need that to be performant. I need to be able to, you know, generate materializations on top of that. I think ultimately, you know, you could maybe see kind of a hybrid approach in the future, right? Where you have this kind of centralized point where a lot of your monitoring and visibility ends up, and then you could maybe push down certain operations into kind of those federated experiences, right?

Mm-hmm.

Particularly for logs that are of lower value, right? where you could maybe do things like, hey, we'll generate some metrics across, you know, this kind of data silo in one cloud, right?

Yep.

And we want some metrics around things that would be useful from a threat intel perspective, right? We want, hey, what are all the unique IP addresses that we're seeing in these logs? What were all the unique users? You know, how often were they accessing things? And then ship those metrics into a central point of visibility, right? Or push down, you know, threat intel matching into kind of one infrastructure environment. And then aggregate findings into one consolidated view. I think some of those may kind of come out in the future, right? I think that we're still pretty far away from that today, right? Today, it's still mostly around that kind of forensics use case, right? Around, hey, I've got all this data in all these different places, all these different clouds, and during an incident response, I need to go and suck it into one place or be able to go and do that distributed grep, so to speak, across all these different logs.

I'm sympathetic to the desire to want to be able to do distributed grep and not want to pay for centralized ingest all the time. That kind of makes sense to me, but I also feel like if cost wasn't an object, you might not want to go that direction for all of the other reasons you might not want to go that direction. But like, thinking about it, and you've been at Chronicle a while, or sorry, SecOps. Well, you were at Chronicle for a long time and you've been at SecOps for a little while. Tell us about why we haven't gone this direction as Google Cloud. Like, why do we persist in this? central and consistent so that you can do easy, like, I want to find this IP. That's a single-line query. Why haven't we gone this other direction?

Yeah, I think for us, kind of the motivation starts around where Google has come from solving this problem, right? And where we've come from solving this problem, you know, even in response to Google's own Aurora attack a decade ago, right, was we really need to get centralized visibility into all these different surfaces that we have. And once we have that centralized visibility, Now we'll be able to not only achieve kind of that basic compliance capability, not only that forensics capability, but actually be able to go and innovate in that detection capability or that higher-order capability. And so where we've come in delivering the product out to customers is trying to get them to that same point, right? And this comes out in a lot of different ways in the product where we do things like threat intel matching and retroactive IoC matching, where we have, you know, detection content that goes and runs over all of your data. Right. In order to do that, we really need this data to be in one place. We need it to be in a consistent format. And so that's been a big area of focus for us initially. I think kind of looking to the future, right, I think there are kind of all these areas of expansion that we can go into, right? And we've already done some of this with SOAR, for example, today, where in SOAR you can kind of do this federated forensics use case and tie that back into SIEM very closely and get the threat intel view on that data. And so I think there's kind of been like the first steps toward it. But I think it still comes back to this. The hard part is around having consistency in your data, right? Having a consistent view of the data where you can now do analytics, right? And then with that baseline of analytics, you get detections, you get threat intel kind of being a proactive view. And then ultimately, I think you'll be able to get into this compliance++ of the governance, right? And be able to get into this, okay, we have this great view of visibility. And now we can actually start actuating the policy that we want to enforce in our environment.

So wait a second. I think that that approach makes sense, but you can also be decoupled and still do exactly that, right? In theory, you can have the off-the-shelf commercial data store and then assume that it's there at the client site and then just build the security layer. So presumably when we decided to not do that, we saw other upsides of a tightly coupled data store plus SIEM plus security experience. So maybe you can talk about these, like what are the other reasons but consistency? Because you can achieve consistency with Deque or whatever as a data storage backend. We didn't do it though.

Definitely. So I think there's kind of that baseline, right, of like why kind of start with this approach, right? And it gets a lot of those pieces like, okay, we have central consistency over the content, we have central consistency over the parsing and the detection rules and the metrics they're generating, kind of all those pieces that are core building blocks of the platform. And so I think that's, hey, the easy way to get that, right, or the way to get that in a more comprehensive fashion is where you can have end-to-end testing over all that content, you can have this kind of very cohesive story where it's all integrated well together, right? So that's kind of the starting point, right? And then on top of that foundation, right, we've really built out a series of best-in-class capabilities for operating over your security data that just wouldn't be available in an off-the-shelf database or, you know, a Parquet file set that, you know, folks are putting their Databricks instance on top of, or, you know, putting in some data into Snowflake, right? And so a lot of those capabilities we've driven around are, how can we bring Google Search to your security data, right? How can we kind of get out of this method of having tons of indices or these folders of data and provide this kind of, hey, I just want to look at this alert, I want to put in an entity in my search, I want to see everything that we have around that, see the threat intel overlay on top of that.

Mm-hmm.

I just want this kind of unified investigation experience and make that as easy as going and doing a Google search, right? And in order to do that, building on kind of Google's great foundational infrastructure, we've built a bunch of application platform infrastructure on top of that in order to support things like real-time search, real-time detection at unmatched scale.

Talk about the aliasing, because to me, that's one of those things that I always forget that Chronicle does. But then when I use the product, I'm like, oh my God, this makes my life so much better and is totally not apparent if you were just trying to shove it all into a Parquet file. Tell listeners about that real quick.

Yeah. So with the aliasing, what we've tried to do is kind of solve this problem that comes up all the time in security data, right? Which is each of these control points doesn't have perfect visibility, right?

Gasp.

Yeah. Shocker, right? You can have a network control point that has an IP address logged, right? You can have an endpoint that has a hostname logged, right? And how do we stitch these 2 together, right? How do we tell a comprehensive story about what's happening in your security environment? And in order to do that, we built out this kind of aliasing or enrichment or data fusion capability where we say, hey, we want to ingest all the context we have about your environment. That could be things from DHCP sessions, from telemetry, where we can stitch together sessions that correlate IP addresses, hostnames, MAC addresses, et cetera, This can be ingesting data from your CMDB solutions around assets or, you know, your endpoint visibility to give us information about your assets, cloud inventory services. This could be things around users and your HCM systems, your directory systems, permission access management data. We take all these different context sources, we combine them into this graph where we reconcile all the entities, we create this time series version of all the entities in your environment where we understand Hey, IP address 10.1.2.3 corresponds to hostname, you know, Tim MacBook, right? And we have this kind of comprehensive view of what the entities in your system are. And now we use this to enrich all the logs that come into the system. And again, this kind of being a place where we really need to, you know, have consistency in how the data is represented and where we leverage that kind of centralized view of the data to provide this enrichment that wouldn't be performant to do at read time. And provide this capability where I can ask a question around an IP address and get the full story, right? I can get all the logs, whether it's that network log from a firewall, whether it's that endpoint log, and get that all together in one picture, and then be able to see that in an investigation, to be able to run detections against that in a consistent way, to be able to actuate response actions using all that data, right? So when you're operating over data in SecOps, You're actually operating over this really enriched or unified view of your data, right? Where you can say, hey, I don't need to be having all these conditionals in my query.

Imagine the joins to do that. Imagine trying to do the joins for, here's my DHCP data, here's my endpoint data, here's my network data, here's my threat intelligence. What the heck kind of world would that be?

You know, people have done this with a lot of lookup tables that have eventually crashed the web.

People built the pyramids without the wheel. That doesn't mean it was easy.

Right, but that's the same exact example. Like, you can— the first one required slavery. They didn't have slaves, they just had security engineers who were very overworked instead.

Fair.

But the lookup tables, and a huge number of lookup tables that are manually updated by humans to be just right, and when you need the query, it just occasionally worked out exactly what they wanted. Not every time, obviously, but yeah, okay, I'm painting myself into a corner. It's a horrible system. But when you have to do it manually, it's a horrible system.

And what you're describing is magic. And it's one of the, like, just foundational aspects of the system and not one we usually talk about. So thank you for describing that, because when I, when I learned that, I was just like, yeah.

And then, like, as you kind of gestured at, right, we take all that enrichment and then that becomes foundational in all the metrics that we derive in the platform, right, on when's the first time this asset reached out to this domain. Right. And you can see that from all the different kind of indicator views of that asset, right? The IP, the MAC address, the hostname, etc.

Right.

All the things that we do around correlating with threat intel, right? Kind of all those pieces build on that enriched foundation. And so we've had customers, for example, that have been able to achieve dramatic noise reduction in their alerts because, you know, I'll use an example where it's, hey, I wanted to alert on any time someone downloads a service account key from my cloud environment, right, that's not part of the SRE or IT team. And before, for them, that was kind of this complicated mix of what Anton was talking about, right? Maybe I have some lookup tables, maybe I have some things that happen in SOAR, right, as kind of these enrichment playbooks. And it's hard to keep all of this consistent. It's hard to have this kind of— now you're responsible as a security team, not only for doing this, okay, I need to load that data into this place. I need to keep that data consistent for here are all the people in my organization, here's their job role, et cetera. Right? But now I also need to keep that consistent with my detection logic. And so now you've turned the security engineer's job into, hey, your job isn't to investigate threats and, you know, look out proactively for the risk of the organization. But your job is to be this human extract-transform-load pipeline of your CMDB data into your security system, and then try and maintain consistency with all these playbooks. And kind of, it just becomes this complicated mess rather than just doing it in kind of this, hey, you just bring your data in, all of this gets enriched together, and we present this unified picture, right? And I think that kind of, again, connecting it to the beginning, right, is a lot of these kind of capabilities Google search over your security data, always-on enrichment, you know, detection at really unlimited scale. These really wouldn't be possible unless you have a platform that's built for security rather than, you know, just taking some Parquet files that are in some cloud object storage.

So I love this. I want to get us back onto kind of the original bent of the show and not just have it be a smuggled SecOps ad under the banner of talking about decoupled SIEM. Anton has one more really interesting decoupled SIEM question.

Yes, correct. I think that it has to do with one of the lessons from Google, sort of like how innovative products ultimately require, or the best innovative products require technical innovation. So they need to be an invention under the hood. It can't be the, oh, we'll just speed it up a little bit, or we'll just do this. So in your mind, if you're thinking about SIEMs that are trying to decouple from data stores, What may be the technical innovation driving them? Like, I'm a little puzzled here.

I think that that's kind of one of the fundamental issues, right? That if you're trying to be this shim layer, right, ultimately a lot of the value that you're providing is kind of interface at the interface level, right? Or it's ultimately at kind of the content level, right? And trying to stitch together this kind of different set of data silos rather than providing a really innovative processing capability on top of it, right? And so it's fine as kind of this, hey, I want to be able to see all these different parts of my environment, right? Or I want to be able to extend into these places where I don't have stuff set up, right? And I wasn't bringing my logs into, you know, the place that I want to, right? But it's not really getting to that, how do I have amazing search over my data? How do I have all these capabilities that allow me to, you know, have better noise reduction, to have better prioritization, kind of all these more mature capabilities up the maturity curve. And I think this is also why folks who have kind of gone down this road, right, of, okay, yeah, I'll just keep my logs in cloud object storage and I don't need to really worry about bringing them into a SIEM. It's like ultimately when they find themselves in an incident, they find themselves, okay, now I need to pull these logs into my SIEM, right?

Yeah. But then you're cheating. Then you're not really doing the decoupled approach. You are hiding some of the complexity and you unpack the complexity at the worst possible moment when you need the solution. So, that's not fun at all. What about the argument that allows the security people to deal with security and leave the data storage to data people? Like, the argument that if I have a team of 20, I can dedicate 20 to security and have the storage to be magically done by somebody else who I pay or who my clients pay. Why does this argument not hold water?

I think this is the argument for not going in the kind of decentralized version of this, right? Or not going in the disaggregated version, right? Because if you have that kind of centralized point and you have someone who's responsible for the storage, right, then your security team doesn't need to be worrying about that, right? If you have this kind of, okay, I have all these different data silos, right? And then you have the incident come through. Now your security team is often playing that data infrastructure role, right? Or again, kind of at the worst possible time, you're throwing in a whole nother part of the IT organization to try and help the security team. Right. And now you have not only the security incident itself and the coordination that the security team needs to be doing with all the folks, you know, they're either impacted by the incident or, you know, that are the trigger of the incident or whatever, right, or the exposure of the incident, right? But now they have to be coordinating with this team that's actually just trying to like get all the visibility into one place.

I have been trying since Anton asked that to think of a suitable metaphor, and I have failed, which maybe means, sadly, It's time for the end of the episode. I have to ask our traditional closing questions, Travis. One, do you have a tip to help people think about the future of SIEM and which way they should go as an org? And 2, do you have recommended reading? Because we hate to leave listeners empty-handed.

Yeah. So I think the first one is really kind of thinking about that maturity curve, right? And where are you on it, right? Are you doing kind of checkbox compliance? Are you doing kind of more around the forensics, or are you starting to build out detection capabilities? Are you starting to build out, you know, and I think this is even more forward-leaning, kind of some of those governance controls, and really taking a risk lens on top of your security data, right? Because ultimately, this is where organizations are orienting towards now, right? Where it's, hey, I've gotten all this IT infrastructure, security's top of mind, and I really need to be thinking about how do I prioritize and manage the risk of my attack surface, right? And so I think that's, you know, where we see a lot of folks putting a lot of energy around today. And then for required reading, I think There are a bunch of different sources.

Suggested reading, not—

Yeah, yeah. Instead of frequently asked questions, you have frequent demands, kind of that logic.

Yeah, recommended reading. I mean, I think there are lots of great blogs out there. The things that I find actually most interesting usually are just when folks post an incident report, right? You'll have Cloudflare, you'll have Snowflake or whoever, right? Posting an incident report or getting a threat report, whether it be from Google Threat Intel or some other source, right? I think those are usually kind of Always interesting, right? Because you really kind of start to get into the heart, especially for the ones that are quite well-written. Okta is another example, right? Where organizations who have done a service to the broader community by really writing up, hey, here was our experience, here's things that you should be looking for, right? And here's kind of a view on how this happened to us and how you can think about preventing it for yourselves.

I think that's such a great way of looking at an after-action report of Here's how you can avoid having this happen to you.

But before we go, before we finish, we managed to run this whole episode without once mentioning the security data lake. I just am curious how they connect to this whole thing because you see people enamored with the concept, but you don't always see people who understand what they're talking about, right?

I see. Okay, Travis, what's the security data lake and why are people enamored?

Yeah, I think at least when I've heard folks discuss it, it's mainly been in a couple of contexts, right? One context is around this idea of next-generation SIEM, right, or SIEM built for scale, right? And being able to put all of your data into a SIEM and have this data lake architecture that can really scale up beyond some of the approaches that have been tried earlier in the industry that you kind of mentioned, whether it's taking off-the-shelf single-node database or taking a single-rack server, right, to really get into this data lake of, okay, it's horizontally scalable, you can kind of decouple the compute and storage, right, and really get great performance, etc., that way. I think the other place that it comes up is basically in this folks wanting to do their own security analytics, right, or folks wanting to have their own Hadoop infrastructure and do security analytics themselves. I think particularly for some of the really mature tech or kind of government organizations, right? And I think in there, that kind of is, I think it's very related into the, in order to do that, and if you want to have your own ALH jobs and things like that, you probably want to have your data in one place. And so I think that kind of ties into a lot of the things that we were talking about. But yeah.

Well, okay. With that, listeners, closing for real. Travis, thank you for being, I think, our first guest to actually give a bonus answer after the closing question.

Great. Thanks for having me. It was super fun.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton underscore Chuvakin and underscore Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.