EP44 Evolving a SIEM for the Future While Learning from the Past

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the product manager for threat detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and member of the cloud security team here at Google. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity. If you enjoy our content and want it delivered to you piping hot every Monday afternoon Pacific time, please do hit that subscribe button on your app of choice. You can follow the show and argue with us on Twitter as well, twitter.com/cloudsecpodcast. Finally, while it pains me to say this, I want to remind our listeners that Anton and I are going to take some time off in December, but we will return with new episodes in January. So stay tuned and don't take us out of your podcasting rotation. Our episode today, Anton, is a fun one. We are talking about Sims, which I guess you've spent the last, I don't know how long of your life doing.

You know, I was kind of saving this for the January episode, but roughly on January 21st, 2022, I would celebrate— please don't laugh, please don't cry— 20 years of Sim.

My God.

And this is scary. This is like, I am saying this and I'm shuddering while I'm saying this. But one thing to note about the episode today is We would talk about failures because, of course, SIEM historically has been associated with many failures, but we will also talk about successes. And we'll talk about some of the key characteristics of a modern SIEM as described by its user. We are not doing vendor propaganda on you. We are having an interview with a CISO who lived through many SIEMs and who found enough evidence and enough pointers to tell others how to succeed.

So 20 years of sim, you've been really logging that time, haven't you?

I've been logging the time and also parsing running reports. Yes, you are right. Must have pun.

Must have a pun. I think this is an interesting episode because we touch on the theme of sim as a blank canvas and how that can be different in a modern situation, which I think is a really interesting idea. We also talk about how sim is often a case of what you put in is what you get out. I joke that many security teams need a couples counselor more than they need more security technology. But I think here again, we get back to kind of a fundamental truth about how human beings relate to themselves and to technology, which is you get what you put in. So with that, let's turn it over to today's guest.

Today, our guest is Mike Oros from Vertiv.com. You're a CISO there, right, Mike?

I am. That's right, Anton.

Perfect. So the topic is probably going to be very painful for many security leaders because the topic is SIM. So I wanted to see if we can talk first about what's modern SIM. Like, the first SIM products appeared almost 20 years ago. So there's 20 years of evolution of history. But Mike, what are your views on modern SIM? What matters today? What should it do? What should it be?

Modern SIM vendors kind of want to dictate what a SIM should be versus really listening to the needs of the end user and the customer. That's kind of led to either too many or too few features that either generate way too many alerts or lead to way too many missed incidents. I think, in my opinion, a SIEM should provide for basic out-of-the-box use cases— alerting, correlation, visualizations. It should be able to seamlessly ingest data and parse it. Who really wants to go buy an expensive tool and not get value from it day 1?

Definitely. And I think that seamlessly collect is already a challenge for many data sources. In my Gartner days, I remember talking to some modern vendors who told me that their key challenges are data collection. And I thought, wow, when I was a SIEM vendor back in 2002, we had collection problems. We are having this conversation in 2021. We're still having collection problems. It's interesting how this is painful even after 20 years of progress, right?

It is, absolutely. You could end up with a dumping ground for log data where you do nothing with it. You're just an expert at collecting it.

This is probably one of the most common failure modes for SIEM that I've encountered, by the way, when people focus all their energy on collection and they have all the data and they sit, as I sometimes joke, on the shore of a data lake and stare at it and hope the insight would come out, but it doesn't.

You're absolutely right.

So, what about the name? Right now, I don't want to push all the vendor wars on you because you have enough stuff to do in your day job, but is the name SIEM at this point needs to go? Is it offensive? Should we call it something else? it something else. Some vendors are trying to push XDR on us, which, of course, is not exactly the same. How do you feel about calling it SIEM today? Do you still think it's SIEM?

SIEM was based out of the basic need for teams to weave disparate data from all their security tools into one pane of glass. But I was kind of mentioning before, SIEM should be whatever I think the security practitioner needs it to be. And you mentioned a minute ago, sometimes it becomes a data lake. Sometimes it's a visualization tool, sometimes it's an alerting platform. It's a combination of all these things. I think to get hung up on what it should be called, it should just be whatever's of most use to the end user. Right now, I think people trying to make it into something and make it fit into that rigid name, SIEM, they're losing the objective, which is to make it something usable, which teams can kind of pivot and react to. And get value out of it.

So let me subtly, maybe not push back on this, but kind of probe into this because I also agree that SIEM should be whatever the specific end user wants. But to me, this also led to SIEM mandate to becoming very broad. Now, some people naively assume that SIEM was born for compliance reporting, even though SIEM was born before compliance reporting in the late '90s, at least in my experience. But I want compliance reports. You want detection. This person wants hunting. Another person wants a great alert workflow tool. So, we really are saying that SIEM should be whatever the team that runs it wants. But doesn't it place a lot of challenges on the SIEM designer to address all of that?

That's a great point you're making, Anton, and I agree with it. But there's also the concept that if you want to retain logs for the sake of compliance, You should do that somewhere else.

Mm-hmm.

You can either do it in your SIEM, or you can do it in a data lake, or you can pre-filter it. Some vendors out there, they encourage you to do this because it reduces on the cost that comes along with hosting the data in the cloud or wherever you want to host it, and it simplifies the management side. But trying to fulfill everybody's wishes— compliance, regulatory requirements— It may not correlate to implementing the best of security practices. And sometimes, when security teams look at data in this massive structure, they just can't make sense of it, and they get lost dealing with it.

Yes, that's actually fair. Again, we're coming up to the same theme of people who collect stuff in one place and then just, like, hope for the best. In essence, I've lived through some of the evolution of SIEM. To where it became more of a data lake. So, to me, it felt like regress. Before Chronicle, again, we're not talking Chronicle as much here, but some of the search-based tools essentially put us back in the 1990s when I had to use regular expressions to search for things. I don't have stateful correlation anymore. I don't even have counting. So, sometimes when I look at the search-based SIEMs, and I'm not going to badmouth anybody or mention names, but I feel like I am being given tools from the previous era. Why do I have to think about string matches? I want categorized data. I want unified schemas. I don't want this silly string matching. Yet, this has been the movement until recently. Sorry, I'm ranting too much. Apologies.

You are. And there's value to that too. I agree completely. And I think you have 2 different kinds of teams, the teams that have been around for a long time and are used to that sort of format when it comes to searching and alerting and detecting unusual activity, those search strings, continuously optimizing them.

Yep.

Which is kind of in line with where things are at in the industry. They just want to be able to detect unusual activity based on patterns of anomalous things. I think that's kind of where we're going, and it's also in line with where the labor market is. There aren't enough people to do the work, so not everybody can be an expert at culling the right information. out of this data.

Yes, or building the 37th version of the regex that just performs a little bit better and matches a little bit righter. I don't know. I'm sorry. I'm a bit bitter about this.

I remember the late '90s. Don't worry. Sad to say.

Yes, the thoughts of late '90s and the very first SIEMs come to mind indeed. Here's the other dimension. I've met enough teams who spend too much time making sure their SIEM actually runs. stays up even. So, the way to solve it is very well known. You use a cloud-native SIEM or SaaS SIEM that backends to the cloud. So, in your opinion, is cloud-native or SaaS SIEM the only way to go for a modern SIEM?

Oh my God.

Can you just install your SIEM in the cloud and hope for the best, or does it have to be cloud-native SaaS?

Oh my God, it's got to be cloud-native. You're killing me here. This is the old lift-and-shift question you're talking about. On-prem applications, I don't care if it's a SIEM or something else, they were built around the constraints of physical hardware. There's no extensibility, there's no infinite scalability or anything like that. So the minute you push something up to the cloud, that's when you're going to start experiencing wonky side effects like the inability to scale, or it just crashes when it experiences a point in time where it really needs to take off and do some hard work. It has to be cloud-native. You ideally want to keep the processing where the data resides.

Mm-hmm.

I have to say, a lot of our vendors for other security tools, they're hosting their applications in GCP. So what does that mean? You know, if I go with Chronicle, the data's already there. You're telling me that I would want to backhaul it and bring it on-prem, waste all that time and energy and money just to analyze it somewhere else? True.

And I think you were kind of harsh on the people who are installing the traditional software SIEMs in the cloud, but I think you weren't harsh enough. I think it's actually worse.

Exactly. You're right.

I feel like when people deploy something that was built in the hardware era in the cloud, they are definitely enriching the cloud providers. So, in some sense, half of me is happy to take their money, but I'm also realizing that It is just not the way to go. If you are copying on-prem thinking, on-prem architecture, on-prem storage mechanisms to the cloud, you are probably wasting money.

You are.

You are enriching the cloud provider, so that's good from the self-serving point of view, but I would never recommend people do that. Like, I don't want people copying on-premise SIEM in the cloud and pretending they have a cloud SIEM. That has so many things that go wrong. So maybe I'm too harsh now.

You're right. Things can go wrong all the time. Like I say, you're putting something somewhere where it was never intended to be. and you're engineering a product, you know, around the physical world constraints. You open the door to the cloud, you have infinite extensibility. I don't know what else to say, but think about it like this. I have to consider where I generate my alerts. I'm using a bunch of cloud applications for security purposes. All that data is being generated within GCP. Why would I even move it out of GCP? And it can just be processed on the edge. I mean, I work for a company, we make all the edge infrastructure supporting organizations like Google. And the whole point is to move the workloads out to where they naturally exist, which is at the edge. With that in mind, why not do your SIEM at the edge?

Yeah, indeed. And that's the result of you avoid some of the challenges from the, you know, frankly, '80s and '90s that you're basically refusing to copy them to the world of 2020s. Like you want to take advantage of the modern stacks, modern technology, modern approaches.

Okay.

So enough of that. I think that horse has been well beaten, even though I do see people still doing this.

Not beaten well enough.

Well enough, yes. In my analyst days, I called it fake cloud when people just install something in the public cloud provider and say, I have a cloud solution. I'm like, no.

Or you mean private cloud, like in a colo? Is that what you mean?

Yeah, exactly. It's a glorified colo approach. Yep.

Yep.

The next question I have that is kind of my favorite question, and frankly, I get asked the same question a lot. What are the top challenges for organizations deploying and operationalizing SIEM? But you live that life today. And so, I wanted to kind of explore what are your top 3 that you encountered. Of course, now that you have a modern SIEM, you have fewer of them, but just wanted to know what are your impressions of the top 3 challenges for deploying and running SIEM?

I'd say the ramp time that includes rolling out a SIEM, which is data acquisition. And then compute and storage. The last 2 are the primary benefits of the cloud. They can also be the most expensive parts of the cloud. Getting SIEM right involves the nature of your subscription and how that's structured. And a lot of SIEMs, they charge you. It's like driving up and down a toll highway. It costs you money every which way, and it costs you money to keep it hot. Keep it cold still costs you money. Then there's challenges with, with keeping it cold. So, what do you do?

Yeah. To me, that's true because when I sometimes I say that scaling something in the cloud is not hard, but scaling something economically is really hard. And again, I don't want to wear my Chronicle hat now, but sometimes the scalability is okay, but economic scalability is not okay. For Chronicle, we focused on the scaling without killing your wallet. To me, this part matters a lot, and it's really much harder to do than scaling the technology. Which is interesting.

Yeah, absolutely.

When people deploy SIEMs, and that's something that I've also studied in the past quite a bit, there are costs that you see, like say, I want to upload this much data. For those SIEMs that charge per gigabyte, you know how many gigabytes, hopefully. But there are also hidden or maybe commonly forgotten costs for a SIEM. Sometimes people forget that they need to hire somebody to tune rules, like if they don't have personnel on site. But there are other hidden and maybe commonly forgotten costs. What's your take on that? What do you think people forget? Forget that it costs money when they deploy or run SIEM.

I think if you're talking about SIEM on-prem, you're talking about the hidden cost, which is the bare metal support. I buy a server today, 3 to 5 years from now, I got to throw that thing in the trash and buy a new one. I have to maintain it, operate it in high availability. I got to replicate data. That costs more money even for more infrastructure. That takes teams of people managing the uptime of the equipment to make sure it's running. So, I think people forget that all day long.

But for a system admin, you need to run the server. Servers connect to the infrastructure. It involves a router, network gear. You need network admin. You need other stuff. So, if you deploy a lot, you have other supporting infrastructure that's supporting costs, right? Storage admin. In the past, we had DBAs being needed for SIEM. Hopefully that time is over, but still.

You're absolutely right.

So you think personnel is the main hidden cost then?

I think it's a hidden cost, and then you have the constraints of the physical infrastructure. Eventually, you're going to run out of space. If you start implementing technology like EDR, it's extremely noisy. You're going to quickly outpace your capacity to store data.

And then that means planning, that's a cost.

Correct.

Implementing the plan, Transitioning, getting rid of old stuff. So, I agree, like the whole on-prem lifecycle involving the SIEM backend is just one big pile of cash that's burning.

It's not elastic. You have to decide what you have to live without. Imagine saying, well, I can afford this to log this, but I can't log that telemetry because I have no infrastructure to put it on. Meanwhile, that telemetry might be the indicator you need to detect a security incident.

Or even something that is not used for detection but is used as context. Again, to use a Chronicle example, DHCP logs.

Correct.

In the past, I would always tell people, hey, collect DHCP logs, but they would usually not do that because they turn to me and say, what kind of rules do I run on DHCP? And I'm like, no, no, no, no, no, you're thinking about it wrong. DHCP supports your rules on firewalls, servers. You don't write rules on that. And they're like, But it would cost me 1/6 of my license for the vendor. And I'm like, okay, then don't collect them. Fine. What can I say? You see what you get.

DNS, DHCP, those are mainstays.

Yep, exactly. And this is why, again, maybe I'm being too self-serving about Chronicle, but I was impressed that it's kind of a required data source. And of course, we don't charge per data source. So people do do that in case of Chronicle, but they don't sometimes with other SIEMs. So I want to switch gears a little bit to yet another false hope that we lived with. Many, many years ago at this point, there was a small team of people in Spain who launched OSSIM, open source SIEM. Also many years ago, if I recall correctly, in 2009, I wrote the blog post that says open source is not the answer to SIEM. And I upset a whole bunch of people. And I kept upsetting them over the years because people kept looking at this blog and saying, Anton, you're wrong. And I said, no, I'm right because a lot of things you have to deal with to succeed with SIEM. are very labor-intensive, and they're kind of boring, like building parsers. And in my mind, they were not a good match to open source. So, is open source the answer to SIEM today? Sometimes this means huge expenses, but sometimes it means other problems. So, what's your take on this?

Well, my take is, you know what, the old expression goes, you get what you pay for. If you have no other option and open source is the only thing you have available, Sure, go ahead. Something is better than nothing. But at the end of the day, just like I mentioned, running your own infrastructure and hosting your own SIEM internally, there's hidden costs. You're going to have to, like you mentioned, parse data. You're going to have to set up all the different things you need in order to alert yourself, and it's going to become a very manual process. Not only that, you're also going to have to be concerned about bug fixes, support, things you'd naturally get with a SIEM. In a fully managed SIEM, you're more focused on security than maintaining the infrastructure or just making sure it's working. That's my opinion on that.

Indeed. And I think the cloud cost, we can kind of fuse the question we just discussed about on-premise tools going into the cloud with this one. So, you may be deploying a tool that was born on-premise, and the tool may be free, but it's so inefficient with using data in the cloud that your cloud cost is just ginormous.

Huge.

Like, I've seen that happen with some tools. Huge.

I've done a POC in one where we hosted in the cloud a VM, and it cost $20,000 for 30 days.

Ah, well, VMs is probably the most wrong way to store data in the cloud. So, I sort of get why, but if the vendor only lets you do it, then you're stuck.

Correct. This was a number of years ago, and it was an experiment. We did a bake-off, and I won't go into what we went with, but it ended up being on-prem. I'll just say that.

By the way, I don't want to sound like this is like a commercial vendor propaganda where we just like pile stuff on open source. It's really not about that. It's kind of about the SIEM specifically being focused on tasks that just don't have good coverage in open source community, like maintaining parsers, maintaining ways to taxonomize and organize data so that the users wouldn't have to do the work. I have not seen open source deliver on that, not in the time of OS SIEM, Not in the time of modern open-source choices in this area. So, I remain to be somewhat of a skeptic of open source for our beloved domain of SIEM here.

Correct.

You have to do what returns value. Nowadays, with security being at such a premium and such a high risk, you really should be focusing, spending most of your time doing security work instead of IT maintenance work, or even just trying to get to the point where you can do security work. Why not just cut to doing actual security work? And skip everything before it.

Perfect. This actually makes sense. And to me, this is what people often forget because they see the glamour of open source and they forget that the glamour is going to be built on their backs, them running long hours and them working over the weekends to make sure this tech stays up.

Correct.

Another fun question to ask is, one thing that bothers me sometimes with scaling SIEMs is that the data they need may stretch further into the past than before. And I really hate to sound like a typical vendor and mention SolarWinds, but Even without SolarWinds, we did have breaches where investigation had to reach back 100 days, 200 days. Typical dwell time still in the dozens of days, if not hundreds. So, to me, this points at a retention policy for a SIEM. And I don't want a cold— cold sounds like the 1990s. I don't want cold data. Cold data is stupid in today's age. I want the data to be searchable. Like, you don't have cold internet on Google, right? Like, you don't say, Sorry, guys, this internet's too old. We're going to search for it in batch mode. Like, that's not how it works. So why do we have cold log data? That's just silly. As far as data retention, I've lived through PCI DSS years, which kind of pointed to the first 90 days and a year. We have some people who want a year just because it's a nice round number. What's your take on modern retention requirements? How much data do you want to keep for detection, for response, for hunting?

At the bare minimum, 6 months. A year is what I really want because it gives me the luxury of that extra 6 months between 6 months and a year.

Mm-hmm.

You mentioned PCI. I don't even care about PCI. We're a very lightly regulated entity. Ultimately, I think nowadays a security breach, you need at least 6 months back because that's what they're saying now, that the dwell times between a threat actor being in a network and detection It keeps spiraling further and further out because these guys have low-and-slow techniques to exfiltrate data and do whatever they're going to do. Sometimes they just compromise the network and sit in there for a while. They move on to something else, then they come back. The dwell times are going to get you. You need that 6 months at a bare minimum.

Yeah.

And a lot of SIEM vendors play games with their prices by saying, oh yeah, we can match a price. We'll put the first 90 days and it'll be hot. Next 90 days will be warm, and then the next 6 months will be cold. But what happens when you need to scale right away? I mean, if I'm going to do a major investigation, I'm going to spin up 10, 20 cores of compute in order to just analyze the data, and I'm going to want to reach back and research everything. I don't want to wait around for 3 days until they can rehydrate the data, do whatever they're going to do to it to bring it back to life. Forget it.

The fact that we discuss it in 2021, this makes me sad because this is such a 1990s problem to have. Like, you're gonna have to order little gnomes to like load stuff from tape into like racks. Like, all this sounds kind of silly old, but I know people do live in that world sometimes, especially with Sims, when you had like re-import old data from storage into your data store. Like, Like, why are we talking about this? Let me get the tapes.

Like, what? Let me get the tapes off the shelf so we can do that, right?

Yeah.

And that's why I'm somewhat shocked. I know working at Google, people do take to assume that most of the world looks like Google, and I know it doesn't. This still sounds like a practice that should have died 20 years ago, yet it still is there. And since then, they're still trying to sell cold storage to reduce money. Like, why? Why do you need that? You can search throughout the year.

They're buying their storage from you guys. They're buying it from Google. They're hosting the app in Google, and they're passing on their fees, and then they have markup and everything. So let's cut out the middleman.

Yeah, there's that. And we can store data way more efficiently than them anyway. So I want to start closing because this is our typical time for the episode. So I want to raise 2 specters. One is a specter of ML, machine learning, because a lot of vendors have been trying to convince their clients and prospects that the only way to do modern SIEM is to use a lot of ML. As a former Gartner analyst, I studied a lot of this in the field with clients, and I was seeing some advantages in some use cases, but I just did not see the magic. As I was sometimes joking, the ML unicorn cavalry isn't coming. It's going to improve things, but it's not going to magically transform stuff in the short to medium term. What's your take on the ML approach to SIEM? Do you demand it from your vendor? Do you have to have it? Or what's your impression?

I don't personally need it. And why do I say that? Because I've got applications, like I said, those edge applications that are doing EDR, NGAV, those are machine learning enabled applications that use heuristics to detect and alert on things. Those alerts are then pumped into the SIEM.

Mm-hmm.

It's already happening. I don't need another set of ML running in the background in the SIEM to then parse it even further. Sometimes I like a little bit more unfiltered data, and I like to use more traditional means to analyze it. That's it. I don't know if I really have the right resources to run a sim that's powered by ML, and I really don't know if I see the value in it at this point. So I'm kind of aligned with you.

Sure. And I guess long-term, I expect to see magic, but short-term, there is no magic.

I'm a pragmatist in that regard.

Yes, exactly. So to start closing, I want to ask our traditional question in the end is, if the audience would remember one thing that they can implement in practice, one tip on SIEM, one tip on deploying SIEM, what would it be? Just give us one practical recommendation.

I'm going to go with something so basic. Just make sure your logs are being fed into the SIEM because I've encountered times where we're looking for this data. Where is it? Oh, it hasn't been logging to the SIEM for X number of months. And it's like, oh my gosh. Just make sure your data is getting to the SIEM, make sure you're parsing it correctly, and make sure your teams are in the SIEM and getting value from it. Just the basic management stuff. I don't know what else to say. Keep the lights on.

Frankly, that's useful. People like it. And I think that if you paid 6 digits or 5 digits for a SIEM, how about you use it? That sounds like good advice, right?

That's the first place to start. Start using the SIEM.

Thank you very much, Mike. This was very insightful, and I think the audience would love it. So thank you very much for being on the podcast with us.

Yeah, thank you, Anton. It was a pleasure being here.

And now we are at time. Thank you very much for listening and of course for subscribing. You tolerated more than 20 minutes of us talking about SIM. Think about that. That's an achievement. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website. Cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe to these episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @anton_chuvakin and @_timpeeker. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. Finally, don't despair. We are not releasing new episodes in late December. We will return in January. We need a bit of a break. See you on the next Cloud Security Podcast episode.

Security.