EP125 Will SIEM Ever Die: SIEM Lessons from the Past for the Future

Security. Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on LinkedIn. Anton, this is a fun episode because it's available as a video that we first livestreamed. And listeners, you should look for livestreams on an increasingly regular cadence. It's also available on our YouTube. And what are we talking about in today's episode? Is it, dare I say, your favorite topic?

It is an episode where we talk with somebody who's been doing SIEM for a long time, and it's not me.

That's amazing.

Which I guess adds to my excitement. And it's also an episode which sort of bridges some of the lessons learned a good number of years ago and the lessons that apply today. So, it's kind of, to me, the main theme is some things you learned in the past are very applicable. Some things you learned in the past, You really want to chuck out of the window.

Oh, that's always interesting when you can look back and say, wow, some of this is still good and some of this is not. I think that's growth. This is an episode about growth.

It is an episode about growth, yes, but it's also about Sim, by the way. Reminder.

Ah, yeah, that too. Okay, well, listeners, with us taking wildly divergent views on this episode, let's turn things over to today's guest. I am delighted to introduce today's guest here on the podcast and the live video stream. Today we're joined by David Swift, a security strategist at NetEnrich, and he claims to be the only person doing SIEM as long as Anton. So, David, I want to get right into it with you. I want to start off with maybe a warm-up question. Could you first tell our listeners what is a SIEM? Because I always get beat up by people, oh, you use so many acronyms. What's a SIEM? And what SIEM lessons of the past— not aging anybody here— N years apply to today's world?

No worries. Good place to start. So SIEM by itself stands for Security Information and Event Management. And this is actually where Anton and I met years and years ago, where it started out as SIM, and I argued it had to have an E in it, include events, because a lot of what we do isn't really information, it's just events. It is collect a lot of data into something so that we can search it and do log analysis and find threats, a lot of times for auditing or compliance. And do some detection on it. Let's automate some things to find the interesting stuff because we'll collect billions of events and only a few of them actually matter. So that's where it starts, is collect and detect something interesting in it, try to dig it out, find the needles in the haystack, find the stuff. And then hopefully these days in a modern SIEM, not only add just detect what we write by rule, but detect anomalies. Let's start adding the ability to automate response. Let's start solving business problems, not just malware problems.

So wait a second. So you're still trying to highlight some of the things that were true when SIEM was being born back in the early 2000s. But are there specific lessons that maybe people learned, well, some people learned 20 years ago that others are still missing today? I think that's a more, maybe a slightly more fun angle on the old questions. Which, what are the forgotten questions or forgotten lessons?

Awesome. And I'll give you the one that I learned the hard way that took me 10 years. So in SIEMs, I thought I knew everything. being invited to present at conferences and deploy SIEMs for Apple and Microsoft. And I'm presenting IP addresses and malware and problems. And I'm going, these are the issues, these are the problems. And the board level is just ignoring me going, bye-bye, just go fix that. And the lesson we need to learn from SIEMs is it's not about protecting machines from malware. It's about protecting the business from whatever's important to the business. That could be downtime if it's an availability issue. That could be data loss. That could be, you know, having the website maligned and showing things that you really don't want to say. So, as security practitioners, the hard part is actually to learn not just our job, but what is the business, and start tying what we're detecting and reporting on to actual business risks. And I think if people get that, they will be further ahead than I was for 10+ years. So, I think that would be one of the great lessons.

I feel like we missed the opportunity to deploy one of my favorite Lord of the Rings quotes here about, you know, Things should not have been forgotten or lost. History became legend, legend became myth, and the importance of focusing on business risk passed out of knowledge.

I think that fits the bill because I definitely see if you go look for some late 1990s book on security, they would talk to you in the book about business alignment and then fast forward some number of years and it's like, do you know that security is relevant for business? And people are like, No way, right?

Well, I mean, it's really hard to sell a product that solves that problem, but it's really easy to sell a product that promises to catch Fancy Bear.

There you go.

Yes, that's a fair point.

And that was one of the ugly lessons I had to learn again a few months ago when I had to rewrite content that I wrote 15 years ago. And I'm reading these old papers going, oh, if there are more than 15 firewall drops in a 10-minute interval, and I'm like, oh, that, that's what, that's Yeah, it might be an early indicator, but I'm like, oh God, let's write something better than that. There are better ways to do it.

Please. Well, what's the better way to do it? Let me take the softball podcast thing and just say, well, what's the better way to do it there, buster?

Well, let's go with your first theme. What's old is still relevant. So I still want a lot of firewall drops. One of the patterns I look for is repeat failed anything. Repeat failed logins, repeat firewall drops. It's one indicator. But it's only one, and it's a fool's errand to think that I can tune that down to never have false positives. It will always be noisy no matter what I do.

Never?

Never. Never?

Never, Anton. Never? Eat death of the universe, baby.

I've tried it a dozen ways, 30 years of trying it. I haven't gotten it to low false positives.

We sound like barking dogs who just like jumped at him like, never, never, never.

No, it's never. It's never for sure.

But here's what I do. I mean, this is the whole MITRE ATT&CK framework. This is the kill chain model from DoD. One event means nothing. The second, the third strike means everything. It is always an early indicator. I see the reconnaissance events when somebody's hitting my firewall and finally finds a port that's open. So the firewall drops are useful, but only 1 out of 1,000 events that it triggers. So I feed it into a threat model that says, okay, failed logins or firewall drops followed by success, okay, they brute-forced, they found a way in. Followed by access to something in my network that I care about, that's business relevant, that they haven't touched before, or files they haven't used before, or quantities that's different. So now I'm moving away from just looking for known malware, and I'm now starting to look for threat chains. 3 strikes, I can tell you from measured results, puts me down about a 3% false positive rate. 1 strike, 97% false positive. 3, 3%. Now I can do something about it.

It's a lot better.

But that's the classic— is that the whole sequence of login failures followed by success? I don't remember it exactly, but that may have been the first rule I wrote for back in my first SIEM vendor days, probably.

Anton, when you first wrote a rule, did you write it in assembly?

No, because I had a UI at the time. That's another forgotten lesson. There's a beautiful visual rule design UI. Which existed in 2003, and then it stopped existing for some products for 10 years. And then suddenly now, hey, UIs are cool for building rules. And I'm like, yeah, I had one in 2003. Thank you very much. It had little boxes you can connect.

No, listeners, the way Anton talks about this stuff, sometimes you'd think that he was writing his first rules on punch cards. So I'm glad you had a UI.

It's the opposite. It's the opposite. That's the thing. And okay, so before I go on a nonstop rant, I wanted to actually bring a useful angle to the question you asked, namely, certain lessons that you learned many years ago and certain lessons maybe I learned many, many years ago actually do not apply anymore. So, are there any lessons, any old SIEM lessons that just do not apply today? And if you follow them, you would harm yourself?

That's a great question.

I would say depending on a SIEM to detect a specific known threat, you're going to waste a ton of time and create false positives. You're going to say it looks like X. Well, if we could detect X and block it, we'd put it in an IDS signature. It's not a SIEM rule. Why do it as a SIEM rule? I think that one is a harmful prophylactic. People think they're detecting stuff, and these rules detect ancient things that should never happen, and they think they're all secure. And it's like, no. And it's not so bad in and of itself because those rules will probably never go off. And if they do, yes, it was confirmed malware. But you're not really looking at what's interesting and new.

So this is profound and it's worth, I mean, pulling on the thread a little bit. I've met enough people, probably including Tim, right, that like precise rules. And by the way, I am subtly changing precise rules for rules for a specific thread. So this is maybe, I guess, a bit of a mind—

Those are different.

Yeah, they can be different. Yes. But they also can be the same in many minds. So, are you telling us if I use SIEM as a— to use another 1990s term— host IDS, then I'm doing it wrong?

I certainly would—

Tim is snorting, snorting like a network IDS and a pig.

If we wanted to do rules that detect specific known threats, I'd leave it to the vendors that do that better, to the EDR vendors at the endpoint, To an NTA vendor, to an IDS vendor. If I'm going to write a lot of very specific signature detection rules, I'm going to do it somewhere else because I can do it at packet level. I'm not doing it at metadata level. I could get into something that's actually a lot closer at a packet level in an IDS or host-based or network-based than I can with a detection rule in a SIEM. So I use SIEM to look for patterns, repeat attacks, success after failed. Anomalous events, something they've never done before, a spike in quantity, an activity that nobody else like them does. And when I use those patterns, for me, it changes the equation a lot because I'm no longer looking for X, I'm looking for any weird. Because my job as a SOC analyst, and I've had to do that job over and over staring at a bunch of stupid things, is to figure out what is normal and what's weird. What changed today that tells me the zero-day got in? And if I'm writing a rule that looks for a known evil, that's not a zero-day. And I really want to find the zero-day. I want to find the new threat. So I work with patterns rather than detection rules. And if I want detection rules, I write them on my IDS.

Okay. So there's a couple of really interesting threads in that one I want to pull on now. One, I live in San Francisco, and so I'm surrounded by weird. Are SOC analysts not also surrounded by weird? When I look at what my cloud customers are doing, Every customer is doing something weird. How do you know weird from bad weird? And that's a problem in San Francisco as well.

But this is like the defining question of anomaly detection for the last— wait for it— 40 years.

Sorry. I think we stumbled into something interesting. It's not our fault.

Yeah.

Yes.

Well, now you're getting into AI versus ML. So I'm not a big fan of AI as used in security yet. I think we'll get there. But AI would say, I know what's bad. ML says, I don't know what's bad, I'll just learn what normal is. And then I can write a rule that tells me there's an anomaly. And now I go back to that 3 strikes problem. One anomaly, who cares? Weird is everywhere. But 3 weird things today? You're running a bank, you have ATM machines. This ATM machine ran a piece of software that no other ATM machine in your network runs. It started communicating with an IP address outside your network that none of the other ATMs talk to. And it just spit out $3,000 in cash in the last hour, and it's never done more than $3,000 in a day. Whoops, I'm hemorrhaging money. So learn normal, find weird becomes relevant. I don't need one weird because one weird, high false positive. And in fact, I like to weave in those old rules. So I still want the old stuff, the known evil. It's on a known malicious IP address or malicious URL or running a malicious hash. But Those are great, particularly when I've got a pattern that mixes anomaly detection, the weird, plus a known evil. That detection is a very high true-positive rate. That detection is the one I can act on because the scored data, the known evil, that supervised machine learning, the idea that AI knows what's evil— we have some scored datasets. We've got a lot of open-source threat intel. There are great threat intel feeds that are commercial that we can pull in. And combinations of known evil plus anomaly are great.

Well, so you make this sound easy. Why doesn't everybody do it that way?

No, you really do want to, like, pull the COVID and it's like, there's a corpse there. So I think, I think Tim is— sorry, my metaphors suck today. I don't know what's up with that.

Well, I'll give you a couple of responses, Tim, because there are some reasons that people don't do it well.

Yeah, yeah.

One of the first problems is it's expensive. How do I put enough data into the set to learn normal on everything? And depending on the vendor you use, that cost can be prohibitive because I want to know vulnerability scan data. What does the asset do? I want to know what IDS events, I want EDR events, I want application events, I want network traffic anomaly events. I want a lot of data.

That's a lot of data.

And not only is it expensive, but most vendors have done a piss-poor job at making that performance scale. So, the minute you start getting above 10,000 events per second, most of the vendors can't chug through the data fast enough to find you what you want. So, now I have to move out to a cloud environment, and I really need scale. And I'm trying to stay away from recommending any specific vendor or bashing any specific vendor, but scale is a huge issue. So, it is cost and scale that have kept everybody from doing it, even if they knew the right thing to do.

So, okay, I want to transition to our next question on this note, and I'll leave my commentary about zero-day threats versus other kinds of threats maybe for another episode. And listeners, you should go back and check out what we did with Matty about that. But you mentioned costs. So what are the benefits and what are the costs of SIEM today? Like when you think about doing it right or doing it this way as you propose, it sounds wicked expensive, but surely there's upside. So how do we, how do we trade off that cost-benefit thing here?

Anton, let you go first this time.

You know, I—

You can't punt to the host.

Yeah, he can punt to the host, but my problem is that we have about 12 minutes left, and I may have written a 60-page paper on this topic a good number of years ago, so I don't want to go in this direction. I think that for benefits, I do plan to send people to some of the more recent writing. And to me, most benefits of SIEM today are about detecting threats and about organizing your workflow around threats because SIEM today includes SOAR, right? And also with having a single platform to triage, confirm, and deal with things, including hunting, right? It's kind of a blah answer. I get it. But it's also kind of a factually correct answer. SIEM is a mature— it's a fairly mature tech, and it has clear benefits today, but they are also not very rocket science. They are kind of more normal. You have a platform to detect threats across log sources, telemetry sources. You have a platform to confirm, triage alerts. You have a platform to investigate and respond. That's SIEM. If you don't call it SIEM, I'm fine with that. But you probably have a platform of that sort. And to me, this is probably a good moment for all the SIEM haters to show up in comments and say, SIEM is dead, SIEM is dead. We have this other thing that detects threats.

And I'm like— XDR.

That's a SIEM, right? No, you can't say XDR. If you say XDR, we have to do something to you. I don't know. What did you do to this guy who had it?

Let's not go down XDR.

XDR, I think—

He's got some strong opinions on that one.

No, avoid the topic. Avoid the topic.

Wait, wait, strong opinions are fun. And our XDR episode got a ton of listens and people really engaged about it. Here's 30 seconds to rant about XDR, David.

Let me give you a couple answers to your question about where is SIEM beneficial.

Yeah.

And I'll tell you from a CISO perspective, and I've done that tour of duty twice, I'm blind without it. I'm like, how am I supposed to secure my environment? I can't see anything. I don't know what's going on. I can't log into every tool every day and search through a billion events. It's not gonna happen. So I have to have it. I have to be able to correlate across the tools. And as Anton was alluding, we want to write— we want one place to write a report. I don't want to have to go figure out 10 different report writing languages. I want one place to do it because I have to write reports for my auditors and I have to store the events for auditors. So whether I want it or not, I'm going to have to have it for compliance. And I'd really like a place to give me visibility about what my other tools are doing. 'Cause I'll tell you, one of the places that SIEM's not used well that it should be is to do volume and variety analysis to tune each of the devices that are feeding into it. I can't tell you how many places I've gone to that have an IDS that can do 10,000+ detections, and they've turned off all but 100 because they thought everything else was a false positive. And I go, no, I need that early indicator to know what's right, and I'm gonna correlate it with other things. By itself, an IDS event means nothing. And I'll tell you the specifics where I see the IDS problem. Working with one large telecom manufacturer where the EDR guy is running McAfee antivirus, and he's going, we keep getting hit by malware, and I don't know where it's coming from. But my IDS tool tells me, hey, instant message logon just happened, instant message file transfer happened. Now the EDR tool goes, hey, I just detected malware. Oh, now I know the vector. So yeah, instant message login doesn't matter. Instant message file transfer doesn't matter. But together, it tells me the infection vector. So I want to turn it all on, and I want the SIEM to sort it out and say, these are the relevant series of events that matter so I know how to block it. Because without that, I just see malware shows up on the endpoint. What do I do? It's just everywhere.

I think that's a fair point. And it's also a fair point for the cross-device, cross-vendor look at things, which is very much the subject of a lot of heated SIEM discussions 20 years ago, like cross-device correlation rules and all that. And it is still of value today. If you're looking at just EDR or just NDR, then you would still miss things. And if you're looking from across, then you're less likely to miss them. But which brings me to the other favorite question, use cases. And I've been meaning to ask you about like, hey, give us your favorite list of use cases. But this is a cloud security podcast. So why don't you give us your favorite cloud security use cases for SIEM today?

All right. Well, I tried to make them easy and published them out on a blog on the Netenrich site if you want to go look them up. But in the cloud world, what I'm looking for, a lot of times I don't have an endpoint, so I cannot look for EDR. I'm not looking for malware anymore. What I'm looking for is—

All those EDR vendors who want to get rich in the cloud are misguided?

I'm like, wait, I have a containerized OS. What the hell are you going to put an EDR on? There's nothing to detect. Anyway, so I start with single sign-on. You signed on from a country you've never signed on from before. You signed on at a time that you don't normally use. You never log in at Sunday at 2:00 AM in the morning from China. Oops. Those are 2 early indicators. Now I look for files, what you access, because the single sign-ons will tell me the application you try to get into, and then I want the app log. All right. After signing in from China at 2:00 AM on Sunday, you accessed a file that you've never touched before. And not only did you access one, but now you access 3,000 files. Okay, I have data theft, I have data misuse, and it's not hard to detect. It's not crazy complex patterns, and they work really well together. I'll tell you the number one I keep getting told about is crypto mining, and I do see it come up. I see somebody log into AWS and they spawn 3,000 instances, very large or XL instances of an EC2, and they've never spawned more than 3 in a day. All right, they're crypto mining. And it usually starts from someplace like China or Bolivia. And, you know, I see the same starting pattern of unusual login, unusual place, and then you spawn more instances than normal. Yep, you're using my infrastructure on my dime to do crypto mining. Thanks. Get off my network. So 2 easy ones.

Okay.

I think Tim has something to say about the crypto miners, obviously, but okay, I'll shut up. Yeah, you, you go say something about that.

No, no, no.

I think we're, we're running up on the clock and I want to ask the other side of this question, which is, Sure, there's lots of good stuff we can detect in the cloud, but what makes SIEM hard in the cloud world? You know, what's different now than 2013 or 2003 or when dinosaurs roamed the earth and Anton was first doing SIEM?

I think he just called us old.

No, no, no, no, I'll survive that.

Wise. Wise. I called him wise.

I'll survive that.

So, what makes it hard in The cloud. Mostly it's easier because I can onboard faster, I can pull everything via API, I don't have to have on-prem devices. So a lot of it starts out a lot easier. What makes it harder is that some of the logs aren't in a lovely structured format, and they might have 500 fields that are relevant, and the old SIEMs only had a very limited schema that was indexed. So searching something that's in field number 398 when you only index the first 64 fields means a full table scan, and you're going to hopefully find it tomorrow or the next day or the day after. So you need a modern system that can handle search at scale because we've got more logs and they have more relevant fields. So those app logs get a little bit more complex. That's a hard part to it, and you have to parse and normalize better.

Anton, you have the next bomb-throwing question.

Yeah, it's the bomb-throwing question. And I think that I— it's also a self-serving, easy question because I know the answer to it. And the question is, drum roll, do you think SIEM can ever die?

Certainly say no.

No, no, I should change the emphasis. Do you think SIEM can ever die? That's better.

You can minimize what SIEM has to do, but I don't think you can ever take it out of the equation. I got to have it for audit and compliance. I need the detection rules to do cross-correlation. I can minimize it by Building threat chains, lowering the false positive rates, feeding into a SOAR to automate the response and not put a whole bunch of SOC analysts sitting in front of it, but I can't eliminate the function. I need the function.

That makes sense to me, the bringing it together, the correlation, certainly the single pane of glass so you're not doing swivel chair incident response. But a lot of those things you could accomplish not with a giant data lake, but maybe with some piece of technology like XDR that brought everything together? Could we replace SIEM that way?

I find XDR to be bad SIEM plus bad SOAR. I don't care for it. XDR says, I have one match, it's on somebody's IP evil list, I'm going to start blocking. And I can tell you from early days of doing ArcSight Threat Response Manager, where I ripped it out more places than I ever installed it, you start blocking based on one event, you will DDoS yourself. And that's what XDR starts to do in a lot of environments. That, or you tune it down and you say you're only going to detect and block on the things that are known, very specific evil. You're no longer looking for patterns. You're not looking for the combinations that are interesting, that are zero-day. You're looking for known evil. And now you're giving yourself another prophylactic, thinking you're protecting, and you're missing realistically about 67% of everything that gets into the network. is not detected because it's on a known evil list. So only a third.

But known bad is not bad, right? Only known bad is bad. But I think you're almost saying that known bad is bad. I don't think known bad is bad. Doing only known bad—

Known bad is bad when it's what you'd block on, but not if it's a component to your 3-ingredient detection sandwich.

Correct, correct. That's very smart. Yes, thank you.

And I'll tell you the specific cases I see. So working with one very large clothing manufacturer, billion-dollar site, They put in XDR, they're going to start blocking everything. And they see an internal machine that does a callout to HTTP request to a website that's flagged as malicious. So then they block that IP address. And it was just on a web page with a drive-by link, and it never got out. It actually got blocked at the proxy. But because it tried, XDR went and said, I'm going to block that. And now pretty soon every machine in my network can't get to the internet anymore. Great, I bricked myself.

But this is— it's a story like, what year is this? I mean, people like back in the warm years, like, I don't know, MS Blaster and stuff, when people wanted to do some kind of a NAC and like all the ancient acronyms forgotten by most people now. The story you told us right now is the story from a bygone era, but you're telling us that it still happens today?

I still see it a lot. Okay, my favorite. I still get When I'm doing SCADA work, energy sector, NERC FERC stuff, the FBI still gives me a list, and I swear every 6 months I'll see 127.0.0.1 on their list of you must block this and report it to me.

What?

And I'm like, that's loopback. Every process running on this host talks to loopback to talk to the other processes on the host. You really want me to block loopback and tell you every time loopback is used? You'd brick my network. Stop it.

Wow. Some things never die. And whether it's the 2020s or maybe 2030s— hey, I'm an eternal optimist— we'll have people trying to blacklist 127.0.0.1. Okay, well, it's something exciting, please, because this is depressing.

Well, I think the exciting thing is I get to ask our traditional closing questions, and maybe we'll have a good recommended tip on how to improve your use of SIEM for cloud and recommended reading.

You have to also say don't recommend too many of Anton's blogs, blah, blah, blah. Tim says that all the time.

Oh yeah, if you recommend Anton's blogs, You put a dollar in the swear jar. Well, your blog is fair game though.

Hey, you know what I found? Other places I've worked wanted to call it intellectual property, and I've been saying the same thing for 20 years. So I just finally said, I'm going to write it all. Y'all can have it all. Come to me if you want help implementing. So I give away every secret I have. In terms of where SIEM in the cloud can be helpful and beneficial, I truly believe it needs to include not just Known evil detection, but also anomaly detection. I want to switch to patterns, and I want to look for things that trigger 2 or 3 patterns. The minute I do that and I start using an attack chain, my false-positive rate goes down. My SIEM becomes less about just a noise-generation tool and more about a threat-detection tool. And it's not the one; it's the series. Learn normal, find weird is what I do with behavioral anomalies. Learn normal for each device and each user and start looking for the weird. When you see 2 or 3 weird things, it's time to look at the device, and it's probably going to be pretty darn useful. I think that works out quite well. Now, in terms of reading, MITRE has a really good paper recently on the 15 detection techniques that have worked well over the last 3 years that detected 90% with just 15 techniques. It doesn't take 1,000 rules. It—

15 techniques.

I missed that paper, so thank you. I appreciate it.

Yeah. It's a really good one. So that's one that I would recommend. I've got others that I like. I mean, to me, I find the SANS Reading Room is full of some great research by people that have done a lot of good stuff. So I like the SANS Reading Room and some of the SANS Gold Papers. Anybody's welcome to reach out to me and I'll give you more links to other stuff. Let me see if I can find the MITRE technique and I'll post it in the chat and you can share it with the audience because that's one that I think is one of the best ones out there.

Well, thank you, David. This was very fun.

Yeah, David, thank you for joining us and putting up with my XDR needling.

I enjoyed it. It was fun, guys. Thanks a lot.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Cevakin and Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.