SIEM Modernization? Is That a Thing?

Security.

Hi there. Welcome to the Cloud Security Podcast. Thanks for joining us today. Your hosts here are, as usual, Tim Peacock, the product manager for threat detection here at Google, and me, Anton Chuvakin, a reformed analyst and member of cloud security team here at Google. You can find this podcast wherever podcasts are distributed and at our website, Yes, we will have a website. Yes, it's coming. You can also follow us on Twitter at twitter.com/cloudsecpodcast. Our guest today is Eric Foster, the president of Cyderis, a phish tech company. And the topic of the conversation is, of course, SIEM, security information and event management. And of course, some of the listeners were asking us, when are you going to talk about SIEM? And, you know, today is the day. So welcome, Eric. I wanna start from the first question we have, and it's kind of the definitional question. In many cases, we talk to clients and between ourselves, and you kind of say, do you have a modern SIEM? Like, do you have a SIEM that fits today's requirements? So Eric, what's your take on that? How do you define modern SIEM? What are the modern SIEM components and requirements perhaps?

Anton and Tim, thank you so much for having me on the show. Big fans, big fans of the show. And I gotta say, it's a distinct honor to be discussing SIEM with the person who's been called the godfather of SIEM. How do I define modern SIEM? I mean, to me, when I think modern SIEM, I really think a SIEM that is capable of dealing with the big data problem that information security has proven to be. In fact, I would say information security was proven to be a big data problem a long time ago. It's just a lot of people haven't realized or haven't conceded that point or caught up to that point, if you will.

Hmm.

I think modern SIEM really means 2 different potential architectural approaches. I think it's either running a quote unquote cyber data lake. I know how much you love that terminology, Anton, but, you know, swimming, drowning in the cyber data lake, and then running an analytics solution on top that is a quote unquote SIEM, or having a modern solution that can do both the data lake and the SIEM functionality in one stack. So if I said a modern SIEM solution, I like to go use cases, right? I think there are 3 sort of primary use cases for modern SIEM. First, it's something that can support the minimum data retention that's necessary for modern investigations. And what do I mean by that? I think it's about a year of data, hot searchable at a minimum. You know, I think SolarWinds really proves how important— the SolarWinds attack really proves how important it is to have 9 10, 12 months of data hot searchable on standby. Use case number 2 for me is something that provides what I would describe as business reasonable performance, meaning that your analysts, your team aren't dealing with the historical coffee break query, you know, to do their day-to-day work. They don't have to run a query, go get a cup of coffee, and wait for those query results to come back. And use case number 3 is something that provides business reasonable costs, so isn't ludicrous cost to accomplish use cases A and B.

I like that, the concept of the coffee break query. I was at a startup for a while where we did the back of the envelope math and we realized we had a very performant data lake, but every query cost a cup of coffee. So you can either have your coffee break query or your cup of coffee query. You call Anton the godfather of SIEM. I just want to know whose bed he left a horse's head in. No, I wanna—

You really don't want the answer to that question, my friend.

No, I suspect I don't. I wanna pick at this modern SIEM concept a little more. Does that require it to be a SaaS SIEM? Is there a future for the on-prem SIEM?

You know, I personally believe that on-premise everything is dead. I just think a lot of people haven't caught up with that either.

I believe people's fleets of laptops and workstations are gonna disagree with you on that one.

Well, sure. When I say on-prem, I don't mean that there's going to be a laptop or desktop someplace, but, you know, from a zero-trust architecture perspective, which I think is very much the future of networking and architecture, that those laptops and desktops and endpoints should be synonymous wherever they exist, whether that's sitting in a Starbucks or, you know, sitting in somebody's office building. But, you know, when I say on-premise is dead, I mean on-premise data center. You know, I think The cloud service providers have very clearly proven that it is not only much more cost-effective, but much more secure and better in literally every way to run things in the cloud. Although I will say everything in IT and security seems to be so cyclical that I'm very much expecting like mainframes to come back any day now. So who knows? Yeah.

And of course there's those Europeans who may have a bit of a beef with that argument, right? That everything's going to be in the cloud because Last time I checked, the only cloud providers are either in the US or in China, right? So if you are not US or China, you're going to use either foreign cloud provider or you're going to use, well, not a cloud provider. So here's the question that I get asked a lot as, should I really say it with a straight face, potential godfather of SIEM. What are your top 3 root causes for SIEM deployment failures? Like, I get the failure question a lot. I had a presentation many years ago about the worst practices of SIEM, how to engineer failure into your SIEM deployment. I really had collected a lot of failure tips for SIEM. So what are your favorite top 3? How to really ruin your SIEM deployment? What caused the biggest failures?

Top 3 root causes for SIEM deployment failures. I would have to say first and foremost, it's lack of resources. And then resources can be resource time, It can be resource expertise, it can be focus, it can be dollars, it can be dollars to architect the system. You know, ultimately, this all comes together in, you know, if we talk about those 3 use cases that I provided above, the concept of being able to retain the data that's necessary for modern investigations, to provide business-reasonable performance and provide business-reasonable cost. I mean, to me, resources and resource availability comes in as the number one root cause. Number 2, I would say, you know, and this is much more once the SIEM is actually deployed, often by, you know, under-resourced deployments or whatever else, but then it's lack of content or lack of tuning, which I guess you could argue anything is a function of resources, but this is such a big failure point for me. You know, we'll see people put in a SIEM, put in a bunch of effort, and then either go turn on 1,000 rules and have them generate just a ton of alerts And then that is where it ultimately blows up. Or, you know, they'll turn something on and turn it on with 30 out-of-the-box rules and not spend the time to develop the content to actually detect. It's, you know, it's never that place in the middle of a sweet spot of content and tuning. And then I guess third, I would say maybe bad architecture from the beginning, and especially trying to adapt legacy SIEM technology to a modern cloud-based high-volume world. You know, we see people stand up a SIEM project and then go, okay, you know, we've moved X amount of our workloads to these cloud service providers, and now all we have to do is egress all this cloud log data back onto our premises, back feed it into the SIEM. What do you mean the SIEM's tipping over day 3? Or, you know, we just stood up this really great EDR product and we're gonna throw those logs at our SIEM, right? What do you mean we're out of log capacity already in, you know, day 5 of our SIEM deployment?

That's a great question right there. We've got modern SIEMs, we've got old-school SIEMs, we've got AV, and thanks to Anton of all people, we've got this new word, EDR. We talked about this on a previous episode, loyal listeners. What's the future of SIEMs and EDRs working together? Are they going to kill each other? Are they going to find a way to live together harmoniously? What's the path there?

I think they're absolutely going to find a way to live together harmoniously. I mean, I think EDR is probably the single most important technology in your defensive detection Wow. Oh yeah. I mean, hands down, you know, would be the first thing I implement at any company if I'm buying something to actually protect myself.

Before you buy a SIEM.

Not uncommon nowadays. Yeah, he's right. My mind blew up when I first heard this view 3, 4 years ago, and I wanted to argue with this really a lot. But ultimately, I also agreed that the EDR-centric SOCs do exist where SIEM or log analysis is auxiliary. For me, that was a big moment in my life because apparently something very important to me turned out to be not true anymore. Yeah, that you don't log analysis first.

Wow.

No, that's legit.

Don't get me wrong. I mean, I'm going to want those EDR logs into my SIEM and I'm going to want to correlate them with other telemetry sources and everything else. But I truly believe, you know, you can stand up that EDR solution, you can start logging that data, you can start protecting your environment right away. I mean, there's no Question, like one of the first things we do, you know, I think you guys both know, but we do a lot of incident response, right, for a lot of customers, you know, not just those of our managed services, but for a lot of other customers. And the first thing that we do when we roll into an incident response for a non-Siderus customer is deploy EDR, you know. And in fact, even sometimes if they've got an existing EDR, we're going to deploy our top 3 chosen EDR into their environment because it's so valuable and so important and lets us gather so much important telemetry. So, I definitely believe that EDR and SIEM is going to coexist. And in fact, I think it's one of the core technologies that is driving that concept of security being a big data problem. You know, I will tell you, for example, you know, I don't want to beat the SolarWinds Sunburst drum too much, but I think that's just a perfect story of how modern information security with the right preventative and detective technologies were in position to respond to this kind of very serious, very pervasive threat. And, you know, those Sidorus customers that had modern EDR and had a modern SIEM in place were able to know by literally 8:00 AM on Monday, the day that that broke, whether or not they were impacted by Sunburst, whether or not they had the stage 1 payload, whether or not they had the stage 2 payload, because we had all that telemetry. The moment those IOCs became available to the internet, You could go back in time and go, hey, did I have this? And the people who didn't, even people who had EDR but didn't have the EDR logs or didn't have a way to access those logs in any sort of easily effective way, were spending weeks, if not months, trying to figure out if their organization was compromised by this very insidious threat. And that's just a huge gulf to me between security haves and security have-nots in terms of how much time I'm going to spend in incident response in trying to help clean up my organization or not.

Yeah, funny enough, this was one of the first lines, I think, that I've written in the first EDR paper back in 2013 was exactly that. Tasks that would take months otherwise take minutes with EDR. It was mostly around endpoint investigations. It was literally months became minutes for some people. So admittedly, that's quite cool. We had another question about what inputs would be most useful for modern threats. And I think it's very clear to me that you'd highlight EDR as one of the key data sources into your SIEM or into your analysis platform. So any other favorite data sources or maybe favorite features to deal with this type of modern, possibly top-tier threats? Like, what would you really like to have if you are facing an elite attacker and a modern environment to defend?

Oh, absolutely. And in fact, I'll go as far to say I believe you can get 80 to 90% of the value in a list of 5 technologies of all security detection and response.

Hmm, cool.

EDR, DNS, proxy, DHCP to know who something was at what point in time. And then I will go past that to go deception tech.

Deception tech?

I believe deception tech.

No way.

Big shout out to Phynx Canary. Ridiculously affordable, ridiculously valuable. Probably the single highest ROI product in security.

Well, given that it costs, what, a couple hundred bucks per thing, it's not a credit card price. It's like your kid's credit card price, as far as I recall.

It's like your coffee budget price.

It literally is your coffee budget price, and it's ridiculously effective. But there are a lot of great deception technologies out there. There are a lot of enterprise budget deception technologies where you can spend a lot of money and have a really robust deception platform and everything else. But You know, ThinkStack, I would argue, is very robust and very valuable and is that exact coffee budget. But quite honestly, those 5 technology sources, and guess what? DNS, DHCP, everybody's got those already. It's just grabbing that data. Proxy, almost everybody's going to have some kind of proxy, whether that's your firewall or something else that's proxying your connections to go get that. So you're really looking at EDR, which a lot of people still don't have, but, you know, some modern EDR tech and deception tech. And I believe that combined with an ability to store that data and analyze that data, and that's the vast majority of what people need to be able to respond to this. Now, I will say the one thing that I'm not touching in there is phishing-related email-specific data. But I will say for a lot of people, a lot of the email-based stuff has gone past like the ability to analyze. And so it's really more emails become like a really advanced technology to just block stuff as opposed to be like a log source. And that's why I would not label it that. I will absolutely say, you know, email security products, absolutely critical in that, but I actually don't find them to be a super telemetry source for detection and response. It's much more of a put in email tech to knock down the bad stuff before it ever gets there. And then I'm going to use more proxy and DHCP DNS lookups to find who actually did something with that phish.

By the way, as a funny interruption before Eric goes into the answer, when I started looking at this back in my Gartner days, I think it's maybe around 2012, we realized that for some cloud providers, you have to call them and say, can I have logs please? And they would like ship them. So this is like the early era of the IR in the cloud. Then you have to call a human and say, can I get my logs? And they're like, yeah, sure. We're going to drop them in the envelope and FedEx them or something. I don't know how they send them. We went a long way from that time. That's what I'm saying. So there's a lot more telemetry now.

I love the idea of FedExing logs.

Oh, absolutely. Never underestimate the bandwidth of a station wagon loaded with backup tapes, right?

So let's shift gears a little. This is the thing I love talking about on the show, and I talk about with users all the time, and you'll think about what I do for a living and find this unsurprising. But let me ask, what's different about threat detection in a cloud environment? What's different about investigation there? What's different about response there? What changes with the move to cloud? We've talked a lot about on-prem. What about cloud?

Big question here, right? You know, cloud, cloud sources, what is detection and response in the cloud in general? I have kind of a really specific take on this, which, you know, while ultimately a lot of people will say, you know, the cloud is just somebody else's computer, and sure, there's a way to look at that, I will say Cloud is not just somebody else's computer. Maybe the concept of the cloud is, but the biggest thing is the cloud service providers have vastly different services and service architecture than just your traditional data center. And in fact, it's the people who look at the cloud as, oh, this is just somebody else's computer and I'm just running a Windows VM at a cloud service provider that are doing the cloud wrong. So while I would say threats in the cloud are not typically that different from on-premise threats, I mean, it's still user-based attacks, it's still IAM attacks, it's still resource attacks in the cloud, it's still malware, it's still compromise of administrator credentials and other things like that. The architecture, the services, even quite honestly, the acronyms are so vastly different. And in fact, it's one of the reasons, you know, we like to say, sadly, that most of the other service providers can barely spell cloud, let alone know how to actually secure it. We take these sort of traditional security guys and you put them in front of a lot of cloud logs coming out of the various cloud service providers. It's all things that you're not traditionally familiar with, and it does definitely require a different level of expertise. I will say, you know, for sure, the cloud has its own attack surface. We are very big fans of looking at a cloud service holistically, you know, whether that's with a cloud management platform, You know, something like Palo Alto's Prisma Cloud, or, you know, each of the cloud service providers have their own cloud-centric tools. And I'm really hoping, quite honestly, that Google does a lot more with their cloud-centric security tools to make them more and more multi-cloud.

Oh, yeah?

Oh, absolutely. Palo Alto Prisma Cloud is basically the 800-pound gorilla in this space, mostly because they end up buying everybody who remotely competes with them.

They have had an open pocketbook.

Yeah. So, it'd be really interesting to see a more agnostic solution approach to cloud management platforms, you know, although a lot of people would argue the CMP tools are more like vulnerability management and compliance best practice and other things like that than specific detection and response. But I kind of look at that as a very complementary thing. You know, if you have a misconfiguration in your cloud environment or your attack surface is widened because you've misconfigured something, you know, that is going to fall into detection and response. And it's something that can very much inform that.

Which reminds me of the Orca episode, right? That same exact theme came up on our previous episode with the team from Orca, right? The cloud security provider, right? Exactly the same with—

You've got the same question of left hand and right hand and how do we as a cloud provider help them know what the other is doing?

That's probably my single most concise answer though, is the, you know, in terms of what's different about investigation and response in the cloud is it's understanding those kinds of services and acronyms and cloud-specific architectures that maybe your traditional security or IR teams might not understand. And I guess to even give you a sad story around that, you know, like, the obvious easy example is, you know, somebody running a microservices-first environment is obviously very, very different than a traditional on-prem network environment, you know, endpoint-based environment. But still, a lot of people don't even understand that. I mean, we fought and fought and fought with somebody who was absolutely convinced that they needed a way to put modern EDR on their microservices. We're standing up these Kubernetes things, and we need CrowdStrike installed on them, obviously, because, you know, policy is we install CrowdStrike on every computer. And so, like, how do we install CrowdStrike as part of our CI/CD pipeline? And it was What? No.

That's where hilarity ensues, basically. That debate is where you really— I'm not going to say anymore.

I would end up with a lot of holes in the walls of my apartment.

So here's the fun one. That's another question that I get asked occasionally, and I usually cringe before I answer. So let's see if Eric would cringe or not. What is your view of the current and ongoing frenzy about AI? And I'm going to air quote AI/ML. For security? Give us your short but hopefully funny take on the role for AI and ML in security.

I love this, Anton, and it's definitely a cringe question, but I will say it's much more the cringe of every time I sit through yet another vendor presentation that has all this fancy AI/ML unicorn, what I would say has historically been very big snake oil in so much of this that it's become a bad word. AI and ML is not snake oil, but I would say most vendors' use of the terms AI and ML are. ML is really good for certain use cases where it's really mostly statistical analysis, but people want to change that statistical analysis buzzword to be machine learning. Fine, right? You know, I'm doing volumetric analysis of a thing. Sure, that's machine learning, but it's really statistical analysis. You know, trained models can be very valuable if you put in the time to train them and the model is straightforward enough. You know, we've got an insider threat-based solution that adds on to our service offering and adds on to our SIEM that uses Bayesian inference and build those kind of models, train those kind of models. You can do a lot with them, but historically, unfortunately, most of this in security today is snake oil, but I guess I also have that rather curmudgeonly approach that I maybe am famous for that I would say sad, too high percentage of security products fall into that same bucket as well.

But we also probably agree that the long-term future on this is positive. Like we are all hopeful that by 2030 we will have effective AI/ML for many security problems. It's just that today's stuff tends to be in this category, right? Like we're not arguing that the long-term trajectory is pointing at that. We are arguing that today's state is not magical.

So we really hate leaving listeners empty-handed. What would you point people to to learn more about, you know, trends in SIEM, managed DNR, cloud security? And the important caveat is that Anton's blog gets mentioned enough, so you'd get zero points if you mentioned Anton's blog.

That's funny. Honestly, my biggest single source these days is the collective hive mind of InfoSec Twitter. It's how I tend to use Twitter the most is You know, as a kind of newsfeed aggregator, opinion aggregator, everything else. There are a couple really good InfoSec lists on Twitter of, you know, InfoSec hive mind, if you will. I don't want to shout out any one specific list, quite honestly, because I believe the way to do Twitter right is to, you know, kind of curate your own list based on some stuff you see. I will always encourage people to follow some people you might not like or you might You know, you create too much of an echo chamber otherwise. But quite honestly, rather than point anybody to, you know, the obvious Anton's blog or some specific blog that I think everybody probably already listens to or follows, I would really encourage you, take an hour, go to Twitter, punch in a couple different hashtags, follow a couple different people, and look at who those people follow. I would not throw myself up as the bastion of, like, thought leadership on Twitter by any means. I like my follower list, so feel free to come, like, check my list as a place to start or check somebody else that you respect. And yeah.

And Eric, what is your handle?

I'm Performify, so perform and then ify, so like modify but with perform as the base of the word.

With that, I think we're just at time. So listeners, thank you so much for joining us today. It's a pleasure to have you on the show, Eric. Performify is that Twitter handle. You can follow this podcast at twitter.com/cloudsecpodcast, my co-host @anton_chuvakin, and myself @timpeacock. Tweet at us, email us, argue with us. If we like or hate what we hear, definitely invite you on the next episode. I promise joining the episode is more fun than fighting on Twitter. And so with that, thank you so much for joining us today, and we'll see you next time on the Cloud Security Podcast.

Security. Security.