EP263 SOC Refurbishing: Why New Tools Won't Fix Broken Processes (Even With AI)

Security.

Hi there.

Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your host here is ever myself, Tim Peacock, the senior PM for, well, a bunch of stuff in Google SecOps. And of course, Anton Chuvakin, a reformed analyst and now senior staff in Google Cloud's Office of the CISO. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You follow the show, argue with us and the rest of our Cloud Security Podcast listeners on the Google Cloud Security Community page. If you're watching this on video, which we now offer at youtube.com/@cloudsecpodcast, please do subscribe so you don't miss future episodes. Anton, we've got a really fun guest today, and it is an episode almost geometrically full of tangents. Is that what we ended up doing today?

It had a lot of tangents, but it's also a really solid, really insightful detection and response episode.

Yes.

And to me, it has an almost perfect blend of magic and utility.

Yes.

Because it has some things you can use, it has some things you can change your thinking, and it has a proper amount of, you know, jokes about security appliances because you got to have those, right?

It also, I think, had such a gem right in the middle where Danny— well, I won't spoil what Danny says, listeners, but he had a perfect gem, like smack in the middle of the episode. So I hope you stick with us through the episode. I hope you enjoy it as much as we enjoyed recording it. And I got to say, the book recommendation at the end, one of the most interesting recs we've gotten in quite some time, I would say. So without any further ado, let's turn things over to today's guest. Today we are joined by somebody I've worked with for many years now. We've got Danny Lyman, the Vice President of Threat Detection and Response at Fiserv, joining us. Danny, thanks for joining us. Thank you for joining us. I want to start off talking about this challenge that a lot of people have of bridging the process disconnect between high-level board objectives, what the CISO dreams about, and the reality of life on the ground when you're actually out there patching things and solving problems in reality. And so, how in your organization or how in your mind is the right way for people to bridge that gap? and translate executive dreams and board goals into reality?

Okay, so we approach achieving objectives through using people, process, and technology. A lot of the times when we basically get initiatives from up high, let's just say CISO direction, teams often will go towards one area. Hey, I have a big team, I have this really cool technology, and oftentimes, Process is ignored in favor of either, let's say, a talented team or what we would say cutting-edge technology, right? And I believe what ends up happening is that there lacks coordination when we start talking about what processes need to be executed to accomplish our mission, what tools are we using, and do we have the right people doing that. So, I find a lot of the times that many people gravitate towards embracing technology only, and technology is wonderful, right? But if we don't have the right process, then it becomes a challenge to perform things correctly. And particularly from a scaling perspective, process is really what drives all those things together. So I think a lot of this stuff kind of gets back to Maybe some oversight or a lack of understanding on what is the right process to bring all these things together, particularly when you're following a federated model with a SOC, because there is a significant amount of proper coordination that needs to be happening, as well as moving the outputs of those processes as decision points for either an individual analyst or multiple teams.

It's fun you mentioned federated SOC because Anton wanted to ask exactly about that.

I actually don't.

No?

I will get there, but I want to—

Oh, you'll get there.

Touch on what Daniel just said, actually, because lately, actually, literally yesterday, before the day of recording, I was like super frustrated and I was tweeting about people who show up and say, we have SOC transformation. And I asked them how they did it and they said, oh, we threw away our old appliance SIEM and we put a new SaaS SIEM in. And I'm like, yeah, that's great, but why do you call it SOC transformation? Oh, we transformed our SOC. We do things differently. And they describe things that they do and they're exactly the same. It's just that the But SIEM product is beautiful, shiny, has AI, is in the cloud, and everything's much faster, and everything is better in the tool. But what they do is the same. And I'm like, yeah, that's not SOC transformation at all. That's replacing a product that did not work with a product that does work. But you can still be constrained by the process quite a bit. You can still have a very compliance-minded SOC, not an engineering-led, not modern. And they look at me kind of funny and say, what did you just say? We have a modern SIEM. And they go back to that. So, How do we talk to people like that? What do we tell them if they hear what you said, Danny, and what I just said, and they say, we bought a better product, they double down on tech upgrades?

Yeah. And I unfortunately feel like that's a very common move, I think, with companies these days, right? We bought a cool tool and we're advanced now, right? And when you start thinking about the way that attacks actually happen today, Compared to the way that attacks happened, let's just say in 2005.

Mm-hmm.

Right? The way we handle them, the way that we investigate them is very different, right? Because your process changes a lot with modern technology. If you don't modify your process, then the question is, are you really embracing the advantages that the new technology gives you?

Mm-hmm.

Right? So I think that there is something to be said there. And as far as how to solve that, that's a good question. Because solving that is really challenging when you start talking about the size of the company, the budget of the company. I would even say to a lesser extent, what is really the vision? There is a lot of concepts of the SOC basically being an audit checkbox, right? If that becomes—

I cringe when I hear it. I'm like, can he stop?

But there are security teams like that. Right. So the question is, how do you educate from the top down? And then if the top basically understands what the right goals are, what modern process looks like, how do we then filter that stuff out into process, right? Because when you start talking about your experts, your ninjas can't handle every single alert that fires into a SOAR.

Mm-hmm.

However, you can develop process and guardrails to ensure that everybody underneath you is doing things exactly the way that it's intended. And then that becomes like muscle memory, right? That's how analysts come in the door and they get better, is they learn how to do it right the first time, and then they gradually get better over time. So what you end up doing is building a better staff through using great technology and great process.

Yeah.

Their knowledge basically is going to increase, and then you have what I would say a trifecta of good technology, Good process, and now you're building good staff. Then the trick you have is how do you retain them? Because, you know, that tends to be the big thing with cyber. When people get better, they go up and out, as they should, you know. But yeah, I really feel like that there's something missing when we start thinking about embracing technology and incorporating modern process with modern technology.

Yep. I think that the summary of this is that if you see people leave the process intact and only modernize the tool. They essentially follow— they drive their EV the same they would drive horse and buggy 200 years ago. Like, I don't know, maybe my metaphors kind of suck today, maybe. But I see that procedurally, they do exactly the same steps, but maybe some of the steps are faster. But nothing changes. No new capabilities, no new grand capabilities. I mean, to use a self-serving example, when Chronicle was born, People realized they can do a lot of searches and they would be sub-second. By the way, not paid by Chronicle here, but the point is that if your search takes 2 minutes to 5 hours, certain things are out of bounds for you. But if searches always take sub-second, there is an opportunity that opens up. If you don't use this opportunity, you still assume searches are slow and you behave as if they are, you would not modernize.

Yeah. Agree.

Yeah.

Okay, so 2 actual questions we want to ask.

Sorry, I'm stuck on this image of Anton's garage where he's got his Tesla parked on top of like, uh, some sawdust and some straw, and he goes out there every night and with a shovel he's looking for the, the leavings from the Tesla, but it's not there. So you said something that I wanted to pick on, which I thought Anton was going to pick on, which is this federated or sub-sock model. Can you talk about some of the, the challenges you've seen with that, or advantages you've seen with that and why you would want to have a world that looks like that?

But we need to define it first before we dive in. We need to define because—

Yeah.

What does that mean? Yeah.

Are you talking about the federated model?

Yeah.

Yeah. Yeah. It's just this concept that you have people that specialize in certain areas, right? So you can keep them specialized in those areas. And I think there's multiple ways you could do that. I think academically you would look at a federated SOC, meaning that you have these, like you said, sub-SOCs that are basically monitoring things the same way, and that's locality-driven, right? So you theoretically would have, you know, if we had an office in Hong Kong, you'd have a SOC in Hong Kong. They follow the same procedures, a SOC in Brussels or whatever, right? I also view it from a technology perspective where you can have people that prefer EDR, and then you have people that are your network ninjas that are living in NDR, right?

Mm-hmm.

And then you have other people that are just, very good with IAM and they say IAM-type tools. And these are the types of things where you are still having what I would say experts using the right tools, looking for the right things. But the big thing is really the coordination. If you see things related to IAM, do we miss that correlation between what is being seen from a network perspective? What are we seeing from an endpoint perspective? And how does that translate from an identity perspective?

Hmm.

Right? That's the part that becomes challenging when you start talking about a federated model, is ensuring that everybody is seeing the same thing. So I think that there's multiple ways that you can, I guess, define federated, but I would say that the academic approach really is that same SOC, same tools, and it's really like location-driven. So things are localized instead of Having, you know, one US-based SOC do everything or one SOC in Germany do everything for a global company.

I think that it's also sometimes federated, applies to federated storage when people have, you know, little data puddles of data, data pools, you know, data lakes occasionally. And then they try to say, and now we're going to do a search, do some kind of analysis, do some hunting, but across many data sources, no data storage buckets. So I was, you're mostly talking about the team being federated, not the tech being federated, correct?

Yes.

Yeah.

I think team would be ideal, right? Because then you have a lot of people that understand everything and they can work with a full stack of tools. I often see, I would say people gravitate towards, well, I'm just going to work in EDR. Yeah. When we also understand that there's a lot of attack surface that EDR doesn't cover. Yep.

By the way, as a complete tangent, why is this such a hard message to some people? I mean, I've met too many EDR-obsessed. And maybe not lately, but a few years ago, I've met people who say, oh, it's time to throw away my SIEM. I have EDR from the top-tier vendor, and they can detect everything. And I'm like, you mean a compromised router? And they look at me again with big round eyes like, what do you mean compromised router? Like, they're out there. Or maybe like one of those famous security appliances that get owned left and right. You're going to do EDR on that, right? Sorry, Tim, I didn't mention the vendor name. Think about it. How much is the name of the show?

No, no, no. You're doing great at not mentioning vendor names. But listeners, don't you think it's a little ironic that Anton, who invented the term EDR, is complaining about people liking EDR too much?

I mean, obsessing really. Liking too much, I'm fine with. But obsessing about it down to, I don't need logs. I mean, I kind of like logs more than EDR, to be honest, but the EDR obsessives are a danger to the planet.

Yeah. And I would argue, when you think of the days when it was AV, and then AV went to the EDR area, I could understand how EDR was really a winner because most of the attacks were really favoring hitting domain controllers, databases, things like that. But particularly with the popularity of cloud, I mean, the attack surface is really spread so wide where you really can't trust one single control. And that really is the most important thing. And I think to get back to what you're saying, Anton, I wonder if it's a business decision, right? The investment in EDR. And listen, EDR technology is wonderful. It is needed. Nobody is basically discounting what EDR does. From the industry, from a detection and response perspective, but it's not the only thing, right? So I think many times they take something that's good and say, you know what, this will work most of the time. This was a big investment and it's working well for us, and perhaps maybe it's a business decision. But when you sit and talk with people, I find that everybody says the same thing. Oh yeah, it doesn't cover everything. So I wonder if what you're experiencing when you talk with people is likely due to, you know, business decisions and ROIs for the most part. We spent this much money on this solution, and we're going to make it work, and I'm going to, you know, hang my hat on this technology instead of thinking about, okay, how can I spend more money somewhere else?

It's too big to fail. The security leader can't spend all that money and then not say it was great. Of course. I love what you say, though, that you can't trust one single control. I feel like, as Anton would say, that pays for the ticket right there.

That's right. But also, even this is EDR. And I know it sounds like we are talking to people in the back row and the salespeople, but you cannot trust a single tech, even if it's market-leading EDR from a very famous vendor who is like top tier in everything. Still, the logic applies. But I have another tangent, which Danny might find kind of interesting, is that for some people, EDR is The detection cost, the cost per successful detection is lower for EDR in their minds than for SIEM. EDR seems like less work. EDR seems sometimes like crisper signal. So they confuse better per rule ROI to this is a better detection tech. Am I hallucinating this or am I going on a Dark tangent. Or am I, am I reasonable?

I don't know that it's, that it's a hallucination, but I wonder if that's really more of a— the output is unintended, right? So maybe people are communicating something that they don't really intend, but I don't know that I've heard anybody approach it that way. The cost per detection. It's interesting, but I've never really thought about that. And I'll be honest with you, I don't know that I would be satisfied with that.

Yes.

Because I'm really looking for completion. Right? And I understand, again, I think there's a benefit. There's a selling benefit there. And I think that comes into that, but it still goes back to, but does it really detect everything? What it can detect, it does very well, but it still doesn't detect everything.

Yep.

So, It still gets back to there isn't a silver bullet when it comes to detection. You have to embrace multiple technologies and find ways to incorporate those together.

And this is kind of the law of the universe. This is not a stage. And I think this is kind of like, I hope the AI system that picks clips would pick this one. Now I'm going to address our— because it's like, it's not a stage that there is no one detection tech. It's kind of a fact of detection life that improving EDR by a factor of 10 would still make it not detect attacks against the something-something appliance, which is compromised, or a router or something.

This gets back to what we were talking about with Royal Hansen, actually. There's not one type of cell in your immune system. You're never going to see an organism evolve to only have T cells. You're always going to have a multitude of T cells and B cells. And a host of other techniques for detecting invaders.

By the way, we are failing to mention AI, and it's been 20 minutes about so much. So maybe, Tim, go ask us about AI.

That's a new record for us this season, to go 20 minutes without AI. You're ruining it, Anton. This is terrible. Gosh. What do we even want to say about AI? As you're thinking about AI—

Has it changed what we just discussed?

Yeah, has it changed this for you? That's a great segue. Has it changed how you're thinking about this?

I think it does, right? Because I think the problem When you start talking about we have multiple tools that can cover everything in an attack surface, the telemetry becomes a little crippling to get in front of, you know, when you start talking about early detection, early containment, right? So I think AI could potentially have capabilities to link all of these signals together so that you can see them in a full stack, right? Because I mean, we have to be honest, attacks happen. We can't ignore the OSI stack, right? They happen from layer 0 or layer 1 all the way up to layer 7. So thinking in terms of, you know, full coverage for the entire attack path, could AI theoretically get all that telemetry together from multiple tools and then create what we would say a super rule, a super detection rule, right? I think there's possibility there. I just don't know that that has been done at scale in a way that is marketable. But I think that there are a lot of technologies, and I think there are a lot of efforts that are definitely working towards that.

Yes. I mean, as long as it actually can touch the data, right? Because this has been another favorite pet peeve is that you just said, if this data of many types goes to AI, AI may see things that humans may miss, whatever. But it's possible that certain pieces of data would not make it to AI because maybe they're in a legacy system or they're in a particular island that's hard to get to.

The puddle is too shallow to access easily.

Like the puddle, yeah, or maybe it's behind the fence or whatever, or the team just doesn't want to give you the data. That happens, right? So I think that starts to increase my skepticism a little bit that AI would fix the problems we have because it needs to see the data.

If it can be exposed somewhere safe, there's potential.

How do you expect AI to change our jobs? sort of industrial calculus around data centralization versus data federation? Do you think it will change it, or do you think we're still going to want, for like good reasons of reliability, to keep it centralized?

That sounds like it's a completely different podcast to discuss.

No, that's very fair. I feel like it's going to go on a very far tangent, but it's a fun question.

But I don't get to have tangents? Anton, you had like—

Yeah, you— okay, fine. Go enjoy your tangents and co-tangents.

We don't have to—

we don't have to go down that way.

No, I will say this.

The amount The amount of data that is being consumed by cyber teams comparatively—

Agents.

It's crippling. Yeah, right. When you start talking about the scale of where to store all this data, we have to protect the data. You have to make sure that only certain things have access to the data. You have to protect the routes to make sure that the data gets to where— so there's this whole concept of data collection. data lakes, data management, that is a completely different topic that absolutely relates to what we're talking about because you want to have that data available. We can say the SIEM is the right place, but there's a lot of telemetry that we typically would not put into a SIEM, right? So you lose a lot of visibility there because not everything is built—

What's your favorite example? Sorry for interrupting. What's your favorite example of telemetry that's useful but usually not in a SIEM? That's a fun question.

That's a great question.

Application logs.

App logs, sure.

Those nasty Log4js, or is it something more modern, or what?

I would just think in terms of, like, I don't want to be too old school, but, like, JBoss logs, right? When you start talking about application logs, how is something interacting with a system, right? And particularly from an ATT&CK perspective, there's a lot of visibility there, but the data is so It's not difficult to normalize.

Inscrutable, for instance.

Oh, I'm sorry. It's very difficult to normalize.

Yes.

Yeah.

Yeah. So there's a lot of challenges there, but there are definitely nuggets of gold when you start talking about application logs.

Yes.

Yep.

That's a really good one.

I have another question that came out of something that you said earlier that I think might be 3 or 4 podcast episodes worth of discussion, but you didn't love Anton's concept of a cost per detection. What are the, like, DNR org metrics that you think are most valuable that people maybe don't always think of?

I mean, honestly, it's the mean time to contain. I think that's where the most value is, right?

Hmm.

Because, you know, I think there's always gonna be challenges related to detection time, mean time to detect, but your response time is very important. That is extremely important. There's a lot of things related to coverage. So when you start talking about, you know, mappings to MITRE ATT&CK, you know, I think that there's aspects there that are valuable mappings to, do I see everything in my organization? Do I see all the different business units? And then when you start breaking it down into processes, maybe different security zones or whatever, it's really this concept of, do I see everything? Do I have the capabilities to detect everything?

Hmm.

But the most important thing is, do I have the capability to contain once I can confirm that something is an incident? I think that's the most valuable thing, right? And I think that's where most incident responders really hang their hat. Is on early containment time or quick containment time. That's where I think the biggest value is when you start talking about SOCs.

And is this the MTT, MTTC? Mean time to contain is basically to me a subset of mean time to respond. It's just a particular response. It's the first response. Yeah, that's what I'm hearing. I'm hearing like a younger brother of MTTR.

Yes.

Or maybe part of the family of MTTR is probably more accurate, which is the time to First mitigation against what we observed.

Yeah, MTTR, I think, would encapsulate. And you're right, because the detection time and the containment time are all subsets of the response time.

Okay. That's a useful thought. And I mean, I don't mean to be snide. I mean, obviously, it's a useful thought if you do it, but it is actually a useful thought for the audience that obsessing about MTTC for containment focuses your mind on the things you can achieve and achieve faster and better. And just spouting general things like MTTD, D means what, early indication, then you triaged it and didn't do anything. Like, I can see why mean time to contain is a good one.

Anton has a fun, I think, 1990s-inspired question for us, which we will ask. And then, unfortunately, I will have to close after that. But Anton, take us back to the '90s, please.

Yeah, sorry. It's the— we touched AI, and now we have to have the kind of like the 1990s mention. And the 1990s player here would be Microsoft Excel. So no, not with Copilot, actual Excel from the '90s. The idea is that a lot of tools that we use for IR, for detection, are still very much, well, from the '90s. Lately, I talked to a bunch of people about detection use cases. And Excel made a very popular appearance in how people actually track, manage, deal with detection use cases we build. They may have some kind of a grand vision of detection as code, but the actual tracking/management system they have is Excel. So, what do you make of it? What do you make of the fact that perhaps for some problems where people are showing up with AI and magic and whatever, for other domains, we are stuck in the '90s?

Yeah, well, I mean, you can take the most skilled investigators and even today they still use Excel, right? Why?

But wait, wait, you do mean it as a positive or a negative? I'm— you're giving me a neutral tone here and I'm like, yes, I agree.

But like—

So I think the way that I would say that from an industry perspective is, can we as an industry create technology That can replace the need to just assume that the only way that I can manage this is with Excel, right? So part of me thinks that it's a tooling thing. Then I think it kind of gets back to, is it really a comfort thing? Is it an issue of laziness? If we have technology where we don't have to use Excel and we can use something that is cleaner, easier to manage, centralized, or something. Is it really a laziness thing, or is it that nobody really cares what you do, right? They only care about the output, and you use whatever tools you have. I would challenge the use of it and say, let's be better, right? However, I'm not discounting the fact that Excel saves a lot of people when it comes to investigations or managing, you know, all of their stuff. Excel essentially is just a very, very minuscule database. You know, so we're certainly not going to toss out databases, but I think that there's ways that we can leverage UI and UX that sits on top of data stores to be able to manage our work better and more effectively.

Yeah, I think the Wiz sweater should make another cameo appearance at this point.

I have a— I was thinking what you were saying there, Danny. Is it possible that we're still using Excel because I hate to say it, it's actually just a good way of interacting with a small database.

It is. It's much more simple.

Yeah. Maybe the answer is we use it because it's good.

Yeah.

Yeah. Sad thought. But I have to do the thing I hate doing, which is ask our traditional closing questions. Do you have a piece of advice for helping people who lead large, complicated DNR organizations do a better job of that? And then 2, do you have recommended reading? And that reading could be anything except Anton's blog.

Yeah, that's a rule.

That's a hard rule, by the way.

So, as far as advice, I think it still goes back to focus on the process. We have great tools, and I think organizations that are buying these tools, learn how to use them the right way and embrace the new technology, but don't lose sight of what your mission is. So I think it still gets back to how do you accomplish what you're supposed to do and really focus on those areas versus how do I buy new tech to solve cybersecurity problems when attackers basically are learning how to maneuver around that new tech. So that likely would be, I think, my number one point of advice as far as reading material. This may not be popular, particularly given what I just said, but I tend to not read technical books or books about cyber or whatever. I kind of incorporate that into the way that I work. So I try to read or listen to books for enjoyment. So the last one that I read was In My Time of Dying by Sebastian Junger.

That's quite a name.

Yeah, it's interesting. And I get this from my wife because my wife is the reader, right? And a lot of the time she says, hey, read this book with me. But, the book, he was a reporter. He'd been in Afghanistan, all this stuff in war zones. There's so many things that he'd been through that are life-threatening. And, this guy gets what's called an abdominal aortic aneurysm, which has an almost 0% survival rate.

Oh, wow.

So, it talks about the whole concept of him going to the hospital, them not understanding really what the problem was. Certain surgeons just magically understood what the problem was and began the The triage process. And that whole thing really reminded me of what investigations are like when you basically have an intrusion or malware infection or something. You'll oftentimes don't know what you're dealing with. So reading through the section where they were triaging and trying to figure out how to make a determination of what is wrong with him, how do we save this guy's life, there were a lot of tangents, I thought, to the way that we operate in cybersecurity operations. So I, I found those tangents to be very interesting. And then there's a lot of things too when you start talking about the technology related to how do you actually recover or repair an aneurysm that is inside your body without chopping the whole body up, right? There's very interesting sections within the book where they're talking about the advancement of that science.

Hmm.

And again, I think about the technological advances that we're talking about here, and at some point somebody basically had to be the first person to try something, right? And how scary that, that is. And then now we have all these things that basically can heal people. So there were some tangents to cyber that I thought were certainly interesting, and there were also a lot of like philosophical questions about the afterlife. What happens when you're dying? And he goes into all these things about hallucinations, right? Yeah, if we want to call them hallucinations, but like that moment—

Visions.

Yeah, everybody talks about visions before you pass, right? And he's trying to— his father was a physician, a scientist, like a physicist. So thinking in terms of how do you explain what's happening scientifically, it was an interesting conversation. It was thought-provoking, let's at least say that.

Wow.

That's one of the more interesting book recommendations we've gotten on the show in a long time. I look forward to reading it. Danny, thank you so much for joining us today.

Yep. Thank you so much, Tim. Good seeing you. Anton, nice talking with you today.

Perfect. Thank you for being here.

And now we are out of time. Thank you very much for listening and of course for subscribing. Please subscribe so you can get new episodes piping hot. And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcast. You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us, and if we like or hate what we hear, we can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast.

By Google.