EP180 SOC Crossroads: Optimization vs Transformation - Two Paths for Security Operations Center

Security running. Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Timothy Peacock, the Senior PM for Threat Detection and some other stuff in Google Chronicle, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on our Google Cloud Community page. You'll find the link to that in our show notes. Anton, what are we talking about today? Is it modernization or transformation?

Well, that's how, in a few moments, you'd know that we sort of tricked Tim a little bit with the question about whether we are transforming and/or optimizing your SOC. But the point is that we are trying to evolve and make the SOC better for the future. And there are multiple routes. Some of them are more transformative than others, but at the same time, some of them are much harder than others. Frankly, for many SOC analysts, I feel like Small change is much easier than culture change. A small change in technology is easier than big change in culture. Let me put it this way.

Well, that's for sure true. Small changes happen every time I open my phone and it's updated software, but culture changes take forever.

Indeed, indeed. And that's what we would observe in the episode when we talk about how we can either fix the minor problems in the SOC or how we can sometimes transform it to some new model, which would of course be magical but require work.

And now, listeners, before you start to think that this episode will include observations as salient and stunning as cultural change is hard, we actually have our most interesting comment towards the back end of the episode. So if you're somebody who doesn't always listen all the way through, I encourage you to stick around for this one. And with that, listeners, let's turn things over to today's guests. Today we are lucky enough to be joined by Mitchell Rudol, specialist master at Deloitte. Can you believe that title? And Alex Glowacki, a senior consultant at Deloitte. Alex, I'm sorry you didn't get the awesome title. We were discussing before we started, listeners, you know, should we call him Special Dragon, Master Dragon? But I think with that, I want to dive in. We're talking today about a paper talking about SOC optimization. And, you know, this is to me such a critical topic because we have SOCs around the world, and nobody thinks, oh yeah, my SOC is the best, my SOC is going to catch every bad guy. And so when we think about optimizing and transforming SOCs, what are we talking about? What's the point of this paper? Where are we going? And how does it help improve security outcomes for people?

Yeah, I could jump in here, Tim. So like, if we really want to cut down to the differential pieces, it's optimization or transformation.

Oh, it's neither or, one or the other.

Yep, it's definitely—

Yeah, it's like the essence of the paper. Sorry, Tim, we set you up a little bit because we said—

You set me up.

Or—

Well, tell me about the or. What's the trade-off?

So think about it, right? You're either doubling down and making sure that what you have today is performing at its optimal pace. You're getting the most value extracted out of your technology. And if you're looking at transformation, you're more so leaning towards the next-gen technology, you want to rip and replace, you're looking for a migration. And the key difference between both of those are what technology is used and the people stay the same. That's the most important part. And in both of those cases, there is some risk involved, right? So you need to factor in different types of pieces in your organization. Are you willing to change? Are— is there anything going on where you could still keep the lights on? Are you able to focus on The technology, the people, or do you need to understand that you got to cut costs and look at your budget and time? So at the end of the day, it is a big problem that all security operations centers have today, but you have to make a decision at one point or the other.

I would actually sneak up on the guests and remind the audience that I was also writing the same paper. So we're all 3 authors. So I'm kind of interviewing myself a little bit. But the point is that we were stuck with one thought sometimes. is that everybody kind of on paper claims to want to transform, but some people neither need to transform nor are able to do it. And for some of those people— and actually what's worse is some people need to transform but they're unable to.

The first step to transformation is admitting that you need to?

I'm not quite going there, no. The point is that for some people the gradual improvement is the right call. And improving what they have is not automatically bad or wrong. Maybe in some situations, you really do need to focus on optimizing what you have. You don't destroy the old tools, buy new ones. You don't fire the old people, get the new ones. No, you actually gradually grow. But for some people, that is just flat out not an option. They would fall behind unless they transform.

How do you know whether you're one or the other?

I think that's question 4, and we're only on question 1.

Oh, shoot. Sorry. Well, you should ask question 2, and we'll get to that later.

Okay. So one thing that came up during the work on the paper is that alert overload, or too many alerts, too many alerts don't require action, is still a major challenge for security operation centers. And this sounds very 2003, I don't know, but it's also 2024. So what are some of the best practices that work today in 2024 for reducing alert fatigue and improving the signal-to-noise ratio with alerts?

Yeah, I can take that one, Anton. So as you mentioned, yeah, it's a 2003 but also 2024 thing. You know, alert fatigue has been, is, and I would argue will continue to be a problem for as long as security operations exist.

No.

2043?

Really?

Well, 2143.

Yeah.

I mean, because at its core—

All right. You just earned yourself an invite to season 20 of the podcasts to see whether it's still an issue.

Well, if I'm still alive, we'll see. But at its core, alert fatigue is not a technology issue, right? So we have a lot of crazy cool tech, a lot is improving over the years, but at its core, it's a process issue. And it really is stemming from a lack of TLC in that, you know, continual cycle of building detection content. Do you have continual tuning and reviews? And, you know, mitigating that is easy in theory, right? Just have more time and more people. So as we all know in the real world, that's hard. It's really hard. So it'll continue to persist as an issue. But that being said, there are definitely a handful of leading practices, to your point, Anton, that can help alleviate this. A lot of them are pretty straightforward and I guess common sense, but some that come to mind are, you know, the first is vet the content that you're building. Seems obvious, but a lot of people just go and turn on content, you know, if you're getting it out of the box from your WAF or EDR vendor. Whatever it may be, they'll turn it on, it alerts, and there's really no thought of the downstream impacts. You know, the consumers or the people actually living day to day with the outputs of these alerts are not the people turning them on. And there's really, you know, a lot of friction there.

Hmm.

So, while it sounds simple, one very simple best or leading practice is just, do you have a response action for the alert that you're turning on? And if the answer is no, maybe don't turn it on. I know, wild. But, you know, the second would be to capture metrics around—

I think before we go to number 2, I think I want to double-click on that because, like, to me, that seems so obvious that the people creating alert volumes should be responsible for dealing with alert volumes. Otherwise, you have a total misalignment of incentives. And at Google, the people who write our internal rules—

That's because you work for a company called Google, right? Yes, it is.

I was going to say, the people who write our rules are the same people triage their output. But again, I forget sometimes, oh, where do I work? I work at Google.

Yeah, no, it is easy to forget. And kind of in our roles, we get to see a little bit of everything, right? And most places don't have that luxury, frankly. And we can get into kind of the people and process transformation that exists too. But from a technology standpoint, that oftentimes exists as silos where, to your point, the analysts triaging the output of an alert oftentimes aren't the detection engineers writing them. Or the tool, you know, teams that may sit outside of the SOC that are maybe owning the EDR. But, you know, that being said, to kind of get into additional components that can, you know, help alleviate some of this alert fatigue, I'd say the second piece is really capturing and maintaining metrics around what you're generating as alerts. Again, fairly obvious, but while every executive would love the fanciest, shiniest dashboard, it doesn't have to be—

Right.

crazy or exotic. It can be an Excel sheet or a text document. Just understand where are your alerts coming from. And if you're dealing with a lack of people and time to address that, it can help to prioritize, you know, where are you getting most of that volume from? What is actually leading to high-efficacy alerting? You know, are they true positives or just auto-close? That's the definition of alert fatigue, right? You get it, you close it, it's not worth it. So understanding where you can address things upstream is really critically important.

So that sounds like a relatively simple spreadsheet, like, count of alerts by alert source. That's enough as a starting point, though, for a lot of people, though, right?

Absolutely. Where is it coming from? And then what is your feedback? You know, it could be simple things. Are people spending time triaging? Is it actually a real issue? Like, are other teams getting involved, or is it just auto-closure? And even starting with something as simple as that can have tremendous impact for a lot of organizations.

And I think sometimes we even highlighted the fact that people who create the alerts and people who triage the alerts should have lunch once a month. And even that's Yep. I mean, everybody laughs and everybody thinks this is so mundane and not magical, but ultimately, that starts to break the barriers. And of course, if you start cranking the handle from this being 1 to 11, you get better results. But at least you have to crank the handle a little bit to understand each other's worlds. I mean, we also sometimes suggest rotating analysts with people who develop detection engineers at fancy places and just content creators at less fancy places. with people who click the signals in their SIEMs. Like, even that type of a one day in the shoes off does have a lot of impact.

Yeah, 100%. It can have a tremendous amount of impact. And I would say the last piece I would do on kind of the alert overload piece is, if possible, aggregate and correlate the alerts that are coming out of your system. So understand where they're coming from, where you can optimize, but then understand how you can aggregate and correlate them together because oftentimes you may be Say an attack occurs, right? From your many disparate systems, you probably have a lot of different views into the same thing occurring. If you can start to stitch those together, you go from investigating 20 alerts into maybe one user, one host. And it doesn't have to be exotic either. I mean, some people are in a position— if you're at a Google or a more mature organization, you may have SOAR technology, you may have automation in place, you'll have a SIEM. But it doesn't have to be that fancy. It could be as simple as, you know, maybe you're using an email inbox as your alert queue. Maybe you have some Outlook rules or Gmail labels that you're using to organize things into folders. Could be as simple as that as a starting point.

I like the simplicity of that and the suggestion of a tool that everybody has, which is their email inbox. I want to sort of draw another thing out of that, which is automation. And not everybody has a SOAR, and it's not necessarily clear from some of Anton's recent Twitter activity that SOARs are the answer we were all hoping they would be. And listeners, that gives you a bit of a timestamp on when we recorded this. But what are some of the areas where automation is most helpful? Like if I was to have, you know, my hello world of SOC automation, what is it?

Yeah.

So like you said, everyone thinks it's a silver bullet, right? Who needs humans anymore? You just have automation, automate all the things and you set it and forget it, right? That's not really reality though, right?

Probably do it with AI at this point.

Well, funny enough you say that. I mean, once we get to the next question, we'll talk about the future of the SOC, but At this point, right? Like, we're sitting at automation at this point, and you got to have consistent upkeep. I mean, we have engineers in every security operations center, and it's just like detections. It's a living entity. You always have to iterate and improve and maintain and support. So automation's used to handle the common problem of scale. You could do something at small scale, great, but how, how does it react when you really increase that for those High throughput telemetry, high volume, high alert, like we talked about alert fatigue. How do you reduce those but also be able to handle large volume at the same time? And more often than not, it comes down to it's who's in your SOC, who's your talent, who are your teammates, right? You're not going away from that talent and that human aspect, but you're leveraging that human interaction with automation, with AI to improve your outcomes in a more productive fashion. So the other thing that I did want to mention is we're really focusing on security operations centers here, but if you really think about security in general, it starts to branch out into other teams in the organization. And those decision-making factors that they do, their engineering decisions, it really impacts how quickly you can respond to alerts. If a singular team managing a bunch of assets suddenly received an alert and needed to respond to it, They should be able to understand how to implement that change or contain that malicious activity very quickly. And all of that goes towards your statistical analysis of how fast could I respond, how fast can I detect, how fast can I contain. So automation is just there to improve how quickly you could do things, but you still need that human element.

So this actually makes sense to me, I think, as far as pillars. But Tim, of course, alluded to sort of my Twitter activity recently. I want to maybe just deviate a little bit from the paper topic back to SOAR and automation. You probably met clients, maybe your own clients, who kind of say that they want to automate, but even SOAR is too much work in their minds. So admittedly, if they say SOAR is too much work, doing your own system from scratch is like way too much work, out of the question. And I mean, while I'm not trying to tell you to tell them to hire a well-known consulting firm, which we wouldn't name right now, but like, what is your reaction to— Yes, I want to automate, but SOAR, yeah, we tried that. It's too hard.

I would say break it down into little pieces, right? It's a big, complex problem, and you don't have to do everything in one swing. So if you could just get to the, like, base layer of, hey, can I email this person when this playbook step happens? What is your standard operating procedure? So really break it down into chunks of, all right, every time this incident occurs, What are the standard operating procedures that I need to do? And along that way, the more frequent that happens, you're able to break it down and say, hey, if I do this in this way, then it speeds me up to get to my outcome. So the real goal is see a complex problem, break it up into chunks, and that's like the classic Programming 101 solution that your university professors would say as well.

Yeah.

It's a bit of a consulting answer, wouldn't you say?

Definitely. I could double-click on that one maybe a little bit. The one other piece I would add is that automation akin to AI, or pick your buzzword of the month, is, like you said earlier, automate all the things. I think a lot of it is not as exotic, right? A lot of it is invisible to us, because if we look at tools over the past couple of years, a lot of the tasks that we would consider 5 years ago as, like, automation and additional augmentation to tools to do engineering efforts to automate things, A lot of that's built in now, frankly. So, and I think that'll continue to progress. So using some examples like enrichment or contextualization, previously that would be an automation workflow, to your point, Anton, like that exists in your SOAR or your custom engineering. But a lot of tools, EDR or SIEMs, are including that earlier on in the pipeline. I think something that's really critical there is not only is it alleviating that alert fatigue, as we mentioned earlier, by helping speed up the automation component of triage and decision-making, but now it's an input to the detection logic itself. So you can alleviate the false positives before they even are generated and happen. I think part of that is also on vendors and tool systems, ones who shall not be named as well. So, like, part of that shifts onto the tooling to keep that automation improving as kind of an invisible component, too.

So that means that automation may be— when Tim built VMTD, right, behind SCC, in his SCC, That's an example of something that automatically delivers magic without being hard to build, configure, define for the user, right? Like, that's invisible automation as opposed to automation that you need to configure. Is that roughly the line of thinking?

Yeah, exactly. Some components like that where things that previously needed to be a custom engineering effort are now just embedded within the products that you use. And part of that will just continue to improve over time as, you know, ultimately, There's a shared fate in this, right? Like, focusing on that is really critical.

So I want to switch gears from machines and systems to humans, because with many types of automation, whether it's SOAR or writing your own scripts and code, it sounds like the skill sets are somewhat different. And I think if you— if I mention Twitter again, or X, people still debate whether you need to know how to code to be a security professional in general. But I think that over the years, SOC analysts wasn't a coder for sure. So we point out that automation, that they need to develop some things. So the paper also highlights the need to change the skills if you transform. So what specific skills and knowledge, in your opinion, would be most important for future SOC analysts or engineers? I don't know, 5 years, however ambitious you want to go. Please don't go 20 years.

Yeah, well, if it's 20, then it's just AI, right?

So Well, AI, probably, but not—

Yeah, yeah. No, but in all seriousness, I think, to your point, there's definitely an engineering-led culture shift, and I do think that permeates into security. So, to Alex's earlier point, there's a dependency outside of the SOC, and I think that becomes ever more present as we try to shift left, or whatever terminology you want to use, kind of within our own security operations and our detection engineering. So, do we still have tiers of escalation, a 1, 2, 3, or whatever numbering schema you use? Maybe not. Again, I can't speak to the ambitious nature of 5, 10 years and migrating away from that model. But I do think it is critically important to have input. As you mentioned earlier, maybe beyond having lunch once a month between your analysts and detection engineers, having a regular cycle of input so your analysts that are responding to these things have input to the detections, creating them, input to the tuning, input to the automations that are making their day-to-day quotidian actions easier at the end of the day. And that's critically important. And do they need to be a developer and a coder? Maybe, maybe it shifts that way, but they don't need to be doing all of it to start, even if that is kind of one of those ideal end models to your point, Anton.

Yeah, a couple things to add though, right? With the developer mindset, you don't have to be a full-on developer to be providing valuable contributions, right? But understanding the different technologies and their purposes in the environment. You just want to be able to know that, hey, this is an EDR solution. This is why it's there. This is who I'm contacting. This is an IAM team. This is what they're managing, and this is what they're supposed to be doing. Letting them know that these are the appropriate security policies. These are the appropriate security techniques to build and engineer safely rather than getting bit by the bullet at the end of the day.

I really like these concepts around how do we get these people to work better together. I think this gets back to an enduring theme on the show, which is that security is a team sport. And the idea of having lunch together seems so basic, but it really does make a difference for people getting along better. I hate to ask questions like this because Anton always makes fun of them when he got them as an analyst, but you guys aren't analysts, you're consultants, so I can get away with it. What are your predictions for the future of SOC?

I could start that off. So I think it's a mixed bag, right? We're talking about technology, we're talking about people. I think some of the non-fancy— I mean, AI is a big topic right now, but think about just like getting your data to a particular platform and being able to funnel it to different locations, right? One input, multiple destinations. Maybe that it's a particular data lake where you're centralizing your data. Maybe you're starting to work with other teams to divert that data and share that telemetry so that You're not doing any shadow IT where one team needs a certain piece of telemetry, another team needs a different piece of telemetry, and you're building your own pipelines at the end of the day. So maybe you're not only feeding your SIEM, right? It could be working with different teams in your organization. So I think the first part of that is it's going to be a mixed bag and everyone's going to find what new approaches are to tackle the old things, the old issues, what new approaches are to tackle the new issues or the unknowns. So, I think that's a big part of just understanding that you could use different technologies, you could use different techniques, and you could come out with the same outcome.

But that's complex. I mean, I don't want to turn it into kind of a maybe too much of a sales story, but like, it does sound like we would be moving to the world that's more complex. And I don't necessarily point that in crystal ball for consultants. I really mean more about the fact that the world is genuinely more complex. I have a SIEM, I have an EDR, I have a data lake, I also have some kind of other storage. and I'm trying to make sense of it all in a coherent strategy, right? Like, that's a rich but scary future, right?

It is, but there is shared responsibility. It's a team sport, to Tim's point, right? So, like Alex was mentioning, understanding that some of this workload for all of this profusion of telemetry can be shared among other engineering teams, to people outside of the SOC. So, while it is growing and becoming more complex, you're ultimately all solving similar yet slightly nuanced problems. So, understanding your platform engineering teams, your telemetry teams, or whatever your organization looks like, you're all solving similar issues and then using it for different outcomes. So trying to share some of that at the front can actually, while it is scaling, probably cut down on your existing workload even as it exists today.

One thing that came up in regards to AI that I think you quickly mentioned is that I think that in the future, would we expect more certainty about, say, GenAI use cases? Because I sense that right now we are in the very end-of-the-mad rush to try to apply it to everything. And it's very clear. And I've seen a few days ago, I've seen some vendor slide that said LLMs aren't enough. And I'm like, yes, we are starting to fall off the hype cycle. That's great. So we are not really in the, oh my God, GenAI would fix all the problems, but we are in a, oh, I see these use cases are good, these are bad. What's your take on this? Do you see any kind of niche for the future? What are the use cases that are promising in your mind?

I'm thinking that there are particular use cases like you mentioned, right? It's specialized use cases. What does it do well? And some things that you could use that to do is text summarization. Let's say you have some sort of threat intelligence briefing or you're going through investigation case notes and you really need to quickly get up to speed. That is what AI strives and does really well today. and really accelerates you to getting to your desired outcome. And on the other side, like, think of just aggregating data, right? You could get statistical approaches to your detection rules and really seeing what is more powerful today, what is coming out of it at the end of the day after tuning, and providing a direction to where you need to go.

Hmm. That's exciting, and I guess also optimistic in regards to how SecOps would be a little easier in the future. But what about the process side? Like, ultimately, Just as SOC, at least some SOCs, not ours perhaps, or not our similar operation, originated from original NOC thinking when the processes are essentially inherited the NOC DNA from the '80s, if not the '70s. So, do you think that process would survive in the future or would this more engineering-led automated approach would become more widespread? I think I'm asking a very rambling question, which I think leads us to think, Do you think the SOC of plus 5 years would look like SOC of today at many companies, or would it be more different?

I would hope not. I ultimately want to continually improve, whether you're transforming or whether you're optimizing what you have, right? I think at the end, to kind of bring it full circle, you have to change to keep up with the times. We work in a discipline where our adversaries aren't just setting and forgetting. So you have to continue iterating on your processes. Whether that's the people, the process, or the technology. So I think a lot of that does change with time and whatever the tools look like 5, 10 years from now, we'll see what they are. But at the core of it, I think it's all human. It's all people, right? I don't think it's going to be an army of little AI robots. Maybe they'll make our lives a lot easier. I think it's important to remember that at the core of this, it's a human problem. It's a human construct. So using the tools and the processes to make the humans ultimately most impactful to deliver the outcomes, which is reducing risk to the organization, right? However we do that, I think it has to change ultimately. And whether that's engineering-led or what that looks like, time will tell with what tools come out ultimately.

I really like that answer.

Yeah, me too. And I was always like, that was— people who sometimes don't listen to the end of a podcast, in this case, better do that. Of course, I'm telling this after.

So, I hate to do it, but I got to ask our traditional closing questions. One, have you got a tip to help people improve their SOC modernization transformation? Or 2, do you have recommended reading?

Well, I mean, we're in luck here. We just previously got done writing up our new white paper about the future of the SOC. So based off that, I would want everyone to take a look at it if they can. And yeah, it's all about security operations. It's evolution or optimization, and you get to choose your path. So I highly recommend taking a look at that paper.

That's great. Yeah.

And I would say maybe on the flip side from a reading and more so on the tip side of things, kind of a reiterated theme is focus on the basics, the simple. It might not be exotic and buzzwordy, but focusing on the simple components of the processor technology problems at hand that you can solve for the largest impact. It might not be the most marketable, but it can be the most impactful in the end.

I love those answers, and I'm really grateful that you guys joined us today. Thanks for putting up with our somewhat silly questions, and looking forward to checking out that paper.

Yeah, thanks for having us.

Yeah, thanks for having us.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Thank you, everybody.