EP58 SOC is Not Dead: How to Grow and Develop Your SOC for Cloud and Beyond

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the product manager for threat detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and esteemed member of the cloud security team here at Google. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you like our content and want it delivered to you piping hot every Monday afternoon Pacific time, please do hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, we're talking about SOX today.

Yes, this episode was very dear to my heart.

Why is that?

Because it touches on the work we've been doing together with the Deloitte team, the team we'll interview on the call today, for actually more than a year.

Wow.

And we are writing a series of, well, white papers or reports or sort of research papers that talk about how SOC is changing and hopefully giving some advice on people who are trying to build a SOC, you know, circa 2022 SOC, not circa 2002 SOC in 2022. We're trying to figure out what's the up-to-date SOC wisdom on the process side, on the skill side, on the technology side as well. So this is fun, and this is very— it's a very tricky area of security because Unlike, say, encryption or endpoint security, SOC security operations is very tied to humans.

Yes.

Pesky humans. They're always the problem, right?

Pesky humans. Yes. Yeah, it's a really interesting area.

You're right.

I hadn't thought about that particular aspect, but we get into that in the show, don't we, today?

We do indeed. And even when we organize the papers, we organize them as kind of like a process paper, people paper. And it turns out the author team including myself, kind of had passions about some things. For example, people really said, oh my God, process paper would be boring. And I thought, no, come on. This is like where the magic happens, is kind of between organizing the processes so you're predictable, consistent, yet allow for creativity. This is really tough and this is fun.

Yeah, that's an interesting challenge. So with that, listeners, I am so delighted to introduce today's guests. We are joined today by Alexei Viemer, a senior manager at Deloitte's Cyber Detection and Response practice, and Dan Larsson, a senior manager also at Deloitte, but differently at the Cloud Security practice. So we have both cloud and cyber on the Cloud Security Podcast today. So we want to talk about something which is both cloud and cyber. We want to talk about where's the SOC, what is going on. And so Dan, Alexei, you know, what's the state of SOC today? What trends in SOC can we talk about? What's going on?

Yeah, thanks for inviting us and having us on first and foremost, and I feel like I'm going to be struggling to get on your level of professional voice talent here, but I will do my— I'll do my best. Yeah, so broadening it out beyond just cloud-specific topics, but rather for SOCs today, we can talk about a lot of the common themes you'll hear from SOC practitioners about overload of resources. You know, the job is growing, the attack surface is growing, More things to look at with fewer people and fewer, you know, fewer resources to do the job. But I think one of the common themes also that Alexei and I will kind of hit on is driving efficiency through tool use, through process, through automation, and really kind of like wading through those waters of being overwhelmed to better direct those finite resources.

We love the theme of tool use and automation on this show, so that'll be interesting to get into. Alexei, you look like you're going to give an answer there as well.

Yeah, absolutely. I think just to expand on Dan's points, right? Data is the name of the game today. And so, technology is able to ingest this at such a large scale. But what we're noticing is, even though you have the capacity to use this data, it's actually properly using it with the appropriate processes is where we start to see a large disconnect, right? So yes, technology is an enabler of these meaningful connections. However, you're dependent on human analysts to make those important events and trends and connections with that context that may not be readily available in the data. And so I think that's a theme that we're seeing is emphasizing that type of information sharing across your toolset in a cohesive manner.

And by the way, Alexei, when you say data, you clearly are quoting from our coming Deloitte Google Cloud paper, where we highlight that what matters for SOC is kind of the fidelity, the quality, the scalability of data. It may not be the brand of a tool that matters, but how you handle the quality of data. Is it conclusive? Is it clear? That type of stuff, right? Am I wrong about this or not?

No, absolutely. And I think, you know, we're seeing the need to ingest this more data because of this expanding attack surface. Organizations are noticing that threat actors are more sophisticated. The borderless enterprise is expanding, right? So obviously cloud monitoring has become a huge point of emphasis and how do we readily ingest that information and then bring it back to specific needs of the business units, right? Because at the end of the day, that's what the SOC aims to protect is the business delivering value to the different stakeholders.

I mean, we all know that, yet when we talk to clients, when you talk to clients, when we talk to clients, we do see SOCs, security operation centers, that are kind of permanently and woefully understaffed. During my analyst days, I've encountered people who were thinking of a 24/7 SOC when they had 2 analysts, and I said, what are you talking about? Like, it's not even close to remotely conceivable. Like, you're talking 9, 10 people, however you count, to do a 24/7 SOC. So what is our advice? What is your best advice for SOCs that are kind of permanently and woefully understaffed? Of course, you can say outsource to Deloitte, and we wouldn't hold it against you. Well, we kind of would, but okay.

Step number 1, call Deloitte.

Well, that's number 1.

Okay, easy.

Okay, what's number 2?

I would say that my own personal take on that question is that there's an immediate knee-jerk reaction when staff is overwhelmed to reach for the staffing lever and, you know, make that claim that I need more people. And maybe part of the guidance would be, you know, take a breath, take a step back, pause for a second, and understand the inputs of that being overwhelmed, right? So it's highly likely and, you know, entirely possible that in today's security operations center kind of milieu that there's going to be way too many projects for your team to cover, way too much attack surface, way too many alerts. It's totally likely. But within that whole crush of things to do, there might also be improvements to process you can make. Maybe there's unnecessary things your team is chasing down. Maybe you have alert fatigue from some things that are misconfigured. So before you just immediately reach and say, I need more people, there's probably things you can improve or that you can make more efficient. And I realize that that's not an easy thing to do, right? Everybody's got 900 tasks. So kind of looking at the process is just one more of those tasks, but it might be something that's worth investing some time in. Because whereas you thought you maybe needed 12 people, maybe you really truly only need 5 once you kind of get more efficient in what you're doing. Yeah.

And just to expand on Dan's point a little bit too, I think one of the ways that we see clients handle these issues, right, is you need to be realistic about your budget and the size that your team will have. Right. And from there being really strategic about what capabilities you want to focus on, right. If I know I'm only going to ever have a 3-person team, I'm never gonna be able to do 24/7, 365 monitoring, and I need to outsource that. What I can do though is develop my own threat intel team, my own threat hunting team, my own incident response, right? And based on what is important to your organization and the skill sets that you have, you can start to develop a plan of this is what we bring in-house. This is where we need to maybe lean on other third-party providers to do that. So, 'cause outsourcing some of that is more cost efficient, I would say. From also a— just to address the permanently understaffed, right? We've also noticed developing a career model, giving people a trajectory, training opportunities. And also, as we've seen with this shift in the last 2 years of more flexibility, right? People don't want to have to come on site 5 days a week, every week, right? Can some of this be achieved remotely? And I think a lot of these trends are also helping retain analysts when they're at a position that typically experiences a lot of churn.

Being strategic is obviously good advice always, but hopefully we'll address it in slightly more practical terms because if the team has 3 people, prioritization decisions become really tough. Frankly, and I'm going to argue with Alexei a little bit, I probably won't invest in hunting if I have a team of 3.

Me neither.

But if I have a reliable outsourcing partner who handles the job, Then, it's a maybe. So to me, this is kind of like, if I trust my MSSMDR, maybe I will hunt.

Yeah. And to build on that point, right, like hunting is a very sophisticated, advanced capability. So if I have a more junior staff, that is not the capability I want to bring in-house and depend on, right? They maybe start as tier 2s or dabble in incident response, right? So I think it's just as important knowing the skill sets of your staff and the talent you can attract in relation to that strategic vision, because I think that's spot on, Anton.

Well, and if there's one other thing that I can add into that point to strengthen it, is that prioritization is key. Because even if you found a partner that performs something like threat hunting to the extent that you desire and, you know, has solid capabilities, without exception I can say there's always going to be a component of collaboration and of insider knowledge needed So even if you outsource certain components of your SOC operations, there's still work needed on your team. So if it truly is not something that you can support a couple hours a week of at the bare minimum collaboration with your MSSP or MXDR provider, then it may not be something you want to prioritize.

That makes a ton of sense. I think the advice boils down to a bit of, if you haven't got somebody who can run the ball, don't try running the ball, which is, you know, that's good advice. I want to scroll back up the conversation a little bit and touch on the automation piece, because this comes up a lot. It's an interesting thing. We have Simplify now here. We hear a lot of people say, well, you need to automate if you're drowning in manual work. What does that actually mean? How do people get started with that? What does that actually look like? I assume there's not welding machines in the SOC.

Well, maybe not in your SOC. There's not welding machines.

Maybe my SOC's doing it wrong.

Soldering machines, maybe making your own lines. I think one thing that gets lost, in my opinion, and Alexei, I'd love to hear your thoughts, but one thing that gets lost in a lot of the client conversations that I have is the kind of the jumping to black box magic conclusions about what automation can do. You need to start with an understanding of what it is you're trying to automate. You need to understand the process and you need to be able to execute the process well manually before you automate it. You know, automation isn't going to be an end unto itself. So I think that lack of starting from a good solid place can lead you to undesired outcomes. I'll just put it that way.

Yeah, I think what you want to do, right, is focus on, as Dan mentioned, the outcomes you're trying to achieve and work your way back from that, right? Because if you have the automation driving your outcomes, this is where you start to have rabbit holes and it doesn't impact the organization the way that you want, right? So if my goal is I'm trying to decrease mean time to detect, that's gonna be a very different automation than mean time to respond, right? Or things of that nature. It depends on where along the process that is. And so understanding your processes well enough, and then also what you're trying to achieve is really key because there's also things that are simpler to initially automate, and then there are things that are much more consequential. Automatically pulling in threat intelligence feeds is really helpful, and that's a very low-hanging fruit compared to automatically quarantining a machine of what could be a very important user. And you need to do that. crawl, walk, run model to make sure that you understand what you're automating and what risk you are accepting by automating certain things.

Yeah, and one other thing to add on to that, which I wholeheartedly agree with everything Alexei said, but you also must understand that when you're approaching automation, it can be a skill set unto itself. So you're likely going to partner with somebody, some technology that's going to be the enabler of that automation process, But you will also likely run into some cases where you're stretching the tool to its farthest extent or need to tweak it just slightly. So do you have Python skills in-house to build automation playbooks? And if you don't, you kind of need to think about that upskilling or partnership to get those skills to get everything you want out of it.

And this one is not going to be negotiable in the foreseeable future. We can talk detection as code, we can talk playbooks, we can talk— even we can talk no-code, low-code platforms that ultimately I'm sorry, this is my sincere belief— will push you to code at some point, at least in this area. If it's not the playbook code, it's going to be integration code.

But it's in the name. It says no code. What are you talking about?

No way, right? Okay, fine. Switching subjects to something more exciting. Well, actually, that was exciting. So the next question is a trick question, and I know that it's just designed as a trick question. So lots of technologies, and again, in our papers, we covered the use of SIEM. We talked about threat intel. We talked about EDR. Of course, we are talking SOAR. So if somebody pushes you in a corner and says, Dan, Alexei, name one technology that's the most critical for your SOC, can you do that? Would you name SIEM? Would you name SOAR? Would you name ADR? Would you chicken out and say, I'm not going to name it? 5 seconds.

I don't know, maybe not 5 seconds. Alexei, how about you jump in those waters?

Yeah, I can take a first pass at that. So I think for me, Anton held me captive and said, hey, you better answer this question or we're no longer co-authors together. What I would say is, for me, it's that data pipeline that everything is upstream of your tools, right? So obviously I'm partial to cloud-based technology, and I think that Google Cloud and Chronicle does this especially well. But how Having the mechanism to ingest all of this raw data at scale, normalize it, parse it, be able to replay it back in an investigation, and have that be in a common model that is agnostic of your tool stack downstream of you, I think is really instrumental. Because then it gives you the flexibility as vendors come in and out of your environment, your data is still your own. It's still in a way that's meaningful and actionable. And I think that's where we're starting to see organizations realize of wanting to break out of this vendor lock-in and having some of that flexibility. So it's a little bit of a cop-out answer because it's upstream of the examples that you mentioned, but I think it's a really important foundational piece because of the flexibility that it offers.

But you were right because I didn't ask a question, name one tool that's sufficient for a SOC. That would be dumb. There is no such thing. I said, what's the most critical tool? And you answered, it's the pipeline leading to other tools. That's fair. I think we have a winner. I may or may not agree with that, but I see the merit behind the answer.

Yeah. And I think I would essentially say the same thing, just move it farther downstream.

Upstream or downstream?

Downstream.

Which way?

So, you, me, and Alexei have had this conversation relative to the paper that we're writing right now. But, you know, my interpretation is that you definitely need to have common— you need to have a common resting place for security telemetry and security alerting and things. in such a way that it's accessible for interpretation, analysis, hunting, anything downstream past security tools, telemetry, so that you can then drive further observations and/or develop more machine learning-based or more broad-based observations off of the whole corpus of data, right? So, a data lake, essentially, that's in a common format. So, it's not probably one technology, it's probably a couple of different technologies that serve that purpose. I think that's probably the most important. technical function, in my opinion, of the SOC today is taking all this disparate technology and all of the logs that come from it, gathering, collecting, and storing in an accessible and common format.

Well, we've got data pipeline and data lake as answers to that. And Alexei, you should go home feeling happy about this episode because Anton said you were convincing. Maybe he didn't say you were right, but he said it was a good answer. And that's, you know, that was high praise out of him. I want to, I want to shift gears a little bit and give you one of my favorite hypotheticals, which is imagine you are a SOC and you have been handed the keys to a cloud. What do you do? You know, imagine you've never seen this cloud before. And for bonus points, tell us what would be different in your answer if it was, say, an Azure cloud or an AWS cloud versus a Google cloud. And extra bonus points if your answer is more different than just the names of the menus you click on.

But before you answer, just a very quick interruption. I'm sorry. Like, we do see clients, and usually it's a mainstream pre-cloud client. I don't know, like a proverbial— in Gartner days, they used like a bank in Kansas. Somebody who didn't do cloud before, and suddenly they build a SOC. SOC is working out okay. It's under-resourced, but ultimately it's okay. And suddenly it's like cloud migration. So, there are real people in this situation. So, Tim's not making this up.

No.

There are companies, there are teams at kind of pre-cloud companies whose SOCs now have to do GCP, have to do Azure, and maybe they have to do Amazon. I don't know about this one, but how would you approach it?

And I talk to these teams every week. Like, these are real people and I feel for them.

Yeah. I, you know, you asked what would you do first. I think that question naturally leads to maybe a 1 or 2 finite list of tasks, but I think it's probably a combined list of multiple tasks for a common purpose. And I think that that's grasping the basics of that cloud you've been handed, right? So, configuration, standardization, checking the traps of that cloud deployment, which are the things that commonly get us owned in this day and age, right? So, misconfigured buckets of storage, misconfigured permissions, understanding what you have in the environment that is, you know, that's SaaS that you have very limited control over versus things that you have more control over, just kind of good hygiene. Like really documenting it, understanding it. And it's a little bit of a challenge for SOC operators because you likely don't own any of this, right? Like you'll almost guaranteed have to work with other members of the organization that own these cloud environments and that can really answer some of these questions. But you can't possibly hope to secure it if you don't even understand what you've, you know, what you've been given, what you've been handed. So that's definitely where I would, you know, if I was CISO for a day and was handed a cloud, that's definitely what I would recommend we do.

Yeah, and I think, um, just to very quickly expand on that for Dan, right, like that governance model of what is my team responsible for, what is being outsourced to the provider, is absolutely critical. Because every other group thinks the other one's holding the ball, then you have no one, right? Mm-hmm. And I think my first also question back to you is, what is this cloud environment supporting, right? Is it a business-critical application? Is there sensitive data? Because not all cloud environments are the same. Now, obviously, we've seen a lot of examples of people exploiting cloud environments and moving laterally to higher-value assets, but if I'm starting from scratch day one, it goes back to my crown jewels. It goes back to the risk I'm willing to tolerate as we get that type of visibility into our SOC.

I think just as a closing point on that, our answers might have sounded a little bit generic or a little motherhood and apple pie, but, you know, I think a lot of SOC operators might naturally Their knee-jerk reaction might be, what telemetry can I get? What use cases can I build? Let me just research some threats to this particular type of cloud deployment. And that's definitely a necessary step. But if you don't even understand what are the native permissions that are available in that cloud deployment, right? Are they more primitive? Are they more nuanced? Are they applied appropriately? Where's the data actually flowing? Where are my VPC boundaries? Like, that's the kind of stuff that, You know, if you don't have a good solid grasp of that, then anything you build as far as a detection will slightly fall flat. So it really is kind of the critical first step.

And we do see people who do exactly what you just said. They jump at collection and they say, I need my CloudTrail in my SIEM. And that's all they have in a year. But like, how is this helping their SOC? It doesn't. Okay, fine. I rant too much. So let me switch gears.

No, I think that's a really good point though, Anton. Before you switch gears, I think that's critical for people to understand. It does you no good just to gobble up the logs if you don't know what meal you want to cook with them, or you don't know who you're trying to feed or what their dietary restrictions are. You got to think things through before you dive into it. And that's, I think, actually a really, really good answer.

Yeah, Tim, throughout my career, I've always said that in the SOC, there are generally 2 classifications of people when it comes to detection systems. There's kitchen sink people and there's scalpel people, right? And you want to be a scalpel person. You want to know what you're doing with that data you're collecting.

Yep.

Okay. So now we are kind of getting to the end of the recording. I wanted to ask possibly a slightly dire question.

Oh, no.

Occasionally, you read about predictions that SOC is dead. And typically, in my mind, when I read the paper that says SOC is dead or a blog, they kind of mean that the physical room full of analysts watching the screens and like talking over, you know, the cubicle walls is dead. Like nobody or very few people run SOCs like this during COVID That's pretty clear. So what's your reaction to SOC is dead? Like, how do you respond to that? Or what's your response to predictions of we don't need a SOC because something else would happen?

Yeah, I mean, I think, you know, when people, as you mentioned, right, imagine the SOC of yesteryear, right? It's a bunch of people staring at pretty screens, maybe picking up the phone, And I think we're seeing that go away, right? It's much more dynamic and collaborative. There's not a phone you pick up to call, right? We have these automated alerts that come enriched with this data, and we are breaking these silos where we're able to work with these different groups. And it's not just, hey, close the ticket as quickly as you can, right? We're trying to get as much contextual information as possible. And being— this is a bit of a cliché, right? but being much more proactive as opposed to reactive. And I think that is where we are seeing technology enable folks to do that and organizations really setting up those processes, whether it is through a common ticketing system or standups or just getting to that point where we're getting the right people in the room to make the appropriate decisions.

Yeah, I would agree that if, as long as we're defining it as a capability and a set of people with common goals, then no, I don't think the SOC is dead. As far as an aggregated and distilled group of people in front of a bunch of NASA screens, yeah, I don't know if that's necessarily in vogue anymore. But yeah, as long as it's those— as long as it's a core set of capabilities we're chasing, because there's always going to be, no matter how much you push DevSecOps and no matter how much you push blurring of boundaries of driving security into other portions of the organization, there's always going to be someone who who has to care about edge cases and really advanced threat actor behaviors. And that just will not be the developers and will not be application developers. It never will be, right? So there's always going to be a group of people that are more security-minded and that have to kind of drive the edge cases no matter where you push the boundaries.

I firmly believe that. I think you're absolutely right about that. So Dan, Alexei, we're reaching the end and I get to ask our traditional closing questions. And, you know, the first one I think is a bit of a giveaway because now Now we've mentioned twice this paper, at least twice. I don't know if I was keeping accurate count. What's your piece of recommended reading? And feel free to plug the paper here. That's a great idea. And then the second piece of our closing question is, what's your one bit of advice for doing SOC better?

Yeah, definitely. I mean, first and foremost, like you already mentioned, I would go back and I would review the 3 previously released white papers that we've developed with Anton. Bated breath, waiting in anticipation for number 4 to be released in the not-too-distant future. That's great reading. Also, I would make a general plug as opposed to any particular book. I've been reading several lately about social media hacking and manipulation related to anything that's elections or just sentiment, public sentiment. I think those are very interesting books to read. Yeah. So then in terms of advice for how the SOC can do things differently. It's an old one, but it's a good one. And it's don't chase shiny cars, like try to make the most use out of what you already have. Tools have gotten so— we've talked about this before, I'm going to sound like a broken record to Alexei and Anton, but tools have gotten very strong and very capable. And most people that are operating at an enterprise level have a lot of excellent tooling at their disposal. Really try to wring every bit of value that you can out of it because number one, you're likely going to make your life easier, and number two, you're going to make your leadership very happy if you're not asking for additional budget, if you're stretching what you can do with your existing tooling.

Got it. Good advice on making leadership happy by not asking for more budget. Alexei, what have you got for us? Recommended reading and one weird tip.

Let me guess, the papers?

Yeah. last process paper on the balance between having defined processes but also enabling creativity. And I think that also ties into my advice that really resonates with me, right? So my one advice for the SOC would be empower your people and really take the time to invest in them. As Dan mentioned, we sometimes see organizations chase the shiny car, so to speak, and say, hey, it doesn't matter who's in these seats as long as we have these tools. And I think It cannot be farther from the truth, right? Organizations spend a lot of time trying to just retain and develop talent. And I think by placing incentives, getting people bought into the organizational mission makes a huge difference once we're trying to frankly counter these really sophisticated threat actors. They don't discriminate. They don't care that, hey, this group typically doesn't talk to that group, right? They're just gonna attack your organization wholesale. And so you need Smart people doing smart things that are empowered to do so within obviously the context of your organization. And I think that would be the advice I'd give.

Got it. I think that's great advice. Alexei, thank you so much for joining us today. Listeners, the papers mentioned will of course be available in the show notes. We will tweet them and they will also be available at the website cloud.withgoogle.com/cloudsecurity/podcast.

And now we are at time. Thank you very much for listening and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @anton_chuvakin and @_timpeako. Tweet at us, email us, argue with us, and if you like or hate what you hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.