EP187 Conquering SOC Challenges: Leadership, Burnout, and the SIEM Evolution

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Tim Peacock, the Senior Product Manager for Risk Analytics and Cloud Threat Detection in Google SecOps, and of course, Anton Chuvakin, a Reformed Analyst and Senior Staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on the Google Cloud Security Community page. Anton, what are we, what are we talking about today?

So this is one of those ever-popular SOC episodes, and I think that we already know that detection and response security operations is definitely something that sells tickets as far our podcast is concerned. And instead of having a hand-wavy discussion about modern SOC that may or may not exist at many companies, this is a real kind of connected to the ground, connected to reality interview with a person who runs a SOC, very real, very current SOC. And I think that there are some really cool ideas about that in the show.

I'm looking forward to this one in particular, given the maturity of their thinking around SOCs and how How much we enjoyed this conversation. So maybe without further ado, let's turn things over to today's guest.

And welcome today's guest, Nicole Beckwith, Senior Security Engineering Manager at Threat Operations at Kroger. Welcome to the show.

Thank you so much for having me.

Sure. So this is, as we sometimes joke on this show, the detection response and SOC episodes really sell the tickets to the show because some of our most favorite episodes, some of our audience's most favorite episodes. are about the SOC. And today I feel like we have a really good opportunity to sort of travel into your mind of a SOC leader and see how you've solved some of the problems people really suffer with. So let me start exactly there. In your opinion, what are the most important qualities of a successful SOC leader today?

Oh gosh. Well, so first I would say everybody answers this question and starts out with how SOC leaders are supposed to act towards their team. And I feel that, you know, although it's really important, and I could argue the usual, you know, empathy, adaptability, communication, which are all important, but over the past few years, I've really realized during the countless trainings and seminars and talks I've been to, that nobody really discusses how you should treat yourself as a leader. So, I'm going to start out with that. So, my number one is you have to give yourself grace. Right? You're going to make mistakes, you're going to get it wrong. So, you really have to, you know, show yourself that grace. There are going to be days that you want to quit, but that's part of growing, and you really can't lead others if you can't lead yourself first. So, that's where I would start with this, is grace. And then, of course, empathy and communication are paramount in a SOC, right? So, listening, that active listening, giving clear direction, And having a pulse on everybody's feelings and concerns really helps foster that trust and that open dialogue within the team, which is so important, right? So, you know, making sure that your team is okay voicing their opinions, they know that they have a seat at the table, they agree that it's healthy to debate topics, and they're not going to be punished for bringing something up. I feel like the greatest failure you can have as a leader is a team that doesn't trust you, enough to bring up their issues and discuss those with you. So, that's first and foremost, you know, communication is key in that trust. The second thing I would say is adaptability. In a SOC, you know—

Yeah, enough said kind of thing, right?

Yeah, you have to be able to roll with the punches, whether it's an incident, a policy change, a disagreement on a team, or even a larger culture change. You know, we've seen a lot of the SOCs recently, and companies doing back to office or return to office. So, you have to be able to take these little nuggets, find the silver lining in them, and use it as a growth catalyst for you and your team. So, just understanding that give and take is so important every day, and you aren't going to win every fight, right?

Especially when you fight sometimes with people who are kind of to the side or above, outside of the command chain. So, I think to me, that part is It's kind of critical.

Exactly. And then finally is the strategic thinking. You know, I think a lot of, especially SOC leaders, struggle with this because you are in the weeds and you're so tactical on a day-to-day basis with your team that sometimes it's hard to get out of that and focus on the strategic. So, I like to say good leaders think 1 to 3 years in advance about their team goals, Mm-hmm. Whereas great leaders think 3 to 5 years in advance about company goals outside their team and that collaboration in the greater good. So, thinking about bigger picture and, you know, what your blast radius is around you, who else can you impact? And even in a SOC role, which can seem so reactive at times, it's really important to get outside of that box and think about the bigger picture.

I like how grace made an appearance in a SOC discussion. So, because I think some of the other points, at least some version of it, people may have heard before, but I think the grace argument is just stuck with me. So, I think that it's probably one of those moments is that as a SOC leader, you need grace. It is almost like a quotable element, which people do not think about, like communication, psychological safety of some form, not winning every fight. These are at least Sort of logically fit. But I think grace is the one that's, to me, I had a brief transformation moment as a result, like, oh, actually, yes, that's a really good one.

Yeah. And we overlook that regularly. It took me a while to show myself grace. So, it's something that I've had to learn as well. And it's not apparent to everybody because you're so focused on your team first and not yourself.

Yep.

So, to switch the subject a little bit to the actual high-functioning team, I think, like, If you look at the impression people get when they may be non-experts who didn't spend time in the SOC, look at the news, look at the headlines, they kind of assume it's all about the burnout and false positives and how stressful it is and how sometimes it's hard to move to higher levels. But to me, these are sort of the outsider concerns occasionally. And so, when you're building a team that's high-functioning, what's the approach? How do you actually make the team that works? And ultimately, that implies they're happy and successful and like what they're doing. Because to me, high functioning is not a cold performance argument. To me, high functioning implies a degree of happiness, satisfaction, and interest, not just performance. I didn't say high performing, but high functioning, right?

Absolutely. And I love that you made that distinction because high functioning doesn't mean KPIs, right? It's not a metric that you can put on a graph. So, for me, Building and maintaining a high-functioning SOC team really depends first and foremost on the scope and the team size, right? But I built mine based on what I observed at others that didn't work well, right? So I took all of those lessons learned and said, what was it about these teams that I disliked and how can I change that?

Wait, wait, wait. Sorry for interrupting. I apologize. But like, you know, I think we ask about the challenges later on, but This approach just struck me as like, so what are those horrible things? Like, everybody likes to hear about other people's nightmares. So, can you quickly detour to that and then continue with the fight?

I can, yeah. So, one of the things that I hated was that SOC teams seemed to be so siloed. Every team has its function and its purpose, and they're not really communicating amongst each other. And so, you know, that's one of the things that I took away. was I really needed to build a team that was circular in nature that allowed for that free-flowing communication. So, when I built this team, communication and collaboration was paramount to how the team was structured. I built this team with no guardrails, meaning my intel team, my hunting team, detection engineering, insider risk, they all have primary roles and responsibilities, but But there are no guardrails, meaning they can jump over to another pillar at any point in time, you know, multiple times a day, multiple times a week. And, you know, that really helped, one, with the communication and collaboration, but it's also inspired growth through learning. It's allowed differing viewpoints, which I feel like in this industry is critical. And then, it's really helped with that creativity and innovation. You know, we talk about burnout. That is one of the things that I would say has really helped my team the most on burnout, is that they are allowed to move outside of the box and, you know, be collaborative and creative on things that really interest them, even if it's not their primary role or responsibility. So, you know, we're all under one umbrella. But with this, it's resulted in a much better work product. In the end, because we're allowing creativity to flow, we're allowing that communication to flow. And so, there are no silos that are causing that stale intelligence or that inaction. They're working together seamlessly. You know, I would also say lots of memes and lots of jokes in your chat channels, right? But my biggest fear, and to, you know, go back to one of the things that I hated, Mm-hmm. Was being bored and being stuck doing the same thing day in and day out. Nobody likes that. So, why would I expect my team to be any different, right? So, I wanted to make sure that they are challenged every single day, that there are new projects that maybe don't fit in their wheelhouse, but that they can move over and work on so that they feel, you know, more productive and they get to think outside their normal box.

So, it sounds like the repetitiveness of some elements of the job is solved by being able to move to kind of other side jobs and other projects. Because when I had a discussion with somebody about SOC and TOIL, they sort of said, what do you mean I can change to other jobs? Like, this is the job, right? And I'm like, yeah, but there are these other elements. There's how rules are being built. There is how automation is being built. There's how Remediation advice is being crafted. There are all sorts of things around it. So, if you think that your job is to click on an alert in a SIEM and decide what to do with it and then click on the next one, then yes, you will burn out. But if your job is to prevent attackers from taking over the company, then you have a lot more things. And then, the burnout starts to be about, you know, overwork rather than about the repetitive, you know, stress injury to your brain, which is, I guess, a term I just coined.

I love that. Yes, that is absolutely the key to unlocking, you know, a team and making them high-functioning. The performance is going to come as long as they're engaged, and they're passionate about what they do. The rest is going to follow that. So, that's been my approach to this team.

So, as far as examples, can you share some specific examples about how you have built it and maybe how you maintain it? Because one thing that comes up again, once people lived in SOX, is that there's this whole, thing I called SOC decay, when people had a good SOC and it was good for a while, maybe after the breach it was built, and then over time, it's either nickel and dimed by the management or some other reasons, but basically, the efficiency, effectiveness, and happiness all decrease. So, my question is, give us examples about how you built it, but also how you made it stay that way because that's the tricky part for many people.

Right. So, you've heard the term, a family that plays together stays together.

Mm-hmm.

It's the same with a team, right? So, if you have a team, you have to inject fun. You have to get out of the day-to-day, you know, really serious, mundane work, and you have to make sure that you're, you know, injecting team capture the flags, do team trainings, team outings. We have a— we call the Friday debrief call, which sounds really serious, right? In reality, we're getting on a call and playing games together. So, it's a little bit of fun. You're promoting some competitiveness, but it's not within the work confines.

Yep.

And I feel that that's really important. One of the things that I would say has helped this team the most is when we hire, is we hire for a culture fit first. And technical abilities second.

Oh.

Hands down, this has been the best strategy that we've had. And I've seen more and more companies going towards this, right? You know, it's the same concept at Google. You guys call it hiring for Googliness, right? So, it's, you know, making sure that that person is a culture fit. You can teach technical skills. I'm not concerned about that. There's nothing that we do that's rocket science. You can't teach somebody to be a good person. So, we can figure out the technical piece. It's more important that the team enjoys collaborating with each other and really appreciates that person. Because one toxic person in a team can destroy an entire team. And I don't hire the people on my team. My team hires the people on the team. So, although I'm part of that process, they're making that decision, not me. Another thing that I would say is most of my team has been hired off of my posts on X. So, where—

Yes, thank you for the glowing endorsement. Yes, definitely. Makes total sense to me.

Or, you know, other social media platforms where I say, listen, don't look at this job post because so many people look at a job post and they pick out the things that they're not qualified for, and then they don't apply to the role. So, I've said, don't look at the job post. I don't care about degrees, certifications. If it sounds interesting, send me your resume. Let me decide if you're a fit or not, right? And so, overnight after the first one that I posted, I had over 200 resumes in my inbox. Oh, so then it becomes a problem of narrowing those down. But I'd rather have that problem than the right person read a job posting and decide, you know, for whatever reason, they're not a fit. Let the hiring manager decide that, not you.

That is really interesting. And sort of like, I'm still kind of processing this in my head, because It's almost like against the trends of like receive 5,000 resumes, have the bot sort them out or something like, but it's also much more logical because like your approach actually makes sense, which is why I'm so surprised it's in the minority. But I guess it's one of those things in life, right? When the kind of common sense, quote unquote, right approach is a minority, right?

Hmm. Yeah. Yeah. I like to review every resume, which at 200 was a lot. But it allows me to see the person for what they are and not what the bot thinks, right?

Yes. It's like somehow you train yourself to sort of see the energy and the person behind it and not whether they— you need qualities but they have nessus kind of thing or they have— like that would be quite immaterial to some of the stuff they need to do. But what kind of person they are would be central. Hmm.

Yeah.

Isn't it hard to get it? Sorry for like going on a detour, but isn't it kind of hard to get it from a resume? Like, It's easy to read the 16 lines of skills. Oh, you never know, are they real or not? But at least you can read it. But like, how do you read the person from that? Like, if it makes any sense.

It does. Yeah. And it's hard, right? Because there are, I've interviewed so many people that look amazing on paper. When you interview them, either you can quickly realize that, you know, they just plugged it into some template, and they really don't know the stuff that's on the resume. Or they're portraying themselves as somebody else. So, you do have to go through the process and the entire HR lifecycle of, let's do the prescreen, and then you kick it off, right? But just being able to look at the resume first, it's interesting, some of the keywords that people will put on resumes. It really tells a lot about them. And so I do feel you can get a little bit of a sense of a person from a resume, but at the end, it is that face-to-face that really is going to sell it. Yep.

And maybe what they've written, where they've spoken at conferences, like you would see more from, you know, their, to an extent, public profile and not just what's in the 2-pager. Yeah.

Yep.

So I want to switch gears to a subject which is equally painful and dear to my heart, namely SIEM.

Any—

as somebody who deals with SOCs for quite some time and in challenging environments, any views on the SIEM from you, like state of SIEM technology? Again, it's a broad question, kind of intentionally so. Maybe what are the things to consider when you're implementing a SIEM in 2024? Because the reason why we put this in is because I, you know, back to my Gartner experience, I remember people who actually are guided by the 2004 blueprint for SIEM in 2024. So, like, they are doing everything right if the calendar is wrong. Like, it's like, you can't judge that they're like, oh, no, no, no, this requirement is bad. No, the requirement is absolutely correct if it were 2004, but it isn't. So, this is more of a modern-day question, modern-day SIEM, and not just like, tell me about SIEM.

Right. Yeah, I really feel that over the next 2 years, next 5 years, it's going to change a lot. So from where we started with SIEM to where we are now is night and day difference. You think back to the 1990s, and I'm dating myself a little bit here, but, you know, it was built on a database. We were focused on the perimeter.

Mm-hmm.

And then it evolved as data processing became cheaper, you know, access was easier to logs, and then ingestion of those, those primary sources was more prevalent.

Yes.

And we saw teams start to utilize them for threat hunting and intelligence and, and things that it wasn't necessarily meant for. Right. And now we've gotten to their full-scale IR tools. They're used by many teams outside of the SOC as well. And so I feel like in the next 2 years, we're going to see this space explode and encompass a whole lot more. So we already see solutions popping up in the past, you know, 6 months, especially, you know, with more and more AI and machine learning coming to the the forefront, we've seen them start to incorporate those robust correlation engines. They're injecting the threat intelligence into the tool. There's a lot more enrichment capabilities. And then, of course, as I mentioned, the AI and machine learning.

Yep.

So I predict that, you know, in 2 years, SIEM and SOAR as we know it is going to be dead. We're going to see a new way of working emerge. And This whole concept of a rule to alerting, that one-to-one feature, and having to dig for details and do that, you know, hour-long investigation or however long it takes is not an option anymore, especially at the speed that we're seeing threats come into the environment. So, SOC teams are really looking for those value-added features that I feel that we're going to start seeing with this, with I mean, dead is, again, my former colleagues, namely Gartner, gets blamed for saying something is dead.

And suddenly, I see the perfectly good end user, you, say that it would be dead. But I think I want to sort of dwell on this a little bit and ask, do you mean, basically, you're saying that the current state of technology would be dead rather than this would be gone?

Correct.

Because ultimately, we'll look at logs anyway, whether it's the this year or 20 years before, or maybe 10 years after?

Correct. We're always going to need a quote unquote SIEM, right? There's going to be some solution there. So, the current technology would be deadened and, you know, we're just going to figure out different ways of doing it and enriching that data. I feel a lot of the SIEM going more towards data lake than, you know, database. And we're seeing the transition to, you know, like I said, those robust correlation engines to not just relying on that one-to-one alerting, but really seeing the whole picture in an enterprise environment. And, you know, not just detecting on, oh, we see a hit on a firewall, right? We're seeing a hit on our firewall. We're seeing, you know, a hit over here, somebody with the same IP address trying to hit an Azure instance, you know, so correlating that and creating a bigger picture instead of that one-to-one alerting.

Yeah. I think that it came up quite a few times in many discussions, kind of as maybe a theoretical, that, I still see a lot of SOCs which are very much built on NOC DNA when they are essentially in the alert clearing business. And it gets in my mind, they have this like big screens, all this stuff. But then you realize it's the 1980s NOC that gave birth to this initiative. And then they forget they're in a security business. They think they're in alert clearing business. And so, this one-to-one correlation becomes something they assume is to be the inherent feature of it. But then you realize, wait a second, no, why don't you focus on the actual threat and not on the alert. And they say, great advice, how do I do that? But then the devil is in the details. So the logic is correct, but the practicality is a little difficult.

Yeah, I agree.

Yeah. So I want to touch on the topic of somebody inheriting a SOC, and it can easily be maybe a whole podcast on its own, but it sometimes comes up that the good leader, like you, I don't know, for example, comes up and builds the SOC from scratch based on their vision and everything is just magically right and funded with unlimited budgets. But like, no, that's not how it happens. You may come and see a SOC built by your predecessor, who of course in Dilbert books is always wrong, but in real life would probably be half right, half wrong with some percentage. There would be things you like, there would be things you don't like. So, what's your SOC inheriting advice? Is there like a 7, 30, 90-day plan or anything like that? What do you do once you inherit a SOC?

It's hard to inherit a SOC, right?

Oh, really? Why? Nobody willed you a SOC before? No.

Right. Yeah, I would say, you know, thinking 70-30-90-day plan, right? The first 7 days, you really should just, you know, first and foremost, familiarize yourself with the team, meet your team, ask about their challenges, ask about the tech stack. What tools are in place? What processes are in place? First and foremost, obviously, understand their IR lifecycle. What are the response procedures? Do they have an incident response plan so that you're not blindsided by something that happens within those first 7 days? But then, you really— this is the time you sit back and you watch the team dynamic.

Mm-hmm.

And so many leaders feel like, all right, I've inherited this team. I have to go in. I have to immediately make my mark, you know, change all the processes. And it's really the wrong way to go about it. So, that first 7 days, you really just need to baseline the environment and the team. Then, when you move into the first 30 days, this is when you're going to do that deeper dive into the team strategy and roadmaps that are already in place or they already have planned. And so, you're going to understand those in grave detail. You know, what are the leadership goals? Why did they decide on these specific tools moving forward? What are those pain points? And then you can start to get a feel for what possibly needs to be changed. And you can even start to make and have those discussions around small corrections and small changes.

Mm-hmm.

And then I would say the first 90 days, this is where you start to dig in. You start to refine the processes. If you need to make changes, make changes. Create a long-term strategy based on your observations, taking their roadmaps into account that they've had already. And, you know, again, I would caution, don't be in a hurry to make your mark and shake things up unless you have to.

Yeah.

And there are obviously, you know, reasons why somebody inherits a SOC. So, if you were hired to shake things up, then obviously it's a different discussion, right? But if the team is not imploding and, you know, things are working well right now, then you just need to take stock of what's going on and then just adjust as needed. But those first 90 days or first 30 days really should be to sit back, and then the first 90 days is where you start making those changes. Hmm.

It sounds like you've seen examples when a new SOC leader is willing to change too much to make his or her mark, right? Like, that's— I almost didn't think about it before you said that, but now the whole thing makes sense, is that they say, oh yeah, I'm the boss now, so let's change everything. And then it's like, wait, that actually worked out really well. So like, why? Is that that syndrome we are talking about?

Yeah, it can go either way. So, I've seen it both ways. I've seen where a new leader comes in, they make changes, and it ends up going really poorly. But I've also seen it where, you know, they were hired to make those changes because things were bad to begin with. And so, you have to understand what your role is before you come into each of those cases. And it's, easy to come in and just completely revamp things. The downside of that is you are probably going to step on a lot of toes, and you're going to lose a lot of people. So, you have to, if you're one of those that you're hired to come in and shake things up and, you know, immediately make changes, understand your hiring strategy from the beginning, because you're probably going to lose at least half your team. People don't like change. Yeah.

And that's why this reminds me of the whole people who try to say, oh yeah, we need to transform. And then everybody is like, we need to transform for as long as everything stays the same the way we like. Right? Yes.

That's what we want.

And it's like, whoa, no, that's not how it works.

Right. Or, you know, you can transform everything, just not my area. Right?

Yes, that's correct.

Yeah.

Everything needs a huge change, upheaval, transformation, next generation, 3.0. Oh, but my part, my part's perfect. Don't touch that one.

Yeah.

So this was very fun. So as sometimes Tim would say, the sad part is asking the closing questions. Typically, for every episode, we ask basically 2 quick things. One piece of recommended reading. It can be anything from poetry to Google Cloud blogs. And one tip on SOC, but I think I feel like I know what tip it would be. So one piece of reading and one tip, please.

All right. So, my piece of reading is going to be actually really relevant, The First 90 Days by Michael Watkins. I feel like, especially if you are one of those people that are going to inherit a SOC, that is a book that I found very helpful.

Okay.

One tip on SOC, I would say, you know, hiring for culture fit. not for technical abilities is probably my biggest takeaway from being a leader. You can teach the technical skills. You can build a team, and you can grow tools, and you build your tech stack. But investing in your people first—

Mm-hmm.

And investing in the culture of your team is going to get you much farther, quicker. than any tool or any process is going to be able to.

Okay. That is actually quite profound. Thank you very much for doing this. And again, thank you for being on the show. This is really good, super useful, and super fun.

Yes. Thank you so much. Appreciate it.

And now we are at time. Thank you very much for listening. And of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.

Security.