EP64 Security Operations Center: The People Side and How to Do it Right

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Timothy Peacock, the product manager for threat detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and esteemed member of the cloud security team here at Google. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday afternoon Pacific time, please do hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, we're talking about SOC again today. This is a perennial favorite topic of ours.

And yes, and it does score pretty well on the stats, so it's a fun topic clearly. But Today we are talking about a part of SOC that to me was always a little bit more mysterious, but I always highlighted it as more critical. So it's kind of more tricky and more important. And that angle is people.

What about people?

What about people? Well, we know we can't have the automated SOC anytime soon or probably ever. So we are talking about who do we want in our SOC? What kind of people? How do we train them? How do we get them? How do we keep them happy? So it's kind of the people aspect of the SOC that's always been very difficult for many organizations.

That makes a lot of sense. Despite our wonderful conversation with Amos and others on the show about automation, we are a long way from the Terminator-operated SOC. And so the care and feeding of SOC operators remains a critical part of your SOC success strategy. And perhaps with that, let's welcome today's guest.

Principal Security Strategist at Google Cloud. I'm almost envious about the title. Welcome, Dave. And also welcome to Google, by the way.

Thank you. Thank you so much for having me. I'm super excited to be here with you guys today, and I'm glad to be a new employee at Google.

So the topic is security operations center again, and we wanted to visit an area that personally I always considered to be my weakness because I deal with processes, I deal with tech. And in my Gartner days, when we wrote something about SOC and there were some intricacies about SOC analyst training or rotas or 24/7 routines, it's always somebody else handled it. So I'm really excited to ask you some tough questions about the people side of SOC. How about we start from the training? If I am hiring somebody to run in my modern security operations center, how should I think of training for those people?

Yeah, I think in my background over the last 7 years, you know, I had an opportunity to build some programs that trained lots and lots of blue teamers or, you know, security defensive folks and people that work in SOCs traditionally. So through kind of a lot of A lot of learnings, a lot of sort of trial and error, we came up with some things that I think are really effective. I think the thing that's most important from my perspective is teaching people how to investigate and how to sort of ask a good investigative question, how to then look at what data they need to answer those questions, how to stay on track or how to pivot and when to pivot, things like that. I think Those are the types of things you often find that they don't have classes or formal training on those types of things. Whereas maybe somebody goes through a programming class or something like that. But I think investigation is the thing that is kind of the most universal skill that's important to teach to your SOC personnel.

Is that a skill you can teach? A lot of what you just described sounds like judgment or instinct or feel or intuition. How do you teach that effectively?

Yeah, in my opinion, that's a common misconception that it's like, hey—

Oh, good. I'm common to that.

That you either have that skill or you don't. I think it's a skill that you can acquire, and the real key to acquiring that is, is having practice, having at-bats in sort of realistic situations. And I think providing those realistic situations, that's really the challenge. How do you provide a realistic situation to train someone so that they can develop those skills.

That resonates with me because early in my career, I worked for a guy where he called situations in advance. He predicted things so well. And one day I say, Mark, how do you do this? And he looks at me and he's just like, it's pattern recognition. I've done it before.

Yeah.

So some of this is really just practice and at-bats. It's that easy.

Well, I think certainly when you encounter someone like Mark, I don't know Mark, but when you encounter those people in the security operations center, really What you find out about that person's background is they've just been doing it for a long time. They've had those opportunities. And I think that when you encounter a SOC that has maybe the concept of different tiers, tier 1, tier 2, tier 3, those types of things, what happens is that tier 1, their job is to often triage an event when it comes in. And the good ones, right? And I don't say good in quotes, usually they're bad. But the interesting ones that represent a real attack usually are escalated pretty quickly from a tier 1 person. And they're sort of the old Saturday Night Live, like, move. They're kind of pushed to the side, right? And then there's sort of a more experienced person takes over, right? And so one thing that I've learned, and you can identify those folks pretty well, right? The tier 1 folks who don't get those opportunities because they're like, oh, thank you so much for providing this realistic situation for me to learn in where I wasn't just sort of immediately displaced or, you know, pushed out of the process. And so, I mean, that's where I think SOC training can be super valuable to give those people those opportunities.

So wait a second, when you say realistic situations, you don't just mean, oh, they should have had the experience in real life. You're talking about sort of a simulated slash made-up environment because Briefly, I thought, well, they should have had the experience, but we can engineer those experiences, right? It's not like they have to have lived through that, right?

Yeah, I mean, this is an area where, like, I've done a lot of work in the past, sort of engineering those experiences for folks so that they can get better. You know, I think the key to doing that is just making it super realistic and as realistic as possible. You know, there's some key things I think that are important there, things like setting up a lab to generate data. Generally speaking, you're going to create some data that represents a real security incident. And so the value there, right, the way that you create value, I think, in those training situations is like doing things like getting real vendor gear, right, that maybe represents what you have in your environment. You know, maybe using a gold image from an endpoint that represents, you know, what a user might be working on every day, buying the cloud services, right? And doing all these things at a smaller scale, but trying to set up things that are super representative of the environment that you're trying to protect. And then running real exploits with proper isolation and all that, of course, but making sure that the telemetry that you're collecting is generated from what you're likely to encounter in the real world. So, I've run into a lot of people who try to generate data for these types of training. So like, hey, I could program that, right? I could write some Python or I could do a template that can substitute in IP addresses and things like that.

Mm-hmm.

It's very difficult to do that. It's easy to do that for maybe triggering a simple rule, but when you're trying to train someone to pivot through related data sources, it becomes very hard to make that data look realistic and have the those pivot points match up, right? An IP address appearing in 2 different log sources and that sort of thing. And that's perhaps the most important, or one of the most important nuances in training someone how to investigate, is that pivoting. And so there's nothing like the real thing as far as data goes for training your SOC.

That makes a lot of sense.

It does. But it doesn't depend on the model of the SOC, because you mentioned level 1, level 2s, right? And we do see more of the metrics mesh organized SOCs when there are no No funnel, but people sort of like focused on skills, not on levels. So do you think training would be affected by the structure of your SOC or not?

I'm a dad of 3 soccer-playing daughters. I have a wife who is a soccer coach, and she always tells me this thing. I don't know much about soccer, but my wife, who's a coach, always tells me, when you're teaching a kid how to play soccer, the best thing that you can teach them is ball skills, right? These are things that they can do no matter what team or what scheme some coach puts them in in the future. They're going to remember how to dribble and how to shoot, how to pass the ball. These are things that you kind of can't take away from a young soccer player, right? And I think of SOC analysts in the same way sometimes when I think no matter if you're in a really rigidly tiered SOC or if maybe you're in more of what I would call, and you, I think, Anton, would call a more modern SOC.

Mm-hmm.

There's certain investigative skills that are useful regardless of that sort of scheme that you're put in. And so I always pivot back to how do we teach people how to be good investigators? And that tends to sort of transcend some of those things. Now, you know, I think another important thing to keep in mind is if we have a virtual SOC, then you should probably do your training with everybody virtual.

Hmm.

Or if you have a SOC where you have, say, maybe a blend of contractors maybe working in your SOC, well, you should include them in your training as well because You know, when you encounter these real events, these are the folks who are going to be involved, right? So they should kind of train the way that you operate, if you will.

That makes a lot of sense to make your training realistic. What exactly is the ball handling skills for a SOC analyst? I'm not sure I followed that one through. What were the things that are universally useful skills that people should focus on training up?

Yeah, I think it's investigation. And when I say investigation, I'm talking about, okay, How do I formulate a question that can help me move forward in my understanding of what actually happened?

Hmm.

Obviously, you have an overall objective when you're doing investigation. We want to figure out what happened, but I think breaking that down into little parts, maybe like, so how did this adversary gain access? How did they move laterally? How did they identify maybe assets of value, and how did they exfiltrate data? Creating those questions and then applying, like, okay, how do I know what data that I have at my disposal that can help me answer those questions? And then answering them and then sort of progressing through. I think a big thing that a lot of analysts maybe don't understand is how to pivot.

Hmm.

I think pivoting can be, obviously, it's sort of the key skill in investigation, but it can also be sort of a primary way that you waste a lot of time in an investigation by following things that don't really matter. And I think that's a key skill. How do I stay on target? Or how do I pivot? Because those are the kinds of things that are critical in determining how effective you're going to be and how fast you're going to be at coming to some useful conclusion.

So it sounds like it's a lot about almost cognitive abilities, almost like— I shouldn't say general smarts, but it's less about whether they know a particular vendor's tools more, right? So we wanted to also ask you whether the training should be about the toolset or more about the skills and attitudes. I think I know how you'd answer it, but are we going too far in this direction? Like, surely you should know your EDR, your SIEM, your threat intel platform, your APIs. How— if I think about the balance between I know the product I don't even know anything else. I don't even know security. I just know how to configure a product, blah. That's too far for sure. But if I'm really smart and understand how people reach conclusions, yeah, that's too far on the other side. So how do you balance this vendor-specific, tool-specific to skills?

I think you need to practice your investigative skills, but you need to do that on your toolchain that you have in your SOC. And so I look at that as if I'm a leader in a security operations center, What I want to do is really hold my vendors accountable for providing both types of training. So certainly I need to train my staff on how to keep the toolchain up and running and, you know, make it so that it's reliable and operating correctly and not costing us more money than we had budgeted and all those sort of practical things. But I think also you need to hold your vendor accountable for helping teach these sort of generic skills kind of against the canvas of the toolchain that you're going to be working in. So, I mean, you have to know how to run a search, right? You have to know how to run a performance search in the tool that you have because, I mean, you can write a great search, what you think is a great search that might answer all your questions, but it might take 3 days to run.

Yeah.

And so then it's not a great search. So understanding how to do those things in the toolchain that you have, and I think it's a blend of understanding the tooling, but also understanding what are the generic skills of just understanding how cyber incidents unfold and being able to ask those types of questions against the technology stack that you have in place.

That performant search question is such an interesting one. I was once in an organization where we were talking about cup of coffee queries.

Oh yes, I've had that.

And we realized that our customer had queries that took so long they could go get a cup of coffee. But with our ELK stack, our queries cost a cup of coffee each time you hit search because it was using so much compute. And so either way, searching for things cost you a cup of coffee. I want to ask something a little more controversial because we haven't done that yet. Do you think SOC analysts need to know how to write code? And do they need to be able to code in order to succeed in a modern SOC?

Yeah, this is a great question. I think—

thank you.

We have to be super careful when we answer this question because I don't think that you need to code to get into the SOC. And I think that if we say that you do have to code to get into a SOC, we're doing ourselves a big disservice.

We're definitely not helping the talent gap.

We're not helping the talent gap, and we're probably solidifying some sort of a bias against people of certain backgrounds and people that have different Skills and experience, right?

Listeners can't hear this, but I'm nodding my head in violent agreement right now.

Yeah, so I don't think you need to code to get into the SOC, but I do think that you eventually need to code to be successful and grow inside of a SOC, without a doubt. I think another thing that's really important is to understand how we define coding, right? Coding can be a lot of different things, right? I'm firmly in the camp of HTML and CSS is coding, Is a Splunk query coding?

I think we just dated us. Oh no, Tim, you went there.

I went there.

I went there. I'm just gonna go straight there.

Is a freaking multi-page query coding? Great. I've seen something of 5 freaking pages.

Yeah. Is writing SPL coding?

Of paper.

Yeah. I mean, I think there's obvious things like Python is coding and Go is coding. Is writing a great search in YARA-L coding? I think you mentioned SQL.

He meant SQL. He meant to be vendor neutral.

Or SQL.

He meant SQL, I'm pretty sure.

Yeah, sorry, I didn't say SPL.

I said SQL. SQL.

Sure. All these domain languages, query languages, the language of your toolset, that's absolutely coding. And so, I mean, we have detection engineers, and detection engineers, you know, they express their work in the form of SPL or SQL or YARA-L.

And there's non-trivial logic in all 3. There is non-trivial logic in all 3. Like, it's not just like SELECT FROM table, which probably frankly wouldn't be coding, but there's non-trivial logic in many queries. So to me, maybe that's why I would latch my argument to this, right?

Yeah, you're absolutely right. Ultimately, when you're a detection engineer, you are trying to say, hey, how do I detect this advanced attack? And a lot of times there's timing, there's combinations of different events that have certain relationships with one another. And there's sort of unraveling all those requirements and then creating detections in your language to identify those. So to me, that's coding. I mean, people can fight me on that, I suppose, and say that's not Python, or, you know, there's not a for loop in there or something.

Well, I want to pick at something else there. I want to pick at kind of a move you just made, and I don't know if you did this on purpose or if this was like a head fake, but we've been talking about SOCs And now you're talking about detection engineers. Is there a difference between a SOC analyst and a detection engineer? Are those the same job or are they different?

Well, I think today they are by and large different jobs. So I think when I talk to folks, and I've had the good fortune of having a lot of experience with a lot of different organizations, a lot of different SOCs, and I think you do see this sort of separation, this dichotomy between content engineer or detection engineer, if you will, And an analyst, right? And the detection engineer's job is to create great detections, and the analyst's job is by and large to deal with the output of those detections. I think moving forward, and, you know, I think this is like an autonomic security operations kind of tenet, right, is that those roles will come together so that the SOC analyst of the future will be some sort of a combination of a detection engineer and an analyst.

Mm-hmm.

I think that that is a very obvious thing to me. I mean, I see some security operations centers doing that already where you have, you know, you can wear multiple different hats. But to me, like, if we're looking to retain SOC analysts, you know, one thing we have to eliminate is sort of the repetitive tasks that, that are just grueling for a human being to go through, right? And dealing with Alerts all day can be that, but if you say, hey, part of your day, maybe 50% of your day, if we're talking like an SRE model, right? Maybe 50% of your day is making those detections better, and then 50% of your day is dealing with the output of those detections. I think that's something— it certainly represents a job that I would want to do a lot more. And I think there's a lot of other benefits to that as well.

Listeners who've been listening carefully to all our episodes will probably recall having us host Julian, Vihant from Google's SST, and his position was describing exactly that kind of lifestyle. Google, our SOC, and they're all gonna be mad at me when they hear that I said this, Google SOC, our team called SST, they live exactly that life. 40% of their time is writing new logic, 40% is doing triage of the logic.

Yep.

They don't have the kind of agent-principle problem of somebody writing rules and somebody else being victimized by their output. They are victimizing themselves all day long.

Which creates the whole loop of improvement. And that's the thinking we sort of try to express and make more portable in the autonomous security vision. But that's where the idea came from ultimately, to see that.

Yeah.

So here's the kind of a, I guess, fun side angle to this. A lot of the red side offense kind of improves when they play all sorts of games, when they do contests, capture the flag. And it's a very self-serving, very Dave-serving question rather. So what's the equivalent of this for the blue side? What is the CTF for the blue side and how do we make it a reality for more people?

Yeah, I think forever and ever, and I don't think it's a secret, I came from Splunk. So I worked at Splunk for a long time. We worked on a pretty big capture the flag there that was blue team focused, and I was one of the co-creators of that. But one of the things that I looked at and we looked at when we were creating that is like there really weren't a lot of blue team defensive-centric capture the flag events out there. I would say there's some very notable exceptions to that. So things like SANS Holiday Hack Challenge is a blue team. It's not exactly capture the flag, but it's a gamified experience where you're progressing through a series of blue team challenges. I think SANS NetWars, also a sort of a foundational blue team competition. I think more and more you're seeing this move towards gamification and capture the flag events. for training Blue Team. I mean, they're super powerful, right? Like, competition is powerful. There's no doubt about it. The first time you sort of see people competing in a competition that you created and they're doing things like, you know, there's 1,000 people in a room all with their heads down and you can like hear a pin drop because they're working so hard because of the competition.

Mm-hmm.

Or they're like, hey, I'm not gonna, you know, eat or drink or, you know, get up to use the restroom because, you know, I'm trying to score points before the timer expires. So it's a super powerful thing. You have to be very careful, I think, in how you apply competition. I mentioned Chris Sanders earlier, somebody who's just a friend to everybody in the industry, I think. And he's done a lot of work on understanding how does the human brain actually work most effectively in general, and how does it work when you are using the human brain in Things like you encounter in a SOC, right? He's made some comments recently on gamification and motivation for— motivation for learning, right? And, you know, he characterized competition as it's motivation for learning, but it's not actually learning. So when you're competing, you're not necessarily learning, but it can be a motivation for you to learn. But it can have, I think, downsides as well. Like, it can demotivate people. As well. So you have to be really careful.

If they lose a lot or if they win too easily, just curious.

It's a great point. Both of those things, I think. You know, if people are like, hey, I don't want to compete because I'm not elite or, you know, I'm not a ninja, I'm not super experienced, right? And they might just say, look, I'm just not going to participate in this because it's a turnoff, right? So I think you need to be super careful in applying competition Especially when you have sort of a variety of folks like early in their career, later in their career, and folks that are more experienced, less experienced. I mean, it's one of the difficult things is making a competition be sort of open and welcoming to people of all different levels. And then there's ways that you can design that into the system and still keep it competitive, but still minimize that sort of downside sometimes that can be actually demotivating to people.

So I want to pick on that point as well. I feel like I'm picking on everything today. This is great. You mentioned welcoming people earlier on. We talked about how coding as a bar for entry, you must be this code competent to ride the SOC, not especially inclusive. How do we build more inclusive, bring more underrepresented people into security? We had a great episode where we talked about how like not having diverse perspectives on your team is literally a security risk. So what can we do in SOC, which is so often an entry point into our industry, to bring people in, into the fold?

Yeah, well, I mean, I'll answer that question, I think, in the context of this episode, which is where we're talking about training. And yeah, you know, I think a big part of making training useful at all is creating scenarios. There's a story, there's characters that exist in that story. Those characters have different traits, they have different backgrounds, and, you know, maybe they're the people who are Those characters are the ones who are investigating the event, or they're the ones who maybe caused it either purposely or they played some role in, in that event unfolding, right? And I think those stories are a big part of what makes these types of training events very successful. But it's super easy to make those types of scenarios actually be non-inclusive.

Mm-hmm.

Choosing situations that don't represent everyone. They don't represent certain types of folks. And, you know, something as simple as what you might think is an inside joke that you threw into your CTF can actually be something that, unbeknownst to you, maybe because you didn't spend time thinking about it, but can actually be something that's exclusive, right? So, you know, I think thinking about those things, thinking about things like English as a second language, keeping things very clear when you're using language and not using lots of things that, hey, that makes sense in American English, but, you know, our friends from India who speak English as well, maybe they didn't get that little nuance, right, that you tried to throw in there because you're using like super complex sentence structure or these things, right?

So if we were letting an Australian friend write the sock example, for example, they wouldn't get away with saying the footprints looked like the person was wearing a thong. That would be very sensible in Australia and not work at all here.

That's great. I was going to say that.

I thought, yep, yep.

It's not my podcast.

I thought I'd let the host drop a joke like that. I spent the last couple of days hanging out with Australians, so I'm biased. Okay. So that's an awesome answer. And I think that's a really interesting aspect to all of this. We're getting up on time and I know this has been full of practical tips, but our traditional closing questions are, Do you have recommended reading, and what's one tip to improve your SOC training?

Yeah, there's increasingly so much stuff available out there, so I definitely recommend a few things. One is use your red team. So when I talked about creating scenarios, like, a lot of big organizations have a red team, and their job is to mimic attackers. And so use them to help you kind of create scenarios for your SOC. Don't over-rotate on infrastructure for these types of things, definitely use CTFD, which is kind of an easy way to build the way that you ask and answer questions and score points and those types of things. If I had to offer some suggestions, I would definitely recommend the Chris Sanders Investigation Theory class that he offers. It's actually super high value and relatively low cost compared to a lot of other options, but I mean, he goes into a lot of detail about cognitive biases that are common pretty much in every, every human brain, you know, how to identify those and defeat those so that you can come to better conclusions. I mean, I always recommend SANS Holiday Hack Challenge, even though it's not the holidays. You can just mute the sort of techno Christmas music that plays in the background.

Oh God.

But you can go through like literally weeks and weeks and weeks of challenges and learning all about different aspects of cybersecurity. The SANS class on the SOC analyst class, it's Security 450. It's written by John Hubbard. Anton, I know you know John.

Yep.

And that's fantastic. If you can get a SANS class, do it. If your organization will pay for that somehow, or there's some way that you can get in there and take that class, you should. It's fantastic. But I would also caution, you don't have to take SANS classes to get into security. They are very expensive. But if you somehow have that opportunity, By all means, take it. There is the Kronos Challenge as well, which is a gamified challenge that allows you to work through investigative scenarios using Chronicle and the Google Cloud Security Suite. And so I highly recommend that as well. It was very nice to find that when I arrived here. It's a really, really well-done gamified SOC training resource, so highly recommend that.

Amazing.

Perfect.

Dave, thank you so much for joining us today. This was a wide-ranging, deep, and super enjoyable discussion.

Thank you both for having me.

Perfect. Thank you. And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security. Security.