SOC in a Large, Complex and Evolving Organization

Security.

Hi there, welcome to the Cloud Security Podcast by Google. Thanks so much for joining us today. Your hosts here are myself, Tim Peacock, the product manager for threat detection here in Google Cloud, and Anton Chuvakin, a reformed, tamed, captured former analyst and esteemed member of the cloud security team here at Google. You can find and subscribe to this podcast wherever podcasts are sold, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered piping hot to your inbox every Monday morning, do hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, what are we talking about today?

So today's topic is very fun. We're going to actually be talking about security operation centers and not just any SOC. We're going to talk about SOCs at large, complex, federated organizations. Think about it. It's hard to build a SOC, but imagine you're building a SOC for the organization that's changing, acquiring companies, evolving some parts of your org, digital natives, some parts of your legacy. That sounds like a tough challenge, right?

That does. How do you even staff a SOC like that?

Well, how do you staff a SOC, or do you have maybe multiple SOCs? Maybe you have some section of your SOC that does— maybe it's on-premise, and the other part is maybe outsourced. So to me, the fun part is not so much that SOC is difficult, but that a lot of decisions for security operations have to be made and remade and then reaffirmed and possibly sold to others in the organization. This is really tough. I would probably give anything to not be in this situation. So I have a huge respect for a CISO or a security leader who can operate like that.

I have a huge respect for a CISO who's even willing to try operating like that. That is incredibly brave. So perhaps with that, let's turn it over to our guest today.

Here with me is Jonathan Keith, Director of Information Security at ViacomCBS Streaming Digital. Welcome, Jonathan.

Thank you, Anton. Thank you for having me. A pleasure to be here today.

As we said, the topic is security operations center. It's been very challenging for many organizations to sort of figure out the SOC model for them. And of course, some organizations are quite complex. So maybe we'll start from the mission and charter of the SOC. Jonathan, what's the mission and charter of your SOC? Has it evolved in recent years? How are you handling this?

So our overall mission is to detect and prevent potential compromises across our applications and our infrastructure and be very proactive responding to bad actors with their techniques, their tactics, and their procedures. who are obviously trying to circumvent our defense-in-depth approach. So that's a priority for us, and basically being more proactive and reactive whenever we have some sort of indicators of compromise or attacks that are triggered within our defense mechanisms.

It's interesting that sometimes I have people who strive for proactive, but it's sometimes better to just get reactive better, right? So you may have a challenge that occurred in your environment, and you need to react faster. To me, sometimes reacting faster is is what you need, right? Proactive is nice, but I'd rather react well and immediately, right?

I agree.

And I think that's been a part of the evolution of our organization, especially with the merger of our organization over the last 18 months, that we were forced to improve in the reactionary perspective.

So one of the things that I've learned over the years dealing with security operations centers is that sometimes the challenge is kind of figuring out the direction how to improve and figuring out how to measure the SOC. So I like to bring up the maturity angle to this. I've seen some security operations centers that are very mature. They know what they're doing. They have a pretty strict process, yet they allow for endless creativity. So how would you approach looking at the maturity of your SOC? Are you up there in the 5s or are you in the middle? How do you feel about that?

I would say with, once again, with the merging of our organizations, we're in the middle. We're probably at that intermediate level. Because of the expansion to a more international footprint and also the expansion of various applications and infrastructure within our cloud environment, it's actually caused us to mature a little bit faster, but we do have processes and techniques in place where we're looking to advance. And we feel like we'll be more in that advanced level by the end of 2021.

That makes sense because a lot of people either suffered over this dramatic year or they grew up fast, right? So sometimes the challenge is that the year made them develop things much faster, right? It sounds like that's your case. You were sort of stimulated rather than damaged.

I agree. And, you know, it didn't help that we were going through a pandemic during the middle of a merger. The expansion of work from home, where probably, you know, over 75% of our employee base that worked in offices, you know, basically had to transition to work from home. So that was more devices connecting remotely. In addition to the merger of our applications into the cloud environments, that just caused the process to expedite.

Yeah, no, that's a good lesson to learn for many organizations. It's kind of like don't let crisis go to waste kind of thing. But you had a double whammy, merger and pandemic, right?

Exactly. Exactly.

When I was a Gartner analyst, I've dealt with some organizations that are very centralized, almost army-like, where whatever they decide at the top, it gets done to an extent. But there are also organizations that are very decentralized, complex, and some organizations that are in the middle of a merger presumably are moving through many moving pieces. They move down complex trajectories. So your organization is definitely complex and decentralized, right? How do you do SOC in this case? SOC implies Security Operations Center, right? But it sounds centralized, but you are decentralized and you are kind of evolving. So how do you deal with that? That's a tough one.

Correct. So there are some challenges to that complexity and that decentralization. For example, response times, as well as, you know, having a single source of truth and accuracy of information when you have multiple resources that are providing you with alerting and monitoring.

Mm-hmm.

So how we dealt with that is through collaboration of our internal security teams. We did an awesome job, you know, working with the different leaderships across information security, as well as the business partners, and setting up collaboration models with us, like, You know, for example, having incident response channels through our Slack collaboration, basically having, you know, real-time exercises where basically we were prepared in cases of incidents around events such as the Super Bowl as well as March Madness, basically going over some lessons learned and just perfecting that process of collaboration and synergy. And from that, it just made that decentralization a little bit more centralized, to be honest, but it provided some compensating controls where we were a little spaced out.

Does it mean that you sort of approached solving the problems by trying to be more centralized, like have more coordinators, or is it more like a federated model when pieces just magically came together? So did you push for more centralization to solve these problems, or did you just push for more coordination within these similar pieces?

Oh, always push for more centralization. We're all about synergy with the organization.

Okay.

But we realized that we had to somewhat still have somewhat of a hybrid model. Because the size of our organization, especially with the international expansion, that just doesn't happen overnight.

Yes, and it does make sense. This is the tricky part for many organizations. I've dealt with some, again, in my Gartner days, I've dealt with some that were just federated. There was no center of control and there was no way to make one. So they had to do a lot of moving pieces that talk to each other as opposed to push for more centralization. So I'm actually happy that you found that path. This is really tricky.

It absolutely is tricky.

So one other thing about security operation centers that I've encountered is that, well, some people think of SOC as kind of a central monitoring tool, like a SIEM and maybe other things. I like to think of a SOC as primarily a team. So to me, SOC is a team. SOC is not something to have as a service because ultimately it's a group of people. So SOC is a group of people, a team, a bunch of processes and technology. To me, it's perhaps all 3, like a triangle of people, process, tech. So how do you approach this in your organization when you are sort of decentralized? You may have multiple spots of tech, multiple types of process, and of course multiple groups of people. So how do you do people, process, tech in your case?

Like I said, we are decentralized and we do have various people within our incident response team. We actually have 2 different incident response teams. One is more focused on the cloud environment and the cloud infrastructure ecosystem. One is more focused on some of our traditional still, you know, data center deployments and applications and more legacy technology. But once again, we do collaborate and communicate through that hybrid model where we do have centralization through some of the tools that we use, you know, using CrowdStrike, using Sumo Logic, for example, where both groups will have access and real-time access to alerts, monitoring, and then we can quickly communicate with one another and collaborate if there is some sort of incident or some sort of indication of compromise.

So does it mean that you build some kind of workflows where you have to hand off certain things? Like, for example, if one team detects something in the cloud, but they need support from the team that manages, say, laptops, there would be some kind of a Visio diagram with boxes, or is it more of an ad hoc approach? So people just would know what to do. I'm curious about this.

There is a procedure in place. So an incident would trigger and it would go through our initial incident response. You know, we have a centralized inbox with that, or if there's an actual detection by our SOC team. And through that investigation, we would determine where that asset, where that application, where that resource actually resides. If it's more desktop-related, then the desktop team would work in correlation with the incident response. If it's more of a cloud resource, then, you know, my organization that manages the cloud security would then be alerted, and then we would respond and work directly with the centralized SOC team. to be able to complete that investigation. Because in a lot of cases, you know, we have access to the central resources to be able to provide that information.

Okay. No, that does make sense to me. This is kind of the coordination is the trick again. So nobody goes on vacation without leaving the next contact and stuff like that. So this ends up being a challenge.

Exactly. And it's always the next person up, you know, if someone is not available to make sure they know their role and their responsibility.

And as long as that chain works, I think you're in good shape. Yeah, that makes sense to me. So here's another thing. You mentioned hybrid, and I think that in our past conversations you mentioned that you do use a degree of outsourcing or third-party providers in your SOC. So to me, again, during my Gartner days, I've kind of coined a line or a phrase that today almost every SOC is a hybrid SOC because almost everybody uses some form of third-party partnering, third-party providers, MSSPs, MDRs, co-managed shops, consulting. So how do you approach the role of outsourcing in your SOC? You do run a hybrid SOC. And you have a partner, I know, but like, how does partner work with various internal teams and how do you make this all a happy family so there's no counter-blaming and finger-pointing and stuff like that?

So we do use partners. One partner in particular, we do use E&Y to assist us with some of our alerting and monitoring, especially around events such as, you know, March Madness and Super Bowl. And they are a part of our workflow as a part of our SOC program and IR incident response. So if they were alerted or they found an indicator of attack or indicator of compromise, then their job would be to go to that next step of that workflow and then communicate and contact someone internally within our IR team and basically share the information with them. And from there, that would launch the investigation to the proper channels, the proper business partners, and proper security personnel.

Okay. So you do include them in the workflow. To me, this is probably a useful takeaway for others. And that's been my favorite highlight, again, in my analyst days, that if you do what people would call outsourcing, you have to run joint operations with them. Otherwise, you approach it as a kind of a Coke machine. You throw coins, security come out, and that's not going to work. It never did. It never will.

They're more than just a vending machine. They're actually embedded in the workflow.

Yes. It's almost like you're quoting from my old Gartner MSS How to Work with an MSSP paper, because this is exactly the trick that works. And the other pronunciations are kind of iffy. So thank you for confirming that.

I'm sure I studied some of your periodicals.

Another fun part is, of course, the cloud. In recent weeks, I've encountered quite a few SOCs that were really challenged by having to look at threats from the cloud, investigate in the cloud, collect telemetry from the cloud. It's been really agonizing for some traditional kind of like old school SOCs to start being responsible for GCP monitoring or other cloud monitoring. So I know cloud is included in your team's charter, and obviously you do cover threats in GCP and possibly other clouds. How did it happen? So how, how are you integrating cloud in your SOC mission and charter? And how was cloud migration or cloud evolution happened in this regard?

So from a cloud evolution and migration standpoint of view, the ViacomCBS streaming world is 100% in the cloud. So we definitely had to come up with a model with our SOC program that was conducive to the cloud deployments and our cloud infrastructure. So because resources and applications are ephemeral and often immutable in the cloud, then we had to basically have some sort of continuous pipeline to be able to not only handle logging, but to be able to handle indexing of those logs and being able to set up real-time triggers and alerts around Cloud Logs, Cloud Trails, anything that comes in from a stack driver, and be able to set up modules associated with those triggers to where we make sure the proper personnel is being alerted to when those triggers occur. So it was the speed of reaction. You know, you talked about reaction earlier, but that was the major difference between being in the cloud and being in the data center, that we had to be agile and work really fast before we lost, you know, potential resource or we lost logging that could help us with the investigation through the SOC.

Yeah, that makes sense. But wouldn't you have to acquire cloud knowledge to do that? Like if my SOC has been involved with like Linux and Windows and I don't know, mainframes, and suddenly you're like, here's microservice, here are the cloud access logs from GCP. Like how did that happen? To me, this has really tripped many SOCs.

Absolutely. There was training involved with personnel. You know, there were personnel that had experience and background in cloud security, cloud investigation, and cloud migration. There were personnel that did not. So there definitely was a knowledge gap. So we took the time to do the proper training and not only do proper training, just make sure that individuals are able to test that new knowledge, you know, in Q&A and dev environments before they actually got a chance to actually apply it in production. So it took some time. As we were doing the cloud migrations, we were also training personnel on how to respond and react to the difference in infrastructure and applications residing in the cloud as opposed to a traditional on-prem.

I'm kind of happy you found the path down in that direction because I've seen people who are really lost on their way to the cloud, that's for sure.

And training is still in progress. It's just never-ending because things always change in the cloud environment.

Very much the case. So it's good that cloud providers like us provide some free training and some of course is maybe not free, but the logic is that we do want people to learn the cloud way before they go rather than do it in the on-premise way. So despite all that, I'm sure there's still challenges with the current model and there are certain things you're kind of trying to improve or boost the effectiveness of. Kind of a double question. What are the challenges with the current model that you see? And where are you going in terms of improvement? Sort of like current possible bad and future good.

I still think the biggest challenge is with decentralization, you take the risk of response times being slow. And I have seen some situations. Fortunately, we still had pretty decent response time, but there were some delayed reactions because through the investigation, You know, one person was trying to figure out who was truly responsible for investigating the incident. So the response times were slow because of that decentralization. And then also the accuracy sometimes of information through the various tools that we use, because tools do not always align with functionality and features. So you don't necessarily always have a single source of truth around logs and metadata within that log. So, you know, that's still a challenge that we are addressing. We do have a consolidation initiative that's basically being pushed down from the top down to where we are trying to compensate that, and in some cases eliminate it, where we're all basically using the same tool sources, we're all using the same services, we're all merging and reorging into coexisting teams to where those lines of roles and responsibilities are identified a lot better. And therefore, we have improvement to that workflow that I mentioned earlier As it relates to incident response.

Yes, that makes sense. But any kind of more strategic changes, strategic improvements you're planning, discarding tools in some big way or redoing, retooling or changing processes, any kind of big plans for making it better?

Yeah, I think the biggest thing is we want to make sure we have a true SIEM tool. You know, one thing that we kind of struggled with over the years was our security incident and event management tools where we have multiple dashboards, and we prefer to have that single pane of glass. So we're currently going through several proof concepts. Google has been very helpful with us in using, you know, some of the feature sets of Chronicle. And then we still do, from an enterprise standpoint of view, have a tool like Sumo Logic. So we're basically just, you know, doing some bake-offs and some proof of concepts because I think ultimately having that single pane of glass, having that true security incident and event management tool and services, That is next generation, next level, basically gives you that visibility that you need to start off with any investigation, whether it's an IOC or an IOA, by having that resource availability.

We're kind of almost at time, so let me hit you with maybe one more question. Any quick recommendations to others, either security leaders or SOC leaders? What's your advice from your lessons? Like, what's your— what you wished you knew maybe in the past?

To be honest, this is not futuristic. This has been security for, you know, 30-plus years, is the people and the technology, making sure that's consolidation between the two within your security organization, making sure that your people are properly trained, that your people clearly understand the mission and the purpose of your SOC, making sure that they clearly understand the tools and the technology and the functionality of it so it assists them. Through more automation of their investigation. And then, just making sure that you have the proper tools and the proper technology at hand with the support of not only just the security team but also making sure that you have the support from the top down, from executive management. So ultimately, you know, you're trying to achieve your mission and your vision and your goal, which is delivering secure content to your customers, and that is a shared responsibility.

It's actually a perfect ending for a discussion. So thank you very much, Jonathan, and hopefully the audience would learn something about how to make their security operation centers better and how to improve for the future.

It was a pleasure. Thank you.

And now we are at time. Thank you very much for listening. This episode recording also became a Security Summit session, Fireside Chat, SOC of the Future. So if you want to watch the video, we're going to post a link to the show at the resource section. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us at our website. And yes, people continue sending us jokes about the website. We have a website now, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. And of course, your hosts are on Twitter as well, Anton_Chuvakin and _TimPico. Tweet at us, email us, argue with us. And if you like or hate what you hear, We can invite you to the next episode. See you on the next Cloud Security Podcast episode. Security.