EP73 Your SOC Is Dead? Evolve to Output-driven Detect and Respond!

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Tim Peacock, the product manager for threat detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and esteemed member of our cloud security team here at Google. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you like our content and want it delivered to you piping hot every Monday afternoon Pacific time, please hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, I feel like today's episode is a little bit of that old Mark Twain quote about reports of demise being greatly exaggerated. We're talking about SOC and what SOC means in the future, It doesn't mean that SOC is over. It just means that it's going to evolve, right?

Yes. But at the same time, our guests in the past have occasionally written and spoken on things related to SOC. And I think in at least one of the posts, he did say that in his opinion, maybe, drum roll, SOC is dead.

Long live the SOC.

Long live the SOC. Well, there is that. Yes.

So, I think there's really interesting stuff in here. And I think what's kind of great is The parts of SOC that he describes having go away are in many ways parts of SOC that, especially for cloud natives, we would like to see go away.

Correct. And some of his examples are taken from those very cloud-native companies where the SOC did in fact go away, at least in the original form, circa 2002 SOC, a big room with a bunch of people in expensive chairs.

Well, perhaps with that just archaic visual in mind, let's welcome Today's guest.

And our guest today is Eric Block, Senior Director of Detection and Response at Sprinklr. Welcome to the podcast, Eric.

Hey, thanks for having me. Happy to be here.

The topic today is going to be focused on SOC, but not just on SOC, possibly on SOC being dead. Let's not jump ahead, though. So, Eric, you recently coined a term or a concept of output-driven DNR and even perhaps output-driven security. So how does it work and why is it better?

I think that might be leading the witness, why is it better, but I am curious about what it is.

Oh, it's fine. Just stick to what it is. is and how does it work also?

Well, what I've been noticing as companies move from on-prem data centers and having kind of their own in-house tech stacks to the public cloud, how we do initially detections and response changed. It went from being, you know, you had to have a SIEM and roll your own detections because everyone's data center was a kind of a unique snowflake to them. When you move to the cloud, everyone's infrastructure is common. So the outputs coming from your CSPs, your cloud providers, are known and they're common. And we're starting to see vendors today start to take these outputs that are known and have kind of known outputs from their product, I guess, outcomes. And I think that for most cloud providers today, because they're fully aware of what the output is, they could actually help drive a lot of these outcomes. And so rather than having to roll your own detections, roll your own GRC, roll your own threat and vulnerability management, since the outputs are known, we should be able to derive these outcomes from the cloud providers. So the cloud providers are actually delivering similar to how you guys do with your event-based threat detection. You guys are just giving an output. There's no inputs required. There's not a lot of rolling your own. The same thing for how some of the vendors are producing your TVM data or GRC data from your cloud. They know what the outcome is that you want, and they're delivering the outcome without you having to go in and roll your own like you would in an on-prem data center.

So by the way, here's somewhat of an inside joke, mostly for me and Tim, I guess. Eric just voted on the side of Tim in our long-running debate about custom detections. And I am really starting to get scared because—

I wouldn't say he voted for me. I actually wanted to pick at that. So Eric, you mentioned the event detection that we do in Google Cloud. Are you a user of that?

I'm not actually. If we were just in GCP, I probably would be. But since we use all 3 of the major cloud providers, that's where I had to go with a more agnostic tool that could speak kind of all 3 languages. But I am a fan of it.

So this is very interesting to me. So you chose a common output across your cloud surface rather than cloud-specific tooling that then your analysts would have to translate on a per-cloud basis?

Exactly.

Okay. Now, if you were a single cloud user and you had gone with that event product, would you feel uncomfortable not being able to see the exact logic that the rules used, or would you accept that as part of the upside of a managed security approach?

As long as I could customize some of it, like add my own things, because I mean, everybody, you're going to have your little one-offs. The rest of it, I'd probably be okay with it. I draw the analogy to like your endpoint protection software on your laptop. Like, I couldn't tell you how, you know, whatever the product is works or what the rules they have or what machine learning engines they're using on the backside, but it seems to work and stop all the bad things. So after a while, you just kind of have to, you know, put your hand over your eyes and, you know, just trust they're doing the right thing.

This isn't surprising to me, and Anton and I are making faces at each other, listeners, because he and I argue about this outside of the podcast all the time. I think the analogy here is a little bit like the shift from, say, writing assembly to writing Python. At some point we accepted, I don't really know what the compiler is going to do with my address layout and my memory management, but that's okay because my life is better this way anyway.

Oh yeah, totally.

Yeah.

I mean, it's a similar analogy to the AV software. I mean, you don't always know what's happening under the hood, but, you know, we have to establish trust with the vendors that we're working with. I mean, trust is kind of the foundation for how the internet works in general. We trust our SaaS providers. Like, we use Google today. We trust you guys are doing the right thing and vice versa.

Yeah, absolutely. So I want to shift gears a little bit and talk about something I've heard you say. I've heard you say that SOC is dead. And now I've met a lot of living SOCs. So could you clarify what you mean by that?

Again, it's drawing from my experience moving from, you know, my last job at Salesforce where we were a large, you know, we had our own cloud, our own data centers. As we move from there to the public cloud, again, we started to realize that the way you do detections, the things you care about, and the things you watch are vastly different than on-prem, again, where everything's unique to you. When you move to the cloud, things are much more common. And we started seeing, again, a lot of these third-party agnostic vendors, they could translate the Googles and the Amazons and the Microsofts of the world into a common language and give us a common output that we could actually make actionable and use. And so as I started to dive more and more into this, I started to realize that the cloud providers are providing us again with a shared common infrastructure, like the same buttons and knobs and dials I have to push in GCP are the same ones you do. Therefore, we should be able to map the outcomes we want coming out of these clouds and have those outcomes delivered without needing a SOC. So if something comes out, say, like, horrible examples, just like an open object someplace like the Google Object Store or an S3 bucket or whatever it is, that's wide open to the whole world.

Mm-hmm.

That gets dropped in your SOC traditionally, right? The SOC guys have no idea. What are they going to do with it? They're going to go ask the team responsible for it, hey, should this be wide open or not? They don't know. And then they let the other team make the decision and they take the action. And what I'm saying is, hey, we know what that output is. We know that's an open object store. We know who needs to handle this. It's probably your tech ops or networking or NOC team. How about we land this with them instead of having everything be kind of default dumped into your SOC? Because that team is a team that actually has the knowledge and know-how to address the issue. And instead of having your SOC decide where this goes and deciding who to escalate it to, let's land the issue with a team that can actually fix it and then train them how to escalate back to your CERT team or your SOC function, whatever you want to call it, if something is out of whack.

Hmm.

And you can do this across everything coming out of your cloud. Your GRC, you can have metrics and reporting go to them. For TVM information, this could go to these guys. For a lot of the new behaviors and activities you see in your cloud environments, from either the console or the endpoints, Most of the time, again, your security team aren't going to be the people that actually can ask the question, is this right or wrong? You're going to be asking somebody else. So let's land the problems with those guys that can give us the thumbs up or thumbs down, and then train them how to escalate back to your security team if something's out of whack. And so that's kind of the basic premise for it.

But so this is a federated/decentralized approach. So basically you're not killing the S in SOC, you're not killing the O, you're killing the C. You're killing the center, essentially.

I don't think it's dead in his model. I think it just has a more narrowly tailored set of things to deal with.

Yes.

The false positives get farmed out, absorbed at the lower layers. This is perfect. I love this model.

That gives me warm fuzzies.

Yes. But so the question is, is it workable for everybody? To me, when you federate, you practice this as a federated approach to DNR where the outcomes are routed to the teams that need them or can address them. What if I am very centralized? What if my organization is very top-down? So is this workable for every organization that uses cloud or for every org, period? Like, tell me more about where this federated approach would apply.

Well, I've seen it applied at other businesses as well. I mean, I'm, you know, cobbling together this information from other companies. Like, I work with, you know, Alex over at Netflix on his SOC-less approach and swathing those guys over there. And I've talked to some people at Facebook when my old boss was over there. about how they decentralize things too. And it's not to the same level, but different bits and pieces of this seem to be working. So when I came to Sprinklr, I mean, I knew I wasn't going to have a 100-person security team, so I needed a way to do things smarter, more efficient. The other teams that we already work with today, they already have tier 1 functions, right? Your SRE team exists. You have a NOC that monitors your network stack, whatever. You have an IT help desk. These guys already know how to do tier 1. They already want the work. So for public cloud companies, I can see this being an easier transition from a centralized kind of SOC model to distributing the load and having your security team focus on the actual issues that they have the skills and time to address versus them just becoming a dumping ground. For some conventional legacy companies that still have data centers, that's a lot more difficult, right? Because you don't have that common infrastructure. You have your unique— it's unique to you. So in that case, I'm not entirely sure if it would work. I mean, unless you had literally all your outcomes known and derived, which again, coming from a unique infrastructure is a lot more harder than coming from a public cloud provider.

But for a second, you sounded like this. Let's pick 3 companies at random, Facebook, Netflix, Salesforce. Based on this experience, clearly this approach is workable for everybody. Okay, I am giving you back a cynical version of what you said, but I think that the point is that there would be some kind of adjustment elements to make the federated approach this type of distributed. Also, who writes the routing logic? It would still be, in my mind, a SOC, as Tim said, narrower mission. No?

No, no, no. You're not thinking cloud. In cloud, it's so easy to write the routing logic. It's whoever owns the project. It's whoever is responsible for this part of your org hierarchy. You got to think cloud with this one.

Yeah. It's similar to what we're doing today is we're working with our partner teams and saying, hey, these are all the outputs that are coming out. What are the outcomes you need? How can we take the given output, like some vulnerability data, say, right, comes out of your cloud provider? Using our automation platform, we enrich this information to produce the outcome that the vulnerability team wants. Like, okay, I want to know the delta between vulnerabilities yesterday and today. Okay, cool. I can grab the data out. My automation team can make that happen, and I can deliver them the outcome they need without ever having to pass through my SOC or touch any of my security guys.

That makes a ton of sense.

And you can apply the same model to GRC, you can apply the same model to misconfigurations, configuration management, new users being added to your environments, new environments being stood up. All these things are basically changes, and somewhere someone's in charge of that change. There's a change control ticket that went through, your tech ops guys are doing it, your NOC knows about it, and it applies all across the board. The only thing that lands on my security team are new behaviors. Here's a guy who's logged in as root and lateral moved and installed the compiler and is compiling code in root. Okay, that should probably go to the security team because we're going to have the context and know-how how to handle that versus, hey, a new user got added and he has access to all your environments. Is this legit or not? I don't know. Let's ask the NOC or the TechOps guys who made it. Is there a change control ticket? Well, my automation guys are going to pull that information for them, deliver them this outcome so they can make the decision. If it's a bad decision or if they say, look, this shouldn't have happened, they can escalate back to us and then we'll take appropriate action, whack the guy and talk to his manager or whatever. But we're looking for those guys that have the skills and the knowledge to tell us what's right and wrong. And then we can either address it or they can address it.

Okay. What about the concern that I sometimes hear when I try to present a similar model? People bring up some kind of a separation of duty, vaguely defined. Admittedly, they vaguely define it. And they say, wait a second, I can't have people who build/run the tech handle security issues in this. I have my own counterpoint to this, but I'd love to hear What's yours? Because ultimately, to me, this argument kind of on paper ruins the model. That means you can't give it to individual teams because their stakeholders, they may know it, but how can they be trusted? I think that's my retelling of this. Tim is making a face at me, but yeah.

No, we've had those questions as well. As we've done this, we've realized that we've had other processes that don't work correctly. So like, you know, change control process, right? If that works correctly, there shouldn't be any, you know, conflict of interest. There should be a separation of duties. So if I'm gonna make a new user that has access to all my cloud environments, say, right? Well, there should be a change control ticket for this. It should be opened by somebody. Somebody in authority should vet and approve this thing. And another person should actually be the person implementing, making this new user. When we pass this back to, say, the NOC or TechOps team, right? What's the first thing they're going to do? They're going to look for this change control ticket. Has this been approved by somebody? Has this been vetted? If not, this is probably a bad change. The tier 1 functions, the NOC or the TechOps or IT, whatever, they don't have any context around, oh, this is my buddy Bob who made this account because he's doing something, right? They're just looking at the black and white. Here's a ticket, this is what it says. I'm going to vet and verify this and give you the answer. And that's pointed out to us that we had some broken controls. Our change control process wasn't working correctly because they couldn't give us a yes or no answer. So that pointed out additional flaws that we had to go back and fix. And as we fix these things, we're noticing the process improve. But again, there's a lot of your process that's been placed. There has to be the whole cultural aspect of your partner teams wanting to do this with you. It has to be bought off by your executive leadership. So there are some hurdles to this. But I don't see separation of duties as being necessarily one of them.

I think that's a compelling answer. What do you think?

I'll buy that. Yeah. I mean, I don't know if a typical European banking auditor would buy that, but I would buy that.

Those guys don't buy anything.

Well, that's my point. They think cloud is some kind of a scary thing that is coming from the future.

I don't know.

I don't want to make too many snide remarks about them. But the point is that, yeah, there are some difficult arguments to be made. Recently, something happened to me when I was trying to explain that if you have a SIEM and you want your people who respond to alerts contribute to writing the rules, and somebody from just a bank just like that stood up and said, this is absolutely impossible because our auditors told us we cannot combine people who write the rules and who respond to alerts because that's a separation of duty violation. He didn't really say it like that.

You should absolutely have those people doing the same thing. That's crazy. So I want to roll back to something I said that also happens to be one of the questions we wanted to ask here, which is this approach sounds extremely cloud native. You know, like I immediately went to using your org hierarchy to route tickets. That's a very cloud native concept, not an on-prem concept. So does this delegated SOC model or partner SOC model, does it only work in the cloud or can people on-prem use it?

Initially, I had assumed it would be far easier to roll out to a cloud environment. And I kind of assumed that would be the only place people would want to roll it out to. Since then, I've actually had conversations with people that are hybrid, that either have their own data centers or moving to the cloud, or a particular bank who just has all their own data centers. And it comes down to a matter of having the correct tooling in place, the cultural issues, having a culture that would change and embrace this is important, and kind of having everyone rowing behind this model. So I have talked to a few companies that are I'm gonna say legacy, they have their own data centers, that want to do this. To me, it seems like it's a really heavy lift, again, because you have so many custom things in your environment. I'm not gonna say it's impossible, but I think it'd be far, far easier to do in a public cloud environment.

And so, you're referring to not just that cloud is kind of a monoculture in that sense, even though it's a bad word, but that you can have the universality of outcomes in the cloud, but it's very hard to make it in a very snowflakey data center where everything is pets, not cattle.

Exactly, yeah.

Basically, maybe that's what it is. Maybe it's the cattle in Maybe if you run your on-prem like that, it's going to be fine.

If you knew all the outputs and outcomes that your on-prem data centers and tools could provide, like, yeah, you could map it. It'd be a one-off version of this, but you could totally do it versus going out for cloud. Again, you can go out and buy, you know, there's dozens of third-party vendors that can take, you know, all your data from, you know, all 3 major cloud providers and it can translate, you know, Google, Microsoft, and Amazon into a common format for you. Which makes it far easier and a much easier lift where you're basically paying somebody some money to do this for you versus in your own data center environment where you're having to do it for yourself. And I see that as a heavy lift. So I'm not saying it's impossible, but public cloud, I think the cattle model would make it far easier.

That does make sense. And I think that it's probably not for cloud only, but it's kind of the, if you practice modern IT model that cloud is basically pioneering, then you can practice this.

That's how I originally came at this, yeah, is because it's common across everybody because they are public cloud providers, it'd be far easier. And it wasn't until later on that I started looking back and I'm like, well, maybe this is possible in conventional data centers and conventional networks. But how you do it, I mean, that starts getting into a mess, a lot of heavy lifting, a lot of one-offs.

By the way, before we go further with the questions we have, I kind of want to go through a few more examples of— I fear that the audience may not always have a clear visual for the outputs or outcomes. So, for example, Cloud Audit logs would be an input, like a log type or EDR trail. That's an input. But an output would be what? Detecting permission changes without authorization? Like, give me a few things that you would want to use in your output or outcome-driven thinking.

Sure. What we're doing here with the tooling we have today for security events, the things we're not farming into other people, We'll take the security event, say it's a new connection to an unknown host or something like that, right? We know what the output of the tool is going to give us. We know what the outcome is you want to see. The outcome is going to be, okay, we want to have this alert. We want to know— we want to look at the connection, the place they connected to. We want to see the application that actually reached out to connect to it. We want to see who did it, what user it was. We want all the context built around the output of this new connection. Based on that, we can merge all that data together to actually produce the outcome we want to see, right? Which is a fully contextualized alert that we can kind of pre-vet by, again, doing repudiation of IPs or domain names and seeing where it's connecting to, how often it's connecting. So when it does land with one of my security guys, they can quickly look at it and make a determination of, okay, I need to look into this deeper or not. So it's a matter of taking the outputs from your tools, enriching them, or futzing with them, tweaking them to produce the outcome that every team needs.

FUTSing with. Is that on the marketing page for a lot of SOAR tools? I think it probably should be.

Totally is.

Yeah.

FUTSing, tweaking, it's all in there. Yeah. I'm pretty sure it's probably on the last page.

And so the responsibility for building the FUTSing, does that also roll back into the teams that are doing the response?

Right now we haven't quite got that far yet. Right now it's still my security team doing it. We're partnering with our partner teams, like with the EOTech Ops NT or whatnot, to figure out what they need. to work with their models. Because again, our NOC already is tier 1. They already have their own runbooks. They already have their own procedures. And it's not a matter of us saying, hey, look, we're throwing this up over the fence. We're working with them to say, hey, how can we deliver an outcome that you guys can make actionable immediately? Just insert it into your workflow. We'll give you a new runbook and you guys can handle it. So a lot of it was learning how our partner teams did their work. I mean, because, you know, the IT team does something different than the TechOps team. TechOps works different than the NOC. You're not working for the engineering team. So it was a lot of us listening, a lot of conversations to figure out how they work and how we can integrate with that work. I'm hoping to get to a point eventually where those teams can do that. When there's new output that comes from one of our tools, they can build their own workflows to produce the outcomes they want so they can deliver it to themselves. But we're not quite there yet.

But it means a lot of mutual learning is kind of built in and is required in this system. Like if security doesn't care about what the networking guy does, And if the networking person doesn't care what the application developer does, this probably cannot work. But if there's mutual learning going on, then it can work or maybe should work. Is it fair?

Yeah. I mean, well, if none of the teams care to begin with, I mean, you probably have other problems, right? Your wheels are falling off the car. But this has actually been really good because my team has gotten to work with all the other teams, all the other partners and stakeholders in the team, and it's brought us closer. Like these guys want to work with us. Like we have, you take your NOC guys, right? Your tier 1 NOC guys who are just responding to networking events. They're excited. They're going to get to look at security events now. It's going to kind of broaden their horizon, right? It's going to give these guys a different path out of the NOC. Maybe once they get experience handling security issues, maybe they want to go be a security analyst someplace rather than being a tech ops guy. They're excited to see it. They're excited that they get to handle the outcomes rather than having kind of the things thrown over the fence at them all the time. They're getting to work closer with us in security. The same thing with all of our teams. It's building those relationships. At first, it was rocky, but now everyone's on board with this. They're excited to do it.

Okay. So what I love about this, so often I see vendors pitching something when what they ought to be pitching is therapy time for security teams and engineering teams.

Culture change or therapy time. Yeah. This is back to the same exact lesson that came in many, many successful episodes we've done, is that it sort of A lot of these things, especially when cloud migration transition is involved, it's like culture change, therapy time, culture change, therapy time, culture change, therapy time.

And we've seen what happens when you don't do that during that migration as well.

Oh.

You end up patching containers in production and looking like a real jabroni.

And 400 firewall appliances in the cloud and patching containers in prod.

Yeah. So this is great.

Yeah.

So I want to maybe go take a look at this formula field. In the past, there were some kind of attempts, let's put it this way, where security team was positioned as a broker decision maker while all the work was done in the individual teams. Like, oh, security just writes policy and then somebody somewhere does the work. To me, it has ended very badly in the past. And there are even, you know, books written about how bad it can get. And I see sort of glimpses of the same model, but modernized and without the negatives. At least that's how it's in my mind. So a practical question to you, Eric. How can this model work with a security team who wants to do the work, who kind of like, no, we're going to go do it. We wouldn't trust the networking team. We wouldn't trust the team. We have to go hack it ourselves. How do you change that view?

That was my first step here, is getting people to embrace this new model. And what I found, I mean, to your guys' earlier point, it's a cultural change. I have to be able to Clearly articulate the vision, clearly articulate where we're going to be at in a year or 2 years from now when this is working. And I have to get the team, not only my team, but my partner teams to actually buy into that vision. And that's probably the most difficult part, which again goes back to therapy time, which you guys were just saying. And it kind of illustrates that almost all problems in security are actually people problems. They're not actually technology problems. I mean, what widget am I going to deploy into this environment? It's not a matter of what widget it is. It's a matter of getting people to agree on it. And this is the same thing. I mean, I had to get people to understand the vision. I had to be able to articulate it. I had to get them to buy into it. Once they did, that made it a lot easier.

What was hard in that process and how did you get over those objections?

The hardest part was me clearly articulating this vision because it was fairly new and stumbling over myself. But once I had kind of my elevator pitch down and people could understand it, then people started falling in line like, yeah, this totally makes sense. I don't see why we're doing it any other way. Again, it's just like pitching a startup, right? Once you found that 30-second pitch, people could understand it and buy into it. You're like, aha, I got it. It was the same thing.

That makes a ton of sense.

Yeah, I pitched my pitch to, you know, on Slack to probably 100 people until people were like, oh, I totally get that.

That sounds about right. So we're just about at time, and I want to ask you our famous closing questions. Uh-oh. First, do you have one weird tip for people that are trying to go down this path? And then 2, do you have recommended reading so that we don't leave people empty-handed?

Tips for people going down this path. First, I would, if this is something you're interested in doing, make sure that your company is in a position to do it. First off, I mean, again, if you're at a 500,000-person bank and you're still running mainframes, you're probably in the wrong place. If you're in a smaller SaaS company with a couple hundred, a couple thousand people, and you're completely public cloud, maybe it makes sense for you to explore this. To get people to buy into this, you need to be able to Paint your vision and clearly and simply articulate it to get people to buy into it. So that'd be my kind of tip if you're interested in doing this.

As far as books go, further reading, we're going to link to your articles for sure because they're fascinating, but we'll have 2 of those.

But anything else besides the articles? I mean, a lot of the things that I've kind of gleaned from this has been, you know, Alex Mastriani's old Netflix blog around, you know, his sockless approach a couple of years ago when he was kind of pioneering this. I've read some articles on how kind of SRE engineers do things, kind of based off some of the links that you've put out there and your learnings from SREs. Those, and I mean, a lot of the books I've actually read are more around kind of people skills, kind of how to paint that vision, right? How to get people to buy into it. And those were actually far more valuable to me than the actual technology side of it. Because again, I mean, whatever vision you're trying to paint, unless you can articulate it, fairly easily and simply so people can buy into it, you're not going to get anywhere.

That makes sense.

I'm not sure if that helps or not.

No, that absolutely helps. The trick, I think, often for security people is how do we get them to be more persuasive with people rather than with getting code execution.

Yeah. Simon Sinek has some good books on leadership and people influencing. Jay Shetty's Think Like a Monk book is pretty good, how to kind of straighten your own mind out so you can get your thoughts out. So there's all kinds of really good people books out there.

Well, Eric, thank you so much for joining us today. This was, I think, a useful conversation for me and Anton in our long-running arguments, but I also hope a good conversation for our listeners. So thank you.

Awesome. Thanks for having me, guys.

Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @anton_chuvakin and @_timpego. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Bye.

Security.

Security.