EP143 Cloud Security Remediation: The Biggest Headache?

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Timothy Peacock, the Senior PM for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of the Cloud Security Podcast listeners on LinkedIn. Anton, this was a fun episode because first we aired it live on LinkedIn. It's available as a video on the YouTube channel. It's now dropping as a podcast, and we are talking about remediation, which is a pretty interesting topic.

It is.

And I think that my impression was that cloud remediation, just as remediation for vulnerabilities on-premise, is about a process a lot more than about the tech. But we do have a tech vendor as a guest. So it's going to be interesting because ultimately, when you have an area of security with massive process breakdowns and somebody shows up and says, I got a product to fix that, my initial reaction is I'm skeptical. In reality, you can engineer your way out of procedural problems, and people have done it. So in that sense, my skepticism may be misplaced. You'd know that in about 30 minutes.

And so maybe listeners, with that, whether Anton's skepticism is well-placed or misplaced, let's welcome today's guest. Today we are joined by Tomer, co-founder, CTO at Dazs. Tomer, thank you so much for joining us today. Delighted to have you. I know it's late where you are, so we really appreciate you making the time. I've been doing cloud security here at Google for— actually, yesterday was my 5-year Googleversary, which, as Anton said, is pretty good. One of the things that I see all the time with my customers is that the challenge when it comes to the posture side or the secure-to-start side is not so much figuring out what the problems are, but fixing the problems. Is that concordant with your understanding of the world?

It's such a leading question, Tim, by the way. It's like you founded a remediation company and you're like, Dude, do you think remediation matters?

Does remediation matter? Now, listeners, I will say this is a question Anton wrote for the write-up.

Okay, fine, fine.

But Tomer, please, was I right? Did I lead you into the right place?

Yes, definitely. I mean, I'm personally very passionate about this particular space, having started a company in this space. But yeah, I think this is a really problematic area. If you think about the cloud security industry in general, we've had quite a few years of improving the way that we detect potential vulnerabilities, potential misconfigurations, potential different kinds of risks and findings. But at the end of the day, we need to action that data. And we're working with clients with tens of millions of different findings from a lot of different scanners, a lot of different types of tools. And at the end of the day, the problem becomes, hey, how can I actually action that data? How do I prioritize? How do I find who is the right person that can actually fix this? And how do I guide them through the process of fixing all of these vulnerabilities? This is what stops them, essentially. Hmm.

So wait. As the way I recall it from the good old times of on-premise vulnerability management, a lot of brokenness was due to process, right? It's not like scanners usually would scan on-premise and patch management would usually patch, but people in between and organizational dynamics, who owns what. So the way I recall it from the good old times, it was process breakdowns, not tech breakdowns. But here you have a vulnerability remediation technology company. So how can Can tech fix something, or is it the same in the cloud? Or maybe I'm asking too many questions.

That was a lot of questions at once. I kind of lost track of how many you asked there, Anton.

Okay, so let me try this. So is—

Is it a process issue?

Yeah. Is vulnerability management problems a process issue? Whether it's true in the cloud as well? And if it's a process issue, can tech fix it?

No, that's good. That's good. That's good.

Thank you, Tim. I'm such a— suddenly I'm more clear.

I think the usual paradigm is between people, process, and technology in a lot of different aspects. It's not a single thing that can fix it all. I think that we're approaching a reality in which maybe technology will be able to fix the whole thing with generative AI, with better automations, with a lot of different things that contribute to the ability to actually automate that process end-to-end. But right now, if I'm looking at the majority of our clients, it's a combination of all. I think the primary difference between the good old days of vulnerability management is that by adopting the cloud, we've seen the industry picking up a much more distributed models of setting up infrastructure and setting up different development environments and actually building applications, right? So right now, if we're thinking about security being very consolidated, generally speaking, in most enterprises, and we have a lot of different DevOps teams, and we have a lot of different development teams, they have to communicate in this one-to-very-many mapping right now internally, just to be able to understand how can we drive remediation. It's not like a point-and-click solution that you can literally patch some host. In the cloud right now because you can't really change the infrastructure behind the engineer's back. They would definitely not like that and things would simply break. So yes, there's a lot of technology behind there. There's a lot of process issues in being able to set something like this up. And there's a lot of confusion with a lot of people. This is why we're trying to solve this because if we can surface the right data, if we can actually automate all of these different steps in the process, if we can actually speak the right language with the right stakeholder that needs to do that next step, We can actually increase the remediation velocity drastically.

So, you said kind of a keyword that took me back for a second to actually an episode, listeners, that I found the most challenging to host because of how far out on the edge our guest was, Sunil Yu. One of his things about cloud is that it has to be what he calls DIE, distributed immutable ephemeral. And I don't know if he's talking about distributed responsibility in that, but you were talking about distributed responsibility. It also, of course, all of this infrastructure is ephemeral and immutable. So, how does vulnerability management change on those 2 axes of cloud as compared to on-prem? Like, the distributed responsibility makes sense.

That's a pretty profound question, right? Because ultimately, the point-and-click or call a server admin never really worked on-premise either because there's no one server admin, and it was distributed in the same sense. But I think that this is, like, way different scale. So, yes, I'm super curious to hear the answer.

We talked about how it's distributed. Let's move to immutability. It's also really interesting because let's compare a classic Linux host with a vulnerability 10 years ago. Let's say we wanted to build some kind of an automated remediation process. We'd probably build some kind of an automated job that SSHs into that host and runs apt-get upgrade on that machine. And hopefully it doesn't really break anything, right? Because—

Fingers crossed.

Once it breaks, then yeah. And I think the reason people are so scared of automated remediation is because they actually tried this. And then after a couple of times doing this, they realized everything breaks up and the infrastructure people definitely didn't like that. So I think what's happening right now with immutability is that even if we're trying to replicate the same construct, we can't really do this. Because let's say we have a running container right now. We can't really SSH into that container. It doesn't really make any sense. If we need to operate—

Well, actually, wait, wait, wait, wait. Sorry for interrupting. Do you remember, Tim, this mind-blowing moment? when Anna told us that there's SSH in 50% of containers they observe, right?

Yes.

Which really blew my mind. And I wanted to say, is it the same as 50% of zoos have a dragon in them? Like, to me, the statistic sounded like this. But of course, Tomer represents sanity. In the sane world, people don't SSH into containers. But apparently, in the real world, occasionally they do, right?

No, they do. And we see patching in containers with our products as well. But Tomer, you're right. They shouldn't be. They really shouldn't be.

I mean, if you have an immutable environment, this is definitely the absolute opposite Of doing the right things. So, in an ideal world, hopefully—

We need this on a poster. An absolute opposite of doing the right thing. SSH into container. I want this on a poster. SSH into container... An absolute opposite of doing the right thing.

We'll print some limited edition posters after this podcast.

Yeah.

Sounds great.

You can get them on our website, listeners. We'll open up an Etsy shop in addition to the YouTube channel. Okay. So, with the immutable thing, though, we shouldn't be patching containers.

We should be Rebuilding, redeploying.

What about the ephemeral aspect? How does that influence our vulnerability remediation in the cloud?

So in that aspect, I think what really is confusing for SecOps teams is that the infrastructure constantly changes. Like if we go back to the same container example, if I've seen one vulnerability right now on a particular container that was live when I ran the scan, it could be an hour ago, it could be 24 hours ago, that container might not be there at all. Right now, I might've lost track just because of the point in time in which I scanned the environment, it's no longer relevant. And it adds additional complexity because if you think about all of the DIE aspects of this, then I'm seeing a critical vulnerability on a container. It's not there. It might be there tomorrow. It might have rolled out to production because of a change that just happened in the CI/CD platform from a development team that I don't even know.

Hmm.

And to be able to remediate that vulnerability, I need to go chase them down, ask them to rebuild their container, which I have no clue if it's going to spin up ever again. So there's quite a bit of complexity in there, I would say.

But this is a major process breakdown. If you copy your mental model from on-prem VM to cloud VM, you're going to be in deep— what's the word we say on the air?

Doodoo. Doodoo.

Thank you, Tim.

Yeah.

You would be in deep doo-doo, right?

Yeah.

Thank you. The technical term is this.

Yeah, that's right.

So, thank you for this. Yes.

I think this is a really good illustration of what Sunil was getting at, and it's really nice to bring it back to that and make it concrete for people, because I think his episode was maybe a little abstract. This is really interesting. I want to get back to one of our questions that we wanted to ask, and maybe this is going to seem obvious having just had the conversation we had, but we wanted to ask you, why is remediation in the cloud such a headache? And aside from all the challenges that come with distributed systems and distributed responsibility, ephemeral infrastructure that comes and goes as you scan it, and immutable images, which are Kind of not designed to be patched in production. Aside from all that, what else makes it a headache for people?

I think technology is actually a big barrier. It's also a function of how distributed development is, because any development team can pick up whichever technology they want. But we're seeing a lot for organizations to pick up a multi-cloud strategy. So, it means that as an analyst, I need to know so many different things about so many different classes of technologies in 3 different cloud providers at the very least. Sometimes they have 4 or 5. It means that the cognitive load that I will have as a cloud security engineer or as a cloud security analyst is huge. I need to be a mega DevOps practitioner in every single space.

And no security and no risks and no threats.

And no cloud.

And I have to stay up to date about all of these new technologies, right?

And with each cloud, which is each different in its own way. Good luck.

Yeah.

And I mean, to really add insult to injury, you may have to learn one of the clouds from the long tail, right? I don't know. I don't wanna name names, but, like, you know, these are the names of clouds you don't know. Like, you need to know them.

Right.

Okay. So that sounds, like, full of both insult and injury. I believe that.

Yeah. And I think that the development teams are not making it any easier, right? Like, they can run fast and break things. They can pick up any new technology, any new library, any new CI/CD platform, any new cool programming language they want to start working with, but they inherently or indirectly add additional risk to the business because their security teams don't really know how to handle all of that. And now they need to build that muscle to be able to understand what are the implications of what we're doing right now.

Sure. Does your security team's vulnerability tool support Rust and Kotlin and whatever else people are picking up these days? Maybe not.

But actually, this, funny enough, at this point, I sort of self-answered my original question about process breakdown versus tooling because in this environment that Tomer describes, it's very clear that the new type of tool will be needed. And I know, I'm sorry for sounding like a marketing person for Dazs. I'm really not. But it's very clear that the tech stack change is required. You have to teach people way too much realistically.

Did you just say a new type of tooling is needed?

Yes.

You just said that.

Well, I'm not a Gartner analyst anymore. I can say things like that, right?

So, how does the new tooling work? What does it actually look like? How do we make this not suck?

So, what we're trying to do is to consolidate as much data as we can to be able to solve these kinds of things. We're trying to bring in information both from the development world as well as the security world and not just keep them siloed. And if we can correlate between the two, we can pivot back and forth between those different views. We can take a vulnerability in production and find where is this coming from, or we can take a particular resource definition and ask, well, Where does it eventually roll up to and what is the potential impact? What is the potential risk associated with a particular vulnerability? Really being able to solidify all of these aspects together, it gives us the ability to understand how to speak both languages. It also goes back to process, which you asked about before, right? We can't really have our security findings popping into ServiceNow while all of our development teams are using Jira or GitHub. It doesn't really make sense. We need somehow to actually get—

ServiceNow though is progressive. Try PDF.

But I agree, you have to meet people where they are.

Just as a minor comment on the side note.

Exactly. And it's fine. I mean, if the change management process is happening in ServiceNow, or if it's happening in PDFs, so be it. If you really want to empower the organization to fix security vulnerabilities, you have to speak their language. You have to meet them where they are. You have to enable a process that delivers the data to their own tools. And then pick it up from there. I think there's a big process element into it too. It's like you have different tentacles into all of the scanners, all of the DevOps, all of the ticketing tools, all of the status of the pull requests to be able to really understand how that picture looks like from the point of detection up to the point that it's fixed.

This actually makes sense. And I think one of the bigger things that you mentioned is that security teams, even on-prem, often have friction with IT ops, you know, even before DevOps, go back to the '90s, like there would be friction between security and the rest of IT. Here, it's like the skill gap and other types of friction and not understanding the environment, security teams sticking to old tools sometimes, the DevOps running ahead. So, it's almost like the friction between security and engineering teams and DevOps teams is kind of a huge contributing factor to cloud remediation being harder, right? So, it sounds like your platform also plays into this type of a bridge-building model between the two. But how bad can it get in terms of friction? Yeah, I guess it can get pretty bad. So, but like, what are the examples? What are some examples of the frictions between, say, security and DevOps or engineering?

Well, I heard some stories that I will not quote from internal VP-level meetings in some organizations. And it really gets up to these levels because unfortunately, some organizations really function in that way where they think, well, we're going to scan everything, take all of these vulnerabilities and just throw them over the fence. And if no one is doing anything, we're going to be there and blaming them for not fixing. a healthy process and creates frustration on both sides. It is so critical if you're thinking about how can we actually reduce risk, you have to think about all of the different pieces of that puzzle and make sure no one is frustrated with the way that they get data and everyone actually has some kind of a positive feedback loop in that process. Otherwise, they won't like you. Think about a typical development team where they have automated tickets being sent over their backlog.

Nobody loves it, like on the whole planet, right? Nobody loves it.

I was talking to one engineering leader back in the day, and he was like, I can't really find my feature requests. I have so many different CVEs in this backlog. They think that everything I do is just fix all of their security vulnerabilities all day long. And then I go into the first one and it's like, okay, there's a vulnerability here. I'll upgrade this package and it fixed it. And it goes to the second ticket, and there's another CVE there, but I already upgraded that package, so this is a duplicate to me. I don't care if it's CVE-123 or CVE-1234.

They have to be grouped by the package that fixes it. Yeah.

Exactly.

Yeah. This is the actionability of response that's needed.

Exactly.

But this is like the scanner vendor solved it for on-prem 12 years ago, perhaps. So it's interesting how This is coming back to bite us in the cloud because the environment is different, right? Like, I remember, I mean, don't hold it against me, but I used to work for Qualys many years ago. And I remember this type of reporting based on what patch remediated, and it was literally 15 years ago. So, it's interesting how this is still a thing in the cloud.

It's becoming harder and harder, right? Let's go back to the containers example. Imagine an architecture with a bunch of applications, a bunch of microservices, great. And then, Now let's say we scan it with any kind of tool. We can pick up Qualys just based on the example, but we can pick up GCP SCC if we're looking for the cloud native options, right? We have a bunch of vulnerabilities in all of these containers, and now we need to actually figure out what is the common denominator amongst all of them? Because we can take the application and say, well, here's the application team that is responsible for that application. Here's a list of CVEs. And then they may prioritize it for the next sprint. They will spend a few days looking into this and then they'll realize, hey, you know what? We're just relying on a base image that's coming from the platform team. It's not really us. And actually right now we don't really have anything to do because we're waiting for them to take the fix before we can rebuild the container to be able to push it to prod. So essentially they don't really have anything to do. We just wasted their time. We took sprint points from their sprint to be able to just find the root cause for a bunch of problems. And we've done this for every single application in our architecture. So we took, let's say, 4 points from each development team's sprint to be able to do nothing, just investigate the root cause for a bunch of problems that they have zero action. Everyone just got back to us to go chase down the platform team, ask them to rebuild or repackage the package or the base image, whatever it is, and then ask the rest of the application teams just to rebuild and redeploy. It's not an easy process. It's happening because of good reasons, right? We want to increase development velocity. We want those developers to actually build and ship new features fast. We want them to reuse code. We want them to use all these abstractions, all these open-source components. But we have to keep that in mind when we pivot to how do they actually fix it and how can we group all these findings together in an actionable way.

That makes a ton of sense, and it really speaks to the need of security teams to understand how development works. And we had a Actually, Kelly mentioned that she's met CISOs who don't know what unit tests are. And so given that kind of lack of knowledge of how development works, it's not super surprising to me despite what Anton's saying. And I will not hold Qualys against him because Qualys gave me many nice bags to collect other swag at Black Hat over the years. Value add for sure. I'm not surprised that people get this wrong today on security teams who don't understand how development works. So maybe with that, we want to shift gears a little bit and talk about the CSPM side of this as well. When it comes to understanding vulnerabilities in your cloud configuration, how are you thinking about that? Are you really focused on the software side, or are you also looking at the cloud config side?

I think you have to look at all of them together because if we think about—

that's a very cloud truth that still is not getting to many on-premise people. This is so good.

It's another possible poster Anton has gone from marketing to marketing posters. This is great. He's going to hit up the FedEx store after this.

That's awesome.

But really about the CSPM stuff, in addition to the software stuff.

I think all of the examples we talked about today are fairly simple. We talked only about one type of workload. We talked about a very simple environment that repeats itself, but in reality, if we're thinking about a typical cloud-native environment, we would actually want to let every single development team, every single DevOps team to decide on anything from infrastructure, to OS, to containers, to serverless, to the technology that they're using to build their code, right? And this is where it gets interesting because actually, if you think about the process for remediation, for actually fixing all of these misconfigurations and CVEs and whatever it is, it's actually very similar. Like, I could have painted the same picture that we talked about before with a containerized environment and just talk about Terraform. So instead of having a bunch of different containers, let's say we have a bunch of different databases that are misconfigured, they're not encrypted, But in reality, my organization is supposed to use blueprints, and we have a platform team that builds those blueprints for all of the different teams. So I get to the same root cause analysis problem. I have the same problem with attributing the right vulnerabilities to the right teams. And I still need to be able to explain to them in simple terms, like, here's the Terraform module that you need to modify. Here's the configuration file. Here's the right way to configure this. Here's what I propose. And if I can't do this, then I'm not really helping them remediate faster.

So it's really the same thing.

Yeah, it's the exact same process, right? Because from a process standpoint, you think about everything that happens from the point in time that you know about a misconfiguration, vulnerability, anything else, right? So, if we chase down that rabbit hole of what happens when I'm seeing a new misconfiguration, I still need to triage it. I still need prioritization. I still need to find who owns this. I still need to communicate it to them in some kind of way. I still need to convert it to something that they actually understand. I still need to track the fact that they modified what they needed to modify and repromoted it to production. And I want to Verify that the new scan comes out clean. It's the same process.

Yeah.

I think there's much less tooling in that space, which makes it even more complicated. But yeah, very similar.

I love that answer because to me they're so obviously similar, but that's because I've been cloud native my whole life.

You live in the cloud. Yes, that's right.

Yeah.

But I think what we're starting to see, which is really interesting, is that some organizations actually consolidate their cloud security and application security teams for that reason. Because it doesn't really make sense to look at those in silos. If I have a CVE coming from a software composition analysis scanner, it might be the same CVE that I'm seeing on my CWPP. So I want to be able to engineer the same process for all of these tools together.

I hate to do this.

We are both speechless, but I think it's actually a really good answer because I think that it's sort of like some of the cloud realities are kind of clear and profound once you live them. But if you are teleported or time traveled from an on-premise land, a lot of stuff looks weird. But once you understand it, it's really simple. Back to kind of megatrends, that cloud really is simple if you understand it the cloudy way.

Yeah.

Okay, Tim, go with the closing question. I think we need to wrap.

I hate to do it because this has been so much fun, but it's late where you are, so I'm sure you appreciate it. First, do you have one tip to help people improve their cloud security posture? And 2, do you have recommended reading? And it could be anything from Dune to really anything.

To your own blog.

Yeah.

To your own blog, of course.

Huh. I should have thought about a better answer for this one.

Actually, for the right, for the reading list, you have a really amazing book on this, right? Like the Dummies book. I've been reading it. I've been showing it to some people here who needed to be educated in a cloudy way. So I think that book is already on the list.

That's great, but it's not as good as Dune. I would recommend it. It talks a lot about cloud security remediation. It's called Cloud Security Remediation for Dummies. It's available on our website, Daz.io. It's available on a bunch of other places. One tip, I think the biggest tip that I have is just to think about how you meet people where they are. If you want to really drive an effective remediation process, you have to empathize with all of the people who are involved in that process. I've been talking to security analysts where I'm speaking to them first time and I'm like, I know you have this huge spreadsheet that you were working on micromanaging the project of remediating all of these vulnerabilities, and you probably hate it because you're passionate about cybersecurity.

Yeah.

And this is not why you're in this job. And I was talking to engineering leaders who are like, they just don't understand what I'm going through. They just don't speak the same language, right? So talk to your colleagues, ask them, why don't we meet our SLAs? How can we make it easier for you? How can we improve the process and start from there? This is the best tip I can give.

Well, Tamir, I can't tell you how much I love that answer. This whole conversation has been so much fun. Thank you so much for joining us. I have to run because I'm getting kicked out of my conference room here in the building. But thank you for joining us. That's been tremendously good. I'll see you guys.

Thanks for having me. I really appreciate it. Thank you.

And now we are at time. Thank you very much for listening and, of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity, Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @Anton_Chuvakin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Bye.

Security.

Security.