Automate and/or Die?

Security.

Hi there, welcome to the Cloud Security Podcast. Thanks for joining us today. Your hosts here are Tim Peacock, the product manager for threat detection here at Cloud, and Anton Chuvakin, a reformed analyst and member of Cloud Security team here. You can find this podcast wherever podcasts are distributed and at our website when we finally launch it. You can also follow us on twitter.com/cloudsecpodcast. Today we have a new guest and a new topic, and the topic would involve remediation and automating some of the tasks in the cloud. So our guest today is Joe. Hey Joe, welcome to the podcast. Could you introduce yourself and tell us a little about your previous role at Citi as well as your current job?

Sure. Thank you guys for having me. My previous role at Citi, I came in as the Vice President of Cloud Native Security Engineering focused on Google Cloud. They were doing a Google Cloud adoption, and my job was to use cloud native tools as much as possible to build security guardrails to protect their workloads that would be going into the cloud. After my boss left, I ascended to Senior Vice President as well as the head of CISO special projects. So I did a little bit of work still mainly focused on Google Cloud, but also with AWS. Now I work for one of the larger cybersecurity companies in the world, still focused on Google Cloud.

So when you were at Citi, you focused on Google Cloud. Is, is this like Conway's Law in action, and was there an upside to that? Was this good Conway's Law?

In some ways, yeah. I think that it was Conway's Law almost in a global way, because I think there's a lot of large enterprise thinking the same way about cloud and figuring things out, trying to close the security gaps, close the gaps in technology between traditional and cloud. And I think that cloud kind of just puts itself out there as one of those tools that allows us to build and customize You said you were focused on native tooling as much as possible.

What did that end up looking like? What did that mean for you?

There was really 3 areas of security guardrails that we built. So there was the preventative, obviously starting with your pipeline and moving forward. Then there's the alerting, whatever SIEM tool you're using and threat hunting tool you're using, integrating into that. And then there was the remedial aspect of it, which we used Cloud Functions. Serverless was the It's the greatest invention to hit cloud as far as security and being able to customize your tools.

And by remedial, I don't imagine you mean like the kind of classes I had to take as a child. I imagine you're actually talking about the topic of our podcast today, which is threat remediation or vulnerability remediation. So, we see a lot of people talking about threats versus vulnerabilities in the cloud, and even within my product group, we sometimes mix them up. Do you think that a crisp definition was important to your role at Citi, or did you guys treat those as the same thing that you had to respond to automatically?

I think they were very much the same, but I think cloud certainly blurs the lines a little bit between the two. The CRISP definition still wins out in my opinion, because there's still some level of human interaction or human-made software that has to play a role. But I think when you think of cloud and you get into the idea that every resource is now an identity, so your VM is accessing cloud storage as the service account that it's attached to. So it definitely blurs the lines and you have to treat it very differently and make sure that you're following best practices. And I think, again, cloud is one of those areas where the lines are blurred, but it's still traditional enough. Your dyed-in-the-wool technician is going to understand it.

Yeah, but here's the one that usually tweaks me because I am that person who would fight to the death over the definitions and over people not confusing threats and vulnerabilities because if you forgot to patch and you have a weakness that somebody can exploit versus somebody just hacked you, picked a password, did something. To me, the chasm is kind of huge, and I am really quite allergic to people who confuse them. However, when we started looking at this specific to the cloud, I kind of noticed that some of the workflows and some of the thinking and some of the processes are in fact similar. So I had to reset my brain a little bit. Like before that, I'll be like reaching for my baseball bat, to tell people, why are you confusing buffer overflow with an attack? Like, these are from different domains. But I see that when people remediate an unpatched system versus they remediate a system where they have evidence of, say, password compromise, they sort of flow through similar steps. And I started questioning my kind of religious fervor in this regard. Is that what's going on or something? There's more nuance here.

Yeah, no, I think you've hit the nail on the head. I think that the threats have changed just taking into consideration what we use to build these tools. So serverless, right? It's a whole new technology that a lot of people don't understand. So you have these functions and you put the code out there one day, but if you're not monitoring that code, if you don't know that it's been updated or changed and you don't have control over those IAM permissions, that becomes a threat and a vulnerability almost at the same time, right? Because someone can add a line of code, start data exfiltration, and you have a serious problem on your hands. So I think you just have to be aware that identity is totally different in the cloud, and your identity and access management is where all of your security has to start.

So then a weak password is still a vulnerability, but like you're one step away from it being exploited in a very bad way, kind of like that's roughly what may be going on. Because to me, I like to highlight the fact that identity is the main boundary in the cloud. You may not have layers of firewall. And that's going to be the topic of our future podcast, I'm sure, where we're going to kind of wax poetic over how identity is really your main security barrier in the cloud. It's not physical. It's not buckets of firewalls. Apologies to firewall vendors. But it's mostly the identity really does separate the good from the bad.

And apologies to the poor firewalls that were being kept in buckets. That's awful.

Well, there is that too.

Yes. Let them loose. Let them go live their lives. They've served us well.

So we touched a little bit on this. I sort of mentioned processes and you guys mentioned automation. So the real elephant in the room is, can we automate remediation of vulnerabilities and can we automate dealing with some of the threats in the cloud? So how do we go about that? So this is kind of my, I guess, introduction to the main feature. Can we hope for automation or not?

I think so. I would say that in the cloud, if you are not open to automation, you're probably in the wrong industry because It's given us the agility that we've always been promised that technology was going to give us. So your products are changing every day, your APIs are changing, the way that they are integrated is changing, and there's a lot of back and forth there. And I think that one of the things that has to be stressed here is from a remedial standpoint, you have to really test things. You have to make sure before anything hits production that it's doing only what you want it to do. So there's a lot of back and forth between the logs and, oops, I didn't want to destroy that, but I need to— layer this a little bit better. But yeah, I think there's very few things in the cloud that I would say you can't automate if you have competent engineers on your staff. And that's the hurdle to clear, right? I think there's a limited number of security engineers and certainly an even more limited number of cloud security engineers.

We certainly talk about the challenge of hiring. And one of the things we talk a lot about on my team is ironically, when it comes to guardrails, How do we, in our system for enforcing guardrails, put up the right guardrails to help security teams scale their human operations with less effort? So one of the things I run into a lot as a challenge though, is I talk to users about automated remediation. And to me, it feels like there's a really big gap between where we are with automated response and where we could be with automated remediation. Let me give you an example to tease that difference apart. If I have a basement and water gets into it— see, I grew up on the East Coast. This is the thing I worry about. Don't worry about it in California. But if I have a basement and water gets into it, it's really easy to have a sump pump that automatically has a float that kicks on and pumps the water out. Now that's automated response, but that didn't deal with the now moldy drywall or the crinkly pictures of grandma. To deal with that, I call up ServPro, they show up with a fancy fan and they fix my house. That's remediation. And so when you were building these things previously as the elevated senior VP, were you building response or were you building remediation? And did you have a path from one to the other?

I think that response and remediation kind of became amalgamated in some ways because we were really solving for the fact that we had a lack of internal knowledge amongst ops teams, amongst SOC. I mean, we were having to solve problems in engineering that ordinarily would've been the problem of some other team, right? So when we were responding to things, it was important that we did kick on the sump pump, but also get all the artifacts that were necessary to build out reports so that we could respond properly. You know, you don't want to just kill someone's resource and then not have a conversation about why it was or when it was. So I think that automation end-to-end is really the key and making sure that you are keeping the artifacts, keeping the documentation. It's just become so natural for us to automate everything at this point.

So you didn't have a human in the loop most of the time. It really was truly automatic.

Truly automatic. As far as I'm concerned, you can get away with having a human in the loop and still call it automation, but end-to-end automation is really the key. The fewer hands that touch things, the less mistakes you make, the less likely you are to have human engineering and influence. So it just makes for a much more secure environment.

But then you have to trust automation. You have to be sure that you remediated the issue, right? Like, as Tim pointed out, that this whole distinction and difference between remediation and response, Automation can act, but like, how do we know that the problem is solved at the end? Wouldn't humans have to check? I'm trying to be a bit devil's advocate, but like, how do we get to trust that the problem is truly solved by the machinery with no humans?

There is a kind of loop that you build with automation where you do the remediation and then you output the logs to report that the remediation was carried out. And then you have another loop that goes back and checks your automation to make sure that it is doing what it's supposed to do. And you really should. I mean, One part of the human element that we really can't get rid of is the auditing aspect of things. You need those auditors to come in and check your code, but you want that checks and balances system. It's important. Automation will take us far, but it will only take us so far.

That's really interesting that automation ultimately rests in an organizational context of, we think we can solve this problem. Did we actually solve it?

One of the interesting things is as a security engineer, you just have to make sure that Especially in leadership, you have these metrics that you have to be able to provide. So reporting up the chain, they may not care, but they want to know that you're doing something. They want to make sure that the tools you're building are doing what you say they're doing. So you have to perform with some sort of metric and take them reports about, you know, here's what we stopped this last quarter, here's what we know happened. Do they care? Probably not. They probably care about money and PR, right? They don't want to end up on the nightly news, but you give them the confidence through your tooling that they're not going to end up on the nightly news.

That's one of those cases where metrics are interesting for both managing up as well as managing down. What were you tracking with your team when it came to this? Were you tracking how many use cases we had, how often they fired? How did you manage this?

I was tracking how often they fired, how often the alerts that we built— because we layered things. We always assumed that one layer of that was going to fail. So if you built a preventative measure, plan for the fact that someone could local exec anything in their Terraform, right, and make a change. So So be ready for that with alerts. And then when the alert fails, you go to remedial. And so we tracked how many alerts we had versus how many actual remedial events we had. We tracked the type of infraction that we were correcting. So if someone built a VM workload or a container workload in a region that wasn't allowed, we would track that so that we knew the kind of threats that we were dealing with. But more often than not, we were just Providing insights into, here's the CIS benchmark, here's what we stopped, we are in compliance at all times. That was really what they wanted to see.

Some of this sounds like it's a policy issue rather than even a weakness, vulnerability, or a threat. Like if I launch a VM in the wrong region, eventually it may end up with privacy issues, whatever. But ultimately it's neither a vulnerability nor a threat, right? It's a policy issue. Am I being too subtle or that's the story?

I think you're 100% right. And making sure that everybody in a massive organization understands what the policy is, that's a challenge. So a lot of it was just learning on the fly in an organization and, and making sure that people had the logs output to understand why their machines suddenly disappeared, right? Why their storage buckets suddenly disappeared. That was a fun time.

Did you get particularly memorable emails from people whose instances suddenly disappeared?

Oh, absolutely. Absolutely. And, and there's always the You know, where something goes a little farther than it should have and gets rid of stuff that they needed. You know, testing is important, making sure that you're giving people the information so that they can build the right resources.

How about we drag the elephant in the room into the site and say some of the stuff you're describing kind of points out that your team and teams around you did not have this notorious security fear of automation. Like you mentioned, some things may have gone too far or whatever. But like, we always hear stories from the, you know, '90s, 2000s when somebody blocked somebody else's laptop and it was a CEO during the presentation. So the whole project was shut down. Like the whole domain of security was terminated. We hear those stories, but apparently in the cloud, things aren't that bad or maybe they even aren't that way. So what about the cases where things went wrong? How come automation survived those occurrences? Like what's different in the cloud?

I think that one of the biggest things is that recovery can be so rapid in the cloud, where before there was such a massive process to correct any of the issues that you created. At the worst you run into in cloud is that you're going to have to redeploy an entire new project. And what is that going to take you from the perspective of infrastructure as code? So I think there's less fear because of that. One of the things that people say about the cloud is that the landscape is huge and it's, it's getting harder and harder to see everything. But at the end of the day, you're actually able to visualize your isolation, your VPC project boundaries so much easier in the cloud than you are in everyday life. So you can target specific instances, you can take things to a resource level for access and make sure that they are meeting the requirements needed for special workgroups. The upside is that you get more benefit than you get negative.

Sounds like you also can automation to fix automation mistakes, right? I've heard that. Maybe I was reading between the lines.

Absolutely. You're 100% right. So you have to build a loop. And one of those loops is actually making sure that the functions that you're using to secure your account are always available. So if something gets deleted, having an automated redeployment of that security function in the backend is really important.

I like the thinking with loops. One of my favorite, like, thinkers is John Boyd. who came up with the OODA loop. So I love where you're going with this. One question I have gets back to people though. It sounds like you had good people, and I actually know you had good people because I met some of your team. What advice would you have for somebody who wanted to get started with this when they had, you know, fewer people or fewer people that they trust as much as you trusted your team?

Check and double-check, obviously. But I think investing in the training and the knowledge is something that a lot of organizations are probably avoiding right now. I think the obvious path is to speed to production, and really what you lose with that is being able to up a team that understands these services in depth. So I think training is really important. The other thing is give people a test environment and let them go. Because the other thing that's happening within cloud is we are so scared of minute mistake leading to a problem that we don't want to face, when really we should just isolate some boundaries and let people get into the cloud and go, because that's the best way to learn. That's how I trained my team, just gave them access and said, start playing with it. And before long, they were probably more competent than I was at the end of the day.

That's really interesting that the best way to learn is just by doing. I like that.

Absolutely.

Yeah. Well, hey, we are just at time. So Joe, thank you so much for joining us. Everybody who's listened through to this point in the podcast, thank you so much for joining us. Again, you can find this podcast on Google Podcasts and wherever else you get your podcasts. You can follow us on Twitter at twitter.com/TheCloudSecPodcast. You can find Anton and myself on Twitter. Tweet at us, email us, argue with us. If we like or really hate what we hear, we might invite you on the podcast, and that's either a blessing or a curse depending on, you know, how much we agree, disagree, and how brave you are. So see you all next time on the Cloud Security Podcast. Security.