EP43 Automation as Paved Roads in Cloud Enablement

Security. Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Tim Peacock, the product manager for threat detection here at Cloud, and Anton Chuvakin, a reformed analyst and member of the cloud security team here at Google. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Finally, addition to the opener today, guests, while it pains me to say this, Anton and I are going to take a brief break for the holidays in December, but we will return with new episodes in January. So if you're missing our content for a few weeks in December, fear not, we have not been hit by asteroids. We are just having a brief vacation. And so with that, Anton, our episode today, kind of a back to our roots episode and a topic near and dear to my heart. We are talking about both automating response to security events in cloud, as well as talking about some of the cool stuff that you get with Security Command Center.

That's true. But also to me, the exciting part is not just that, but the fact that we are talking to a really well-put-together operation. We are talking to a team that has a very robust operation in this area. Like, we are not just giving you concepts and ideas. We have people who actually are doing it and are doing it in a very robust manner. So to me, this is where the magic happens.

I agree. There are some things that the team says in here that really say we're doing it right. And so with that, listeners, let's turn it over to our guests today. I am delighted to welcome today's guests. We've got quite a panel today. We've got Amber Shafi, a product manager, just joined the dark side at GlaxoSmithKline. Svetlin Zamfirou, a senior platform engineer, and Ivan Anjelov, a principal platform engineer. So we've got a great crew today, and we're going to be talking about automated response, which is a perennial favorite topic of listeners. I'll start off real easy. Can you tell us about your team, what y'all are responsible for, and what's the team set up to make that happen? What part of cloud security do you cover?

A bit of a high level on the team. We're a cloud enablement team which accelerated the adoption of GCP within GlaxoSmithKline. We created a secure and compliant by default platform on top of Google Cloud, and we aim to allow product teams to experiment and iterate fast on GCP in a safe environment. With our company being in the healthcare sector, it means that we're always working in a highly regulated environment, hence our heavy focus on security and compliance. To put it in more detail, we try to make it really easy for engineering teams to develop products on top of GCP and leverage fully automated CI/CD processes, reusable infrastructure, having security and compliance guardrails baked in all of that. So, the teams can operate on their own, which I personally think is the cherry on top. At the same time, we are constantly collaborating with security and compliance teams within the company and providing them with telemetry and alerting for the platform. And we're essentially trying to push towards having a single pane of glass experience as much as possible.

Got it. That makes a ton of sense. So you're there to help make cloud adoption go faster.

Yeah.

To pick up on that theme, I wanted to sort of start exploring one particular angle of this. Of course, when we had security on-premise, we had vulnerability scanners, we had configuration scans. So we looked for configuration weaknesses. It's not something new. What we observed, I mean, what everybody observed is that there's a lot more focus on the cloud misconfigurations. There's a lot more almost obsession with not having misconfigurations. So could you give us a bit of a maybe slightly high-level view about what are cloud misconfigurations, what's included in that set, and why they seem to be so critical for security in the cloud compared to on-premise?

Yeah, absolutely. Cloud functions misconfigurations could be defined as errors made or gaps left while configuring a cloud environment. And obviously, these have the potential to expose cloud environments to various security risks, such as data breaches and cyberattacks. I think that where they Differ from on-prem misconfigurations is that the first difference is that on-premises there is usually this external hard shell, which is what network perimeter acts on. What this does is that it is a bit more forgiving to misconfigurations, which are otherwise constantly happening in on-prem environments as well. In contrast, in the cloud you have these public APIs, which are accessible to. Everyone, of course. And there is a shift from this network security-based control model that is the perimeter in favor of an identity and access management-based control. The second difference is that in the cloud, you have the shared responsibility model. And according to this model, cloud service providers and their customers have different and sometimes overlapping roles in securing data and services in the cloud. I think that at this point, it's become quite evident that cloud security providers are very good at upholding their part of the bargain, but their customers still have a sizable task in making sure that they're configuring their environments properly, including promptly addressing any misconfigurations, of course.

By the way, to kind of pull on the theme, you mentioned identity-based things, and that's a theme, of course, that comes up in many podcasts so far. Do you like or do you not like the line, identity is the new perimeter? I've met people who are kind of passionately pro and passionately against that line. So is identity a new perimeter or not?

This is actually the first time that I'm hearing about this, but to be honest, it makes sense to me. And that's a much more helpful way to reason about security in the public cloud. I think that we're not really doing ourselves any favors by simply trying to do what we have become accustomed to doing on-prem and shift it to the cloud. In fact, I think the opposite is where you can bring more value by leveraging practices which are made possible by the cloud and bringing them on-premises as well.

I love that answer. And Anton, you don't have to be shy. It's me who says that line.

I'm pretty sure one of my former colleagues at Gartner made it up. So sorry, team.

Really? Oh, I got beaten by Gartner. Well, yeah, I'm happy with that. That's fine. So on the topic of, you know, these misconfigurations and these issues, and maybe we'll hint at discovering threats as well. How are you all discovering these problems? Like, what's bringing these things to your attention?

Yeah, absolutely. So we've built custom solutions for different use cases. We're talking about IAM, so let's talk about identity and access management. One of the ways you can have a security breach is if somebody misconfigures access and they mistakenly grant public access to, let's say, a storage bucket or a dataset. So this could happen if they misunderstand the group all authenticated users, which is an identifier on GCP for any user or service account. So essentially your resource is now open to anyone that can access GCP. you may be exposing sensitive data. So thankfully, GCP provides us audit logs, which help you answer who did what, where, and when. So we're tapping into that to tackle this particular scenario. What we do is we're filtering and exporting all access-related audit logs to a Pub/Sub topic using a log sync. This Pub/Sub topic is then triggering an event-based Cloud Function every time a permission change is made. So in this Cloud Function, we've written a logic that reviews the access against certain predefined rules. So a rule could be that we don't allow access to the all Authenticated Users group. And if it sees a misconfigured access grant, it automatically removes that access grant. That's it, really. So, within seconds, you've automatically remediated a misconfigured access. And yeah, job done. Obviously, we don't stop there. Once the access is updated, another Cloud Lock event is published, and we use that in order to publish a message to Pub/Sub. This message then triggers a Slack or a Teams notification to the resource owners, the resource where this misconfiguration happened. And we know who the resource owner is because we're putting labels on all of our resources. Our team also finds out about the information included is about the access grant, who did it, what time, what was the remediation action, and also if by any chance there was an error, whether the remediation was successful or not. So, this really helps us autoscale based on the volume of log events. A person doesn't need to go in and interact with the solution. It just happens. All the while, we're being notified of the remediations that are taking place. There's a more detailed explanation we've put in a blog that we can link to if it helps. And we've got similar solutions in place for removing users that are outside-of-the-company domains, as well as individual users, because we prefer giving access only on group level. It just helps us manage users at scale.

That makes sense to me. And I think one theme I want to pick from that, as well, because I think the patterns of automating responses to misconfigurations are kind of starting to emerge, at least from some of the conversations. And one thing that we do encounter is that there's still somewhat of a resistance, at least in some circles in security, to automated responses. So beyond the obvious upsides of reducing toil on the team and time to respond, What are other benefits that you observed? But of course, I kind of have to bring up the risks. Are there risks to that, and how are those handled? That to me is sort of a fun, balanced question about good sides, bad sides for automating.

Absolutely. And I think you've already mentioned some of the great benefits that we get. So essentially, less things for you to worry about. You're basically handling any trivial misconfigurations in an automated way. I mentioned scale earlier. So that's, again, a really big benefit. You're also minimizing the time potentially sensitive data could be exposed. So, huge benefit there as well. Another big benefit is that if a misconfiguration occurs, forget the remediation, we actually are giving visibility to the misconfiguration itself. Oftentimes, companies have cloud misconfigurations that they don't even know about. They have long exposure times, and they only find out when things go really bad. You asked about risks. So, one of the risks I can think of is remediate any necessary changes, which could lead to an unintended effect. For example, we might have downtime or an outage. This could be a case, for example, if you have buckets that are using CDNs, so they do require public access, but the solution takes away that access. A mitigation for that is to allowlist certain resources, certain projects, so that a solution like that doesn't come in their way. Another one I can think of is the Cloud Function failing. In order to mitigate that, we've mentioned the alerts. So, we're receiving alerts whether remediation was successful or not, but across the board, we're monitoring our Cloud Function executions, and we're getting alerts coming into Slack. And if anything goes wrong, we're immediately alerted to that. Finally, I think as with any code, if there's bugs in the implementation, that can lead to unexpected behavior. How we tackle that is by sandboxing the implementation before we deploy it to any other environments. We've also got unit tests built in. So, if you're making any changes to the code, it's gonna go through those unit tests before it even sees any environments itself. So overall, yeah, risks with misconfigurations can be mitigated if you're Using automation, if your environments are looking similar. Yeah.

That makes a lot of sense. And you're not the only user I've talked to lately that is using Slack as their alerting channel back to the team when something goes wrong. It's funny how universal that has become. So tell me about the origins of this idea. Is this like, you guys were tired of fixing stuff by hand and you were like, damn, I'm not burning the midnight oil anymore. I'm going to automate this. How did this come to be?

Well, to be honest, we have always had it in our mindset. We are a team of lazy people, and I'm saying this is a good thing. I think there were multiple reasons why we decided that this was actually a natural next step for us. One of them was that we already very much had a defense-in-depth approach already, and we actually saw our role as using security as an enabler by allowing teams to both move quickly, but also responsibly. One important way to accomplish this was by building as much confidence as we can that we're capable of preventing some of the worst possible outcomes, these ones that keep you up at night, we saw automation and improved visibility as some of the best weapons in that fight. When it comes to automation, in our environment, engineers practically do not do manual changes in all but dedicated sandbox environments. And we are instead mostly relying on secure-by-default reference implementations, infrastructure code tailored for various use cases. Getting to a point at which you now have fairly uniform environments allows you to quickly deliver changes and means that you're in a position to spend a bit more time focusing on what the biggest potential risks are. Additionally, we had developed pretty comprehensive visibility over the state of all environments by leveraging automation, custom and open source tooling, and of course, various capabilities offered by GCP itself. So from that standpoint, we had a really good understanding of the scenarios that were covered and what we were seeing in terms of detected issues in the environment. As far as lessons go, since you asked about lessons as well, we actually learned that the non-functional aspects of the system were really important to its success. The 2 I would like to call out are reliability and providing good visibility. Being a potential last line of defense from a preventative standpoint, it's really important that the automated security response solutions must be reliable. And that's one of the things that we put emphasis on by testing and monitoring their execution. For us, visibility was very important, not just for the successful mitigation of potential threats, but for the wider organization's success by creating a feedback loop that we could all learn from, including the teams building products in the cloud. And lastly, I think it's worth mentioning that one of the things that actually helped us be able to focus on our goals instead of just implementation is the capabilities that GCP provides at our disposal.

I want to ask one thing about that, which is you were automating changes to things that other teams did. Did you then reach out and let them know, hey, you jacked this up and I had to smack it? Like, did you let people know when you changed their things?

So that's where the labels on the resources come in really handy because automatically the notifications can also go to the team. So we had a team that actually wasn't using Slack. We were mentioning Slack, they were using Teams. So they would get these alerts into their Teams channel if they want. You can send it to email as well. So before we grant somebody a resource, we make sure there's labels against each one of them and we've got the appropriate notification channels. So don't have to do it manually. Actually, they just, they just know about it. Of course, there's a caveat here, which we were talking about earlier with Ivan, was that if you're using automation and they've put something in the code and yes, this mitigation has happened, maybe there is a conversation to be had around updating the original code. So yeah, potentially, but they get alerted immediately.

Did you ever get into cases where their infrastructure as code robot would fight your Like, did you have automated systems going back and forth?

Not really due to the way the build pipelines were implemented. They were actually pretty much identical across the entire organization. So from that standpoint, we knew what sort of behaviors they would exhibit. Now, teams were also able to kind of opt out of that. But to be honest, when you make the right thing the easy thing to do, people don't really stray away from that. That's what we have observed in practice as well.

See, listeners, this is like an enlightened security team. They're a security team that's making other people's lives easier and giving them the easy paved path out of the mud to do what they need to do.

And the theme of making the secure part easy, so many people have said it and so few people live it. It's not even funny. Like all sorts of, you know, talkative CIOs have said, oh yeah, we're just going to make the secure thing the default or the easy one. And then you look at their environments and the same people shoveling manure with like manual shovels. Not anything automated and graceful. So to me, the fact that you guys live it is amazing. That's the fun part. So I wanted to switch gears a tiny bit and toss out one question that I've encountered during my Gartner days quite a bit in similar circumstances. And people kind of debate whether to primarily rely on cloud provider tooling, whether to purchase separate tools from startups that focus on the cloud, or maybe try to bring up legacy tools to the cloud. And of course, I know you're not in the legacy Bring out old tools to the cloud bucket. But how do you balance the cloud provider tooling? Well, our tooling, how did you integrate it? And do you actually need to rely on any third-party security tech for that setup?

In terms of dealing with misconfiguration, we are currently relying on logs from Cloud Logging as our main source of information. The reason for that is actually quite simple. We just created the implementation at a time during which Security Health Analytics scans were running once a day or so. Which wasn't quick enough for us, given our desire to respond to security-related events in real time, or at least near real time. But nowadays, CI/CD has the ability to make all sorts of findings available through Pub/Sub, making it really an attractive option for teams starting with automated security response today. I would personally encourage everyone to have a look at that. But in general, we have a great opportunity to observe CI/CD maturing and expanding since it was first introduced. It plays a huge role in our cloud environments. For example, if we go beyond misconfiguration and actually look into security events more broadly, we already have an automated solution which blocks outbound traffic to malicious IP addresses at GCP org level. And in this case, we are consuming events from the Event Red Detection service, again, using Pub/Sub for that part of the implementation. So it's merely the start for us to leverage CSCC as a source of information. And basically, this is just an example. Of course, but it'll definitely give some inspiration to the listeners for what's capable and what's possible out there.

Listeners, to be clear, these people have been using our product for so long, they still prefix it with the C. If you want to go search it up, it's now just Security Command Center. We got rid of the cloud prefix. We figure people know what we're focused on, but that's so cool that you're consuming our stuff automatically. Which alerts are you finding valuable? Like what's being useful out of there?

A couple of people can already speak to this. I would start by saying that there are plenty of alerts coming in. So we're receiving notifications, as Amber mentioned, for a lot of these events. Some of them are not applicable to our needs. We have to be honest about this. So we're trying to filter these out as much as possible. But for the majority of the findings, we find them useful. This is basically our source of truth. It's kind of like, in case you receive an alert, then something really serious has happened. So we're always having a look. I'll let the other folks also chip in in case they have some concrete examples.

I personally find the malicious activity ones quite interesting because it just sets you up into this detective mode. So yeah, I think those ones come in really, really handy for my sake.

These are something that, speaking of cloud-native versus non-cloud-native tools, I think that these were one of the biggest value adds, and especially the integration points that SCC As a quick unplanned question, do you triage them straight in SCC or do you triage them somewhere else?

Like, where are the alerts being confirmed by humans or by machines for that matter?

We use Slack as our main communications tool, and this is where investigations start from. Then people would typically go to SCC and acknowledge the findings there. But we have a very light-touch process for dealing with those, and we're trying to keep it as simple and as cloud-native as possible.

No, that makes a ton of sense. So we're just about at time and I want to ask our traditional closing questions. First, do you have one weird tip to improve people's cloud security automation? You know, what would be the one piece of advice you would give? And then second, what's recommended reading beyond your awesome blog, which we will of course link to for the listeners in the show details?

When it comes to tips, I would say the number one tip is actually talk to people since everyone wants this secure by default world where everything is secure without having to do anything. One thing that I frequently saw in the real world is that there is a gap between what security teams are doing and what products teams are doing. And I think that the best way to address is for security teams to develop a bit more understanding around what the product development lifecycle is looking like, the challenges that teams are struggling with, and so on. And likewise, I think that it will help product teams build some empathy with challenges that the security organization in general is dealing with.

My advice would be use automation. A lot of the times these misconfigurations are down to human error. So the more automation you use, particularly infrastructure as code, you might have peer reviews as things go into the process. So definitely rely on automation as much as possible, get upskilled on infrastructure as code and utilize that. In terms of reading, go to the Cloud Blog, you know, the Google Cloud Blog. That's where I end up a lot of the times, just trying to learn more about the services that are there.

Yeah, second that. I think I would recommend Google Blog as well. I found a lot of useful information there. And in general, it's a great resource to learn from.

My quick tip would be to just try to provide visibility as much as possible. It's one of the core things, not a lot of people are thinking about it. But the majority of times where incidents are overlooked, or, you know, security-related issues are overlooked, it was just basically lack of visibility. So having visibility not only for a particular team, but also a bit wider visibility, also including security teams, compliance teams, or whoever is responsible for incidents is really important in my opinion.

Svetlana, Amber, Ivan, this was an incredibly wonderful conversation today. Thank you so much for taking the time. I know it's late where you are. It's early where Anton is. Listeners, this is another transatlantic episode. We've had transpacific, now we've got transatlantic. Team, thank you so much for joining us today.

It was a pleasure.

Thank you.

Thank you.

Thank you very much.

And now, your time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcasts. And of course, your hosts are also on Twitter. @anton_chuvakin and _timpeacom. Tweet at us, email us, argue with us, and if you like or hate what you hear, we can invite you to the next episode. Please remember, we are going on a break at the end of December, but don't worry, we will return in January and we will be even better, probably louder too. See you on the next Cloud Security Podcast episode. Bye.

Security. Security.