EP140 System Hardening at Google Scale: New Challenges, New Solutions

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us, and the rest of the Cloud Security Podcast listeners on LinkedIn. Anton, we are talking about what I think is one of the greatest investments teams can make today, which is preventing issues in the first place. We're talking about hardening, which is great.

Yes, but it's also, some people would say that this is a take from 2002, and now everybody needs a system that has AI and big data and scale and good UX and not hardening. And to me, this is a fight that would go on because I feel like you're right, yet this take has become unpopular over the years. And this is— what do we do with it?

Well, I mean, I've never been afraid of being unpopular. If I was afraid of that, I wouldn't get out of bed in the morning. I think what's interesting here maybe about our guest is he's been doing it for a very long time at Google scale. And his metric, his first metric he cited for whether he knows his team is doing a good job or not, I bet our listeners will not guess what it is. And so with that tease of something that doesn't come until 15 minutes into the episode, let's turn things over to today's guest. I'm delighted to introduce today's guest. Today we are joined by Andrew Hoang, Senior Security Engineering Manager at Google. Andrew, I'm really excited about talking about hardening today because I spend so much time doing threat detection. It's like I'm where hardening has fallen short, and I really think most orgs are probably better served by hardening than trying to catch the bad guys after they're already in. So maybe I want to start with a bit of a step backwards in time and ask, you know, when we think about hardening systems at scale today, hardening cloud systems at scale, what's different and what should people who are leading programs hardening clouds keep in mind that's different today than what they know for, say, the past 20 years?

Yeah, great. Thank you. So if we go back 20 years ago, the early 2000s, that was really the rise of the computer worms. You know, we saw first email worms. We had ILOVEYOU, Melissa virus. Then we saw sort of direct machine-to-machine worms, Code Red, Nimda, and then we were really hit by SQL Slammer and Blaster and so on. What these worms had in common is that they were definitely operationally disruptive. They were occasionally mildly destructive, but they were not anywhere near sort of the capacity or the abilities of the viruses we see today. But what they really did was raise the awareness that, hey, we need to invest in basic hardening or our systems are going to get taken down time and again. We have to have good perimeter controls and protection. We have to invest in vulnerability management and patching. We have to do isolation between different workloads so that we don't see lateral movement. So this was really the starting point of sort of the hardening industry. We invested in firewalls, we invested in intrusion detection systems. Patch management, as I said. And that was really good groundwork. It was effective against these types of sort of broad worms and the things that we were seeing at the time. But year over year, there's been a steady increase in the sophistication of the attacks we have to defend against. And there's been an increase in the impact of those attacks as the attackers have gotten deeper access to our systems and the data that is really important to our businesses and all of the people who depend on us. And so we have to take that into account and continue to modernize our approach to security. Today, the threat landscape is complex, and the role of the security defender is critical for businesses of all sizes. At the same time, the amount of technology choice we have is ever-expanding, and this is creating a number of new attack surfaces that we all have to understand and stay on top of. Cloud, of course, has brought a whole new dimension to this. In terms of our understanding of identity and perimeter and the key areas that those are integrated into our business. So one of the ways that I think we all need to stay ahead is we really need to hold our software vendors, our platform providers, and others, and across our technology supply chain to take a shared fate model with us where we're really working together to build systems that are securable but also secure, secure by default, secure in operation. And so that's sort of one of the key takeaways that I'd have is, is as a community of defenders, we need to work together to make our systems secure.

Mm-hmm. What you just said, shared fate, that's clearly super different, right? There wasn't even cloud 20 years ago, really. So how does maybe aside from the shared fate and the fact that there's this different relationship between, say, a cloud vendor versus, uh, I sold you some servers and now you put them in your own rack, has that changed the picture for hardening as well?

It starts from, let's make sure that we're not having products that come out of the box and have default ways that an attacker could get into them. Like, having hardened systems that we rely on, whether that is from a software vendor or from a cloud provider, is really key. The next is making sure that we are training our people on how to use the systems in a way that is secure. So when we have examples from vendors or examples from partners, that those examples take the security best practices into account and aren't asking people to do things that are short— that are a little easier but take shortcuts that leave them vulnerable. And then last is where we have a shared platform, investing in the security of that platform so it keeps all customers safe. And we don't have to— like, there's never going to be as many security engineers as the industry needs. We need to really scale out our approach to security.

I think the scale-out part is— I wanted to kind of drill into a little bit because I vaguely recall the time when kind of the previous era of people being obsessed about hardening, of course, when there was a question about, sure, you can give me a bunch of config advice and a bunch of things and I can apply it to a server. But once I have to apply them to 5,000 servers, suddenly a lot of things change. Nowadays, we are not talking about 5,000 servers. We may be talking about the millions, or if you talk about cloud instances, probably even larger numbers. So, what is the magic in scaling the hardening? Because ultimately, I still have this possibly misguided view that hardening is easy, scaling it is hard. Yeah.

Scaling requires a few things. It really requires you to be able to Have hardening capabilities that set guardrails, but don't dictate the fine-grained specifics to every single service that's launching. What I mean by that is being able to set allowable dimensions within which teams can operate that keep them safe from the specific threats that you're anticipating, but don't overfit those safeguards to a small number of use cases where you have to work with every single team and customize for every single team. So it's that balancing of risk management, threat containment, and threat reduction with the velocity of the teams that are trying to adopt and use all of these systems. The other way we work on this is paradoxically through somewhat limiting choice We give people more velocity because they can take and execute on a path, and they don't have to invent the entire security architecture for their system. They can work within a set of constrained, somewhat constrained choices and focus on the business outcome, the application-specific configuration needs, et cetera, that they need instead of having every single engineering team have to be a full expert on all cloud infrastructure security and everything else. That's key to scale and key to velocity.

So I think that's really interesting. Could you maybe say more about what that looks like in practice? And I guess I imagine teams must often come to you and say, oh, well, XYZ doesn't work for me for this reason. So how do you deal with that at scale? And maybe how do you do that without becoming the team that says no all the time? Because security teams seem to struggle with that everywhere.

Yeah.

So it gets back to finding the right balance between safeguards and flexibility. So let's take a, you know, maybe an example from Google Cloud. Specifically, one of the Google Cloud controls is for network protection. We'll go all the way back to network protection that we started at the top of the blog post or the podcast on. So for perimeter protection, you have 2 layers that you can do that in, in Google Cloud today. You can do it directly on the VPC network with ACLs. And they have also launched a feature that's called hierarchical firewalls that lets you set a set of constrained parameters that prevent certain types of exposure, certain types of network access from being allowed on any network that is under the configuration of that hierarchical firewall. So you have 2 levels of control. In that hierarchical firewall, you can say, I allow normally this type of access, I deny this type of access, but I don't talk about the fine-grained configuration of every single endpoint and every single service. And then you let the service owner manage that fine-grained sort of on-the-ground access control. And this really allows your security team, by allowing them to focus on sort of the top threats in the baseline controls and allow your service or engineering teams to continue to move fast and manage the things that are closest to them that they know the best.

Well, that makes sense. And that sounds a lot like shared fate and distributed responsibility, not just at the micro, like team to team, but also the macro cloud to customer. That's fun. That's like fractal examples of how security works.

Yeah, but it's also something that's hard to explain to less mature organizations. It's very logical and it's kind of quote unquote the right approach, but there are enough people who show up and say, hey, I went to the Center for Internet Security site and downloaded the GCP templates. Can I just apply them and be done with it? And then you have to go pull your hair first and then kind of go back to the gas cloud and dinosaurs and explain the whole history of hardening and why it's not a good idea. Like, how do you deal with people who say, oh, cloud, oh, hardening, yeah, that's easy. I'm going to go to CIS, get the templates, and call it a day?

Yeah. So, like, standard hardening templates, like those from CIS, are a useful baseline. They're peer-reviewed. They generally set effective secure defaults. And they may also cover threats that you don't have familiarity with or services that you're not as familiar with. So they can reduce gaps in your defenses. And they also have the property of being measurable, which does help you monitor whether you stay compliant over time. But that said, they are general templates. They don't take into account the context in which the system you're hardening is running. They don't take into account whether you have other layered controls that may better deal with the threats that they are designed for. Some of their configuration recommendations are not necessarily effective at reducing risk in your environment with your configuration, or they may be very complex and costly to implement compared to alternatives. And they're really just focused on that one technology stack. And they can create a false sense of security because they're not validated against sort of the threat model you have for your particular system that's under protection. You know, they can't cover every single threat because they are generic. And they're also updated relatively infrequently because they do have peer review. So threats can shift. System capabilities to deal with threats can change. So you might not be taking advantage of the newest capabilities that are most able and maybe better effective and cheaper to operate to manage the threats in your environment if you're only relying on those templates to set your baselines.

That makes sense. I think that it's always— even on-premise, it always struck me as like the system has multiple roles and just drop a template, call it a day. Sometimes adjusting the template takes more time than almost like coming up with your own carding ideas. Probably infrequently, but I guess I can see the generic nature being both helpful and occasionally not that helpful.

So, on the topic of generic advice and engineering teams, how do you help people get back to the right state when they've unhardened something?

Yeah. So, I'm going to use the term defect as sort of a shorthand for when we've set an expectation for a state that system should be in, and there's a difference between that and the state that the system is in. So, unhardening, if you will. But really, we consider that a defect.

Wait, can I stop you for a second? But actually, that's a useful mindset that I want our audience to be aware of because if there's risk-based hardening advice and somebody says, but for my application to run, for my business to function, I need to remove this setting, that's a defect.

That's right.

That's not them being right as being wrong. that their system is broken, there's a defect. This is hugely important.

Yes, thank you for reinforcing that. It is really an important mindset to have when approaching this particular problem. So, you know, as a principle, and again, it gets back to scale, we really try to design our systems so that engineers can fail safely. They can fail into safety, that there are guardrails that will prevent a defect from becoming a vulnerability. But that's not always possible. We can't always protect against accidental misconfiguration. So, we also need to continuously verify the state of the system against what those hardening expectations are. And when we see a defect, you know, we want to, where possible, identify and apply fixes with standard processes rather than sending out individual tickets for every single team to understand what the problem is, prioritize it, and address it. This gets back to the question of scale again. So by having a common infrastructure as code solution, for example, that is always used for configuring cloud services, we can propose fixes by the system that is doing the monitoring or the defenders, send those out as code changes, which the teams can review, approve, and then roll out via standard deployments versus every single team having to sort of figure out their own way of fixing a problem. We also need to, you know, understand that not every defect represents the same amount of risk. So we really want to fine-tune our response to minimize the productivity hit. You know, I think there's a key part of this question, which is about maintaining, you know, velocity within our engineering teams while managing our risk. So the first, of course, is urgency. Does this defect actually create an exploitable vulnerability? Can it be used to gain access to the system? Is that a vulnerability that's well-known or commonly exploited by people across the industry? That really tells us, is this really urgent or is it something that we can take a little time on? Second, of course, is the impact. Behind that, what are the systems or the data that could be affected if that defect was turned into a vulnerability, if that defect was exploited? Is there a potential for lateral movement post-exploit that could, you know, compromise further systems inside of our infrastructure? And then the last is span of control, by which I mean, can a central team manage and push a fix for this, or does this need to be rolled out by every single team that's affected by it? You know, we really try to leverage central teams as much as possible through ensuring they have appropriate role and responsibility, appropriate span of control, versus having to interrupt every team that needs to make a change.

So, it makes sense, but I think there's a lot of moving parts behind this judging the risk of a defect.

Absolutely. It can be really hard in the moment, especially if you don't have a great understanding of the systems, what role they serve inside of your infrastructure, what is the data that they're processing. And so, it's clear that you— the more information you have about your systems, about how they are connected to each other, the better you can quickly judge the severity, the potential severity of a defect, and make the right decision on how you respond.

You know, listeners, I never do this. I never interrupt a guest to say you should check out the product I work on, but I have to here because, Andrew, the thing you're describing, this scope of impact and the challenge of understanding it, is exactly what we just addressed in SCC, Security Command Center for Google Cloud, with our attack path simulation feature. And so what you're describing there—

that—

no, no, like, like Listeners, I've been doing this for 2 and a half years now and I've never done this, but it's so topical. I'm going to do it tonight.

He hasn't.

I haven't. I really haven't. But what we have now in SCC is the ability to, for any given defect in your environment, whether it's a misconfiguration, a software vulnerability, put that defect into a model of your environment and then simulate how an adversary might or might not be able to use that defect given the connectivity and the value of the resources in your environment. And so one of the things I love about cloud and being a cloud-based control plane-enabled vendor is I know everything in your cloud environment because I am your cloud environment. And so I can make those calculations more correctly, more frequently, more accurately than anybody else. And when I think about the opportunities that I have as a vendor to help users be secure, this is one of those times where I like, I'm so excited about what we're uniquely able to do that I can't help but interrupt and say, check out how we're solving this problem in SCC. So, 2.5 years, I've earned one of them. Anton, the next question is yours.

Yes. And yeah, thanks for— in the beginning, you started to sound like, wait a second, there's an ad from our sponsor. And then I realized, yeah.

It's actually exactly what we do.

So, I wanted to go and bring up a sad subject, namely that some of the hardening in CIS frameworks and other places has been inspired largely not by risks, but by— well, not by threats, rather, to correct myself, but by regulatory compliance arguments, other things of that sort. So I guess we do have that in the cloud as well. So how do we incorporate new controls that come from outside, from auditors, from the desire to either incorporate another mandate or other stuff that isn't directly mappable to risks?

Yeah, so I mean, at Google, one of our core values is respect the opportunity. And security exists as part of— in part to accelerate our ability to take advantage of new technology, but also to maintain and extend competitive advantage. And that can mean staying ahead of the regulatory landscape. So we apply the same principles when we're rolling out new controls as we do for other ways of managing threats. Look for ways to fail into safety, which we can talk about in more detail. Design for standard processes to make them easy to operate and manage over time. Leverage our span of control to minimize interruptions for individual engineering teams. But really, the biggest challenge when rolling out a new security control is understanding the impact of that control at scale across all of your infrastructure, across all your systems. Which of your systems are in scope for that control? How will it interact when it's turned on with those existing systems? Where are you already compliant with that regulation, and where do you have defects to address or need to apply a control that doesn't exist today? And so it's really important as you are managing your security environment to have the capabilities and processes that make answering those kinds of questions easy. And it starts with really— and I'll double down on it— that comprehensive inventory of the services you have, how they're deployed, what are their dependent assets, along with the business metadata about them. What is the purpose of that service? Who is responsible for maintaining it? What is the data it has access to, et cetera? All of that makes it easier to sort of stay ahead of regulations without becoming, you know, drowning in just constantly evaluating and deploying new controls. And, you know, in general, as a best practice, we always try to engineer our systems to adapt to change. We know change is inevitable. And I really have to say that Google Cloud creates opportunities to understand your services and design for this kind of dynamic environment that's a lot harder with an on-prem or legacy infrastructure.

Yes, it is. You know, one of the things I think is really interesting about, again, cloud and why I chose to work here is there are such powerful and unique opportunities to help people have not just a more secure state, but a more secure journey around adopting better state in the future. And so I want to shift gears a little bit with our time left and ask, you know, when you look at the kind of the history of cloud, you've been here a long time, you've been at Google a long time, Is there a case that comes to mind for you where we've taken a lesson from hardening and then improved a product as a result? Like, have we baked these improvements into the service in the past?

Absolutely. You know, an advantage of working at Google is our persistent and dogged focus on scaling security. It's the only thing that really matters here. And so I've worked for many years with product teams, and really it's about shifting responsibility for hardening usage. along a spectrum. So starting from ensuring that the product is securable, like customers can safely use it, and then making sure that those products have secure-by-default configuration, and then making sure that customers can manage guardrails in a way that allows teams to fail into safety. So let me talk about another example. We already talked about hierarchical firewalls. I did work closely with that team. Another great example, I think, is the new IAM deny capability. It is able to allow a central team to manage who can have certain types of IAM permissions across your infrastructure. So for example, if you really wanna restrict the ability to create cloud buckets, just as an example, to only your robot account, your service account that is part of your CI/CD pipeline. And even if someone were to be granted an IAM permission to create a bucket or delete a bucket, you wanted to be sure that they could not exercise that permission. IAM deny allows you to layer that kind of control. So you can say, hey, I know that all of this kind of change is happening from a system that I manage and has certain properties and has certain sort of guarantees that it's providing. And I don't have sort of on the backend sort of people managing or changing the environment outside of that. So a really powerful way of enabling not just security teams, but other teams to have good governance over cloud usage. Again, without slowing down the velocity of the teams that are actually, you know, working on the systems in cloud.

So, velocity came up a few times, and it's also something I've noticed when in the old, kind of 20-year-old approach to hardening, things assumed to take a while, you know, file forms, make requests. So, what are the metrics that you use today in the modern world to figure out how well you're doing with hardening? What metrics do you look at to understand how other teams are doing? Like, velocity presumably is one of them, but, like, I'd love more details because this comes up a lot in discussions. How do you measure this?

Yeah, I break down into, like, 3 big areas for my team. So the first would be broadly business enablement. How quickly are new services able to get up on and running on Google Cloud with appropriate security hardening based off of our understanding of their threat model and their business impact? So it's not just about velocity, but it's about safety and velocity.

Wait, wait, listeners. What Andrew just described, his first metric is not how much risk did I reduce? It's not how many viruses did I stop? It's how quickly did business get up and running? That for so many people out there is going to be like, brain exploded. And listeners, I guess you couldn't see it, but I made my brain explode with my hands. So that's amazing. What else do you track?

Yeah. So the other half of that is how frequently do we have to interrupt teams because they have a security issue? Where are we, you know, seeing consistent defects or vulnerabilities and we're having to interrupt teams to address them? That's another example where we may have an opportunity to take a different approach so that we're not having to do that interruption. We can get ahead of the security issue. Now, we do care about security risk as well, right? So that is an important metric. In that space, we think about what is the percentage of all of our cloud resources where we are able to monitor for vulnerabilities and defects? Do we have total visibility, or do we have gaps in our visibility? Within that, then, what is our mean time to discover new vulnerabilities and defects by the severity? And then, what is our mean time to resolve those vulnerabilities and defects, again, by severity? So it is really important to keep that risk focus as a defensive team. But our first metric is making sure that we are helping the business move fast while staying safe.

Oh, Andrew, I love that answer so much. And it pains me to do this, but I have to ask our traditional closing questions. First, do you have recommended reading for the audience? And 2, do you have one tip to help people improve their velocity of improving their hardening?

Yeah, absolutely. So on the reading side, I have to give a shout out to CloudSecList, cloudseclist.com. It's a curated newsletter by Marco Lancini.

I love Marco's newsletter.

Yeah. God, yes. This is incredible. Yes, we're all fans. I retweet him every time. And yeah.

Yeah. So—

How come it's the first time somebody brought this up? Let's briefly agonize about it.

How come we haven't had Marco on the show? Anyway, Andrew, that's That's a great reading tip. What else have you got?

So for velocity, you know, we've talked a lot about the technology, but it's equally important to stay ahead of the process side. Security should always lead with customer empathy. We need to understand that these teams that we're engaging with have tight timelines, they have competing priorities, they don't always understand the security issues. So the easier we can make it for them to maintain a secure service, the faster overall we can be at adapting to change and staying on top of the threats.

I love that answer. Andrew, thank you so much for joining us today.

Thank you very much for having me.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton underscore Chuvakin and underscore Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.