EP200 Zero Touch Prod, Security Rings, and Foundational Services: How Google Does Workload Security

Secure running. Hi there.

Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior PM for Risk Analytics and Cloud Threat in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of the Cloud Security Podcast listeners on our Google Cloud Security Community page. Without any further ado, let's turn things over to today's guest.

Hello, and today we have Michael Chapinsky, security and reliability enthusiast at Google. Welcome to the show, Michael.

Thank you for having me. It's a pleasure.

So, this episode is inspired by a really fun, if you're into that sort of thing, paper that you wrote, and you and the team wrote, on how Google secures production services and servers, infrastructure security. So, who should read the paper? I mean, let me ask you a strange question that's off the list. So, who is this paper for?

I think it's good for any leader, like technical leader for a product, also for the infrastructure, to think of possible risks to their services and potential techniques on how to prevent them.

So basically anybody who cares deeply about technical security. So that's a pretty broad audience, right?

Absolutely. Like you don't want to be hacked after all.

Yeah, that's, yeah. From reading the paper to not being hacked is a bit of a journey. Multi-year journey, in fact, which I guess we'll get into a bit later in the So when we talk about our infrastructure, apart from its immense size, we also cover security, reliability, development speed, and also maintainability. To a somewhat naive person in these matters, at least me, like me, it sounds like some of these are kind of conflicting maybe a little bit. So how do you prioritize these competing demands in a real-world setting of, well, our network?

Oh, absolutely. Honestly, these days, this is pretty much my daily bread. Like I interact with, with many teams. that come with those exact questions. And you are absolutely right that there is a tension between all of those. Funnily, a few years ago when I was writing like this first multi-year strategy, I semi-jokingly put in conjoined triangles of success, which is a reference to Silicon Valley, and had 4 edges of the square there. One was reliability, one was security, One was cost/efficiency, and then velocity/maintainability, which roughly maps to the terms that you've used. And if you remember that series, there is a big fat compromise in the middle. So, as an organization, as a product owner, you likely don't want to be in any of the corners, but you want to pull yourself somewhere in the middle of that compromise. This is difficult. It takes negotiations because like different parties—

With humans, not machine to machine. I'm not talking about— Absolutely.

Absolutely. And those negotiations can be harder. I don't know, like humans have different incentives. They have different backgrounds, different experience. They don't always understand the other side of the coin, right? Like people who may be amazing in like designing, developing new features may have zero experience in security. risks or vulnerabilities that they are introducing. My background, as you introduced me, is security and reliability enthusiast, which expands to SRE, which is, you know, Site Reliability Engineering at Google. I've been in this role for 10 years. So, naturally, security— well, naturally, reliability is close to my heart, but security became a thing like 7 years, pretty much. But then I work with the devs every day, and they push for, like, we need to deliver this feature too, and there is a time-to-market element, right? How can we reconcile? We need to hit this, otherwise we have a missed opportunity. After all, we are here to do business, to provide features to our customers because the most secure system is the one that doesn't exist, but it's not very useful.

But if you balance— so this sounds like, I guess, philosophically an obvious answer. You have 5 things that are all very important and you aim at the middle of all of them intersecting and you have a winner and you celebrate, drink champagne, whatever. But do you also not sacrifice things? And I think that I don't want to prod fun at people who say, we take security very seriously, but, like, we kind of take security very seriously, and we mean it when we say it. So, does it mean that we would sometimes balance between security and, say, maintainability, to pick a random one that I know the least about?

We certainly do. What I do when I'm facing those trade-offs, I try to seek, actually, synergy and places where you can actually hit multiple of those targets with one solution. And I could name easily multiple examples where security and reliability goals are accelerating thanks to some solution. Okay, then you will sacrifice on the cost because someone needs to engineer, like someone needs to design and implement that. We have another one where security improvements are actually unblocking tremendous efficiency savings thanks to stronger isolation.

Yeah.

Which may sound counterintuitive. When you say, do we sacrifice sometimes? We do, but there is always like a hard line where it's like, this is a level of risk that we are not taking, whether it's on security front or reliability. There is a line that is held pretty high at Google, and we defend it. This is my role, partially.

Yes. And I think that's actually a very useful reminder, sort of to rephrase some of this, is that you would have tolerances on some things, and no matter where you are in the overlapping triangle, some things are in that sense absolute. You have the sort of SLOs for some things that you, no matter of moving triangles around, should bend. Just with security and with maintainability, with reliability, or with dev speed, certain things are, you know, hard constraints. And we're not going to be like, oh yeah, to have these 2 people happy, we're going to make this one person happy. That's not happening.

Yes. And I will add one more, like in some cases, we would, for example, accept like a temporary suboptimal security posture to accelerate velocity, but there is clear risk acceptance where a VP or a senior director signs off and says, like, business needs us to deliver this by then and then, and here is the plan of getting back to the happy place, which we will prioritize in the next 3 months, for example. So, that's the kind of trade-offs that sometimes happen.

I think that's what it reminds me of is that it's the— we had the same thing happen during our episode on vulnerability management at Google when the person described how we do it. And I thought, that's how everybody should do it according to theory. But the problem is we actually do it and many other people read the papers about how they should do it and then they don't do it.

Absolutely.

So we actually do it.

We are doing this. We are doing this also in the SRE on the reliability side, when we will sometimes accelerate some feature knowing that it may cause extra toil to the team. But there is a clear contingent, like, this is the plan of getting out of this situation, and we will prioritize it when the first business needs are met.

That makes sense. So, I want to switch subjects to one of my favorite subjects, topics, namely threat modeling and sort of specific attack vectors in this case. So, when you look at this complicated relationship of requirements and demands connected to our infrastructure, what attack vectors do you consider most critical in prod security? And how have you Evolve the defenses against those?

Absolutely.

Like, a somewhat cheeky answer is the ones that we considered in the paper, which is administrative access over production services, administrative access over production machines, and then the workload isolation issues that became fairly visible in the last years in the presence of, like, mostly in the presence of CPU side-channel attacks, but also kernel vulnerabilities and so on and so on. But honestly, like being in this space for many years, I can tell you that Google is very proactive and very good at thinking about those attack vectors before it's popular, like before you can read it on all the blog posts and so on. One example is the software supply chain integrity. This has been like a popular topic in the last couple of years. Google has been working on this like 5 years before. And what you like externally can use, like, which you know as Salsa, which is available to people outside of Google. Like, Google has been developing this for 10 years and it's really delivering on this front.

Yes. And I think this is kind of one of those fascinating things that if you tell them to non-Googlers occasionally, they find it quite incredulous that we, some of the things that they can see in the headlines, in August '24 shows up in Google internal papers in like January 2015. I mean, I'm not going to joke about it. Like, I had a couple of my own mind-blown moments when I was like, yeah, that's a really cool thing. And somebody says, yeah, we kind of had it since 2017. And I'm like, oh, never mind, ignore me. Like, it does happen and it does give me a lot of confidence in how we do things and our approach.

Absolutely. And just to give you also like some picture of how things changed over the years, when I joined MIT like Borg SRE, and like you can read on Borg in publicly available papers, which is pretty much our distributed operating system. This is all of Google Compute, pretty much.

Mm-hmm.

When I started, after like very difficult training, I was actually given root access to all the machines in the fleet. Pretty scary today, but 10 years ago, nobody really feared insider risk that much. But over the years, we got to a point where all of this administration is done. There are no more session-based Like, there is no SSH, there is no access to all-powerful ACLs. Everything happens through safe proxies. Everything is transactional, everything is peer-reviewed, or it goes through automation. And this is like 7 years ago, the director, when I told him, this is our vision, he said, this is impossible. And I kind of hold my beer. And like a few years later, there we are. Like, it's possible.

I think that we did an episode with Roel Hansen, and I think he brought up the same point about how in the ancient history of Google, we did have people with root access to things, which was like, admittedly felt normal for every other company. Like, sure, you have administrators, like you can't not have them. And then suddenly you realize, wait a second, you can rewrite this book and you can write a completely different book. In fact, it's a video, not a book at all, or which is based on different principles.

Absolutely.

So, from the threat factors, I want to shift a little bit to this concept that shows up in the paper, which takes a bit of explaining, namely that there are foundational services. And is it like one of those turtles all the way down things when we rely on things that rely on other things? Or so, what are those foundational services, and how are they significant for security of our infrastructure?

Of course, like every organization will have a set of services that get some form of control over most of production environment. And, like, if you think certificate authorities, like, there is a root of trust somewhere. And this is what we mean by foundational services. And when you ask, like, what this means for our security posture, it's like, those are crown jewels. You protect them at all costs. More, you don't introduce new ones because, well, like, every new service with such powers is a huge huge risk, right? Like, if it gets compromised, it gets compromised. So, you want to keep those services simple. Those services undergo regular reviews by security engineering at Google. They are protected with, like, the state-of-the-art solutions, like the ones described in the paper. And here, like, if you go back to the conjoined triangles, like, we shift maximum to security and reliability. And we will pay the additional cost and inefficiencies, for example, of having private dedicated machines, which is the workload security rings concept. Or you will maybe even pay some extra maintainability costs because, for example, I don't know, you would disable core dumps because there may be some very powerful secrets in there, even if we try to filter them out. So we will not have them. Debugging will be harder. But we don't risk losing our, well, root keys. You don't want to lose that.

Yes. I mean, that part seems like a good example where you can crank it all the way to security because so much other stuff relies on this. And I think that it's very clear that maintainability and cost elements would be forgotten. Well, maybe not maintainability, but cost. I don't want to talk too much about this, but like very clearly cost argument would be forgotten when you have this type of service. Like nobody would look at the numbers.

Not forgotten, but certainly lower priority.

Lower priority, yes. If you want to win the Understatement of the Year prize. Yes.

Exactly.

So to shift a little bit more to the foundational services, do we— I don't recall if we mentioned the list of them in the paper. Like, how small is the list? If we don't share the list, then how small is the list of services?

I'm not sure if we should be disclosing accurate numbers.

No, no, not the list, but like, are we talking about dozens or single digits or probably not hundreds, I would assume, right?

Doesn't say fingers.

Yeah, that makes sense to me because I think that this is an example where it's almost like, again, to repeat the same joke, we take security most seriously and the list should be short. Yeah, by definition. Of course.

And we want to keep the list minimal. In fact, we do have efforts to even remove the powers from some services that have the foundational powers, so to say. And yes, because we treat them as pets, not cattle, you need to keep the list small otherwise. It becomes too burdensome.

It overwhelms you. Yeah. So in that sense, a good reminder about pets and cattle. I'll put it in the notes because people kind of forgot that excellent metaphor, but you almost described some of these ultra-critical things as pets, which means they get the care and feeding and attention and pet food budgets of pets, not sort of the classic sort of like you individually don't matter, but the service matters.

Right. Absolutely.

So, at the same time, we have— when I read the paper, one thing that stuck out to me is that we, of course, have machines of different required levels of sensitivity. We process public things. We process things that we can reprocess next time if they fail. But there are also things like what you described, which are sort of so secure, we treat them as pets, in seeming violation of the traditions of SRE and automation approach, but not really. But how do we deal with this spectrum of services? Like, how do we secure things, given that there's a universe of requirements that range from ultra-secure to, yeah, okay, it should run.

Exactly. Like, this is where your workload security rings center, like, as part of the paper. I mean, of course, we can go into many more examples where we offer different levels of SLOs to services with different requirements, like, for reliability. For security, it means we decide based on the security posture of the service what level of isolation it gets. And like, of course, we don't want any service to be compromised, but like, if any of them get like—

to have—

take informed trade-offs exactly here between efficiency and security.

Yeah.

So we talked about foundational services. They get absolutely like the premium service and premium isolation. Then you would have the services, for example, which are processing user data, like all your Gmail, photos, and so on. Those get a pretty high standard, pretty high level of protection as well. But then you have services which are maybe not even services. Those can be some experiments or some batch processing that if it happens, it happens. If it doesn't happen, there is some graceful degradation and so on. So, like, roughly, you keep those separate, for example, from security position.

Yeah. So, it sounds what you're describing, again, sounds like what I theoretically would describe as a common sense approach. Like, of course, you would have the range, of course, you'd have the slider in this reliability versus security versus other requirements being moved in a different direction. But some of it is really hard at scale, right? And I think that the concept of zero-touch prod that's mentioned in the paper, plays into this because a lot of this is very easy to describe but very hard to do, especially at scale. So how do we use the zero-touch prod for this type of— for securing all that? Like, basically make it come to life for the audience. Like, I think they can read the paper, but this part is a bit dry, I would admit.

Of course. Like, that's why in the paper we reference the SRECon talk from like 5 years ago exactly on the zero-touch prod philosophy. which was primarily focused on the human aspect. This is coming, again, this is coming from the reliability angle when having too broad access to production leads to a situation where you will sometimes fat finger a command or make some other silly mistake. And then, like, oops, there is a massive outage, even if it's unintentional.

Yes.

And any unintentional outage can clearly be modified into a malicious outage if you have an adversary who gains some level of access. And we try not to distinguish between those because they lead to similar consequences.

Yes.

Similarly, we try not to distinguish between what we call compromised account, whether it's compromised by credential theft or phishing attacks and so on, or there is actually someone malicious who has this level of access because of, say, business needs, right?

Yep.

At Google scale, you never know what happens, right? So you want to be prepared. But your question is about difficulty. Yes, it is difficult, and it takes years. I explained the zero-touch prod presentation was literally motivated by our efforts to remove those root powers that Royal mentioned. and I've been working on for several years. Why is it hard? Well, first you need the controls. Like, you need technical solutions that, for example, offer you this temporary membership in a group rather than you always have it.

Yes.

Then you need infrastructure for multi-party authorization and, like, clear audit trail. You need better automation or, like, you need those safe proxies that we discuss in the paper.

Yes.

So, we need to implement all that, but that's not enough. Then you need to actually work with the teams who have those powers, like, for them to rethink how they do operations, how they do on-call. And instead of, oh, I have access to everything, let me hack up this quick shell script here, it's like, oh, I need to show the intent, I need to explain, and also put safety checks in place, like rate limits or otherwise. So, this is where we move to how do you adopt zero-touch. It's not really zero-touch prod for services. It is more deprivileging services, like removing or limiting ambient powers. So, you do this by, for example, checks and balances philosophy, where one system cannot really proceed without consulting with the other system. So, you would have some rate-limiting backends, which cannot be... easily bypassed or cannot be bypassed at all. Maybe operator, like, oh, you know, everything is on fire, we are in an emergency, maybe then. But like during normal operation, you wait, you follow the safety mechanism because the systems are becoming infinitely complex. There is like no human that can comprehend this all and sooner or later will make a mistake.

Yeah.

And similarly from what we discussed on foundational services, you want to have as few of them as practical. And just to give you an example, we had one system, like, very old, I don't know, 15+ years old, that had full control, like, it could schedule any jobs anywhere. And that was a complex, stateful system full of vulnerabilities, as it turned out from security engineering review.

Right.

We invested a serious engineering effort, like, multi-year effort, to change the completely re-architected system into a statefulness that had to stay, but it was deprived of powers. The powers resided in a very small, stateless, very easy-to-reason-about component that was also much more reliable than the original thing. And the risk reduction was tremendous here.

And this was also a good example where security and reliability were not at all in conflict. In fact, you improved one and the other improved. And that, to me, is a useful reminder that these cases are more common in a well-architected environment than people assume. They kind of always have a naive view that if you encrypt the data, lose the key, you're fragile. So, like, security, in their minds, kind of fights reliability. But in reality, well, in our reality, it helps a lot more than it hurts.

Yeah, I think one common misconception is people confuse security versus reliability with sometimes security versus time to mitigation. Because from my experience, security and reliability very often go hand in hand. One example is zero-touch prod philosophy that we discussed. Like, it started as a reliability thing, and then it became a very big security thing at Google.

Yeah.

Similarly, what we just discussed here, that this effort improved security, reliability, and maintainability of the system because new architecture was much cleaner. There was a cost. Like, that's where the trade-off went. We invested significant engineering effort to make it happen.

I think that makes sense to me. And I think that also makes sense as a reminder that the dimensions that we described in the very beginning of the call aren't— in fact, they can harmoniously move in one direction. Triangles can overlap. And then But only after you invested a lot of, you know, quote-unquote PhD time into thinking about it, so that your system is built so that moving the slider to improve security also improves reliability rather than automatically fights it. So, this is probably like philosophical thought of the day for the audience.

You always need to have architects. You always need to have those people in your organization who understand the bigger picture. And can really tell exactly, like, yeah, if we follow this path, here are the consequences. But, like, you cannot satisfy all 4 dimensions. Like, one of them needs to give, cost or, like, maintainability or otherwise.

So, we're almost at time. So, I want to move to our traditional closing question, which is give us one tip on securing infrastructure or pretty much anything we talked about here in the paper. And of course, one piece of reading. I guess I can put the paper as the piece of reading, but please give us a tip or maybe more reading. Okay.

For the piece of reading, one author that I can highly recommend is Google's, like, Building Secure and Reliable Systems from a few years ago. Each chapter of this book was authored by a representative from SRE, so, like, reliability-focused. Security Engineering. I think that book is a very powerful source of, like, how to design systems, how to build them for good recovery, and so on and so on. Of course, I already mentioned the Zero Touch Prod presentation from SRECon, which is more of a listening than— there is a recording available at the Xenix page. On the tip, like, something that I need to do often as a technical leader is like, do not hesitate to have potentially strong opinions and defend them. Like when, for example, I approach some product owners initially and say, well, here is the significant risk that you have. It's like, yeah, I don't want to really listen to this. And it's like, but here are the business risks that you are taking. Here is the consequences that may happen. Here is how you can alleviate them and so on. So, like, for TLs, maybe a tip is don't be afraid to have opinions and defend them well. Of course, in a thoughtful and respectful manner.

Yes, yes, for sure. And I think that's how the truth would emerge. Thank you very much for being on the show. This was very insightful. I think it almost helps technical leaders and maybe some of their bosses to understand what's involved. So, very much appreciate this. And of course, I hope more people who listen to this would read the paper as well.

Thank you, Anton. It was an absolute pleasure.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @antonchuvakin and @anton.mikhailov. @Anton_Chuvaikian and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode. Security.