EP 107 How Google Secures It's Google Cloud Usage at Massive Scale

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. You can also find us at cloud.google.com/podcasts. We've made it to the official Google Cloud podcast page. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well. twitter.com/cloudsecpodcast. Anton, what are we talking about today? This is a fun one.

So we started planning another episode on how Google does threat detection.

And I hated that idea.

It was a great idea, by the way. And you hated it.

I hated it. But why did I hate it?

Because it was a mismatch between a guest and a topic.

Well, partly that, but partly because we were missing the opportunity to talk about the left hand of security, that is preventive controls and how organizations can successfully build out a set of preventive controls and how security teams can work with application engineering teams to really make those work well.

Yes, but actually we also have a perfect guest for that because this person is literally doing exactly that at Google at scale and solving challenges. And to me, this is like a superb opportunity to cover things Other than detection. How about that?

Exactly. So perhaps with this bit of banter, guests, about how Tim and Anton put together topics for the show and how we argue and agree in advance on what we're going to talk about, let's turn things over to today's guest. And with that, I'm delighted to introduce Karan Dwivedi, Security Engineering Manager, Enterprise Infrastructure Protection here at Google Cloud. Karan, thank you for joining us today. You've had an interesting run of different jobs here at GCP. We want to focus today day on what you're doing now when it comes to helping us harden our use of cloud. I think this is such an interesting and different set of challenges than we have on-prem. So first, I want you to maybe paint the picture, set the scene, give us some context. Google uses Google Cloud for all kinds of different things, wildly diverse use cases. Could you just give our listeners a sample of some of the things that you see on a daily basis in Google's use of Google Cloud?

Yeah.

First of all, thanks, Anton and Timothy, for having me here. It's a pleasure.

Yeah.

So you're actually absolutely right. It's very diverse. And with that, I have a few categories in mind for the kind of things we see on the cloud, right? There's obviously production projects, which serve real customers, right? And they're running all externally exposed for reasons. We then have dev projects, right? People want to test out their ideas and make sure they can you know, run it and test it before they deploy it to prod. Then there's like a bunch of experimentation happening, right? Like Googlers are very curious. They want to try things out. So we want to enable that fast iteration, right? So that's another, I guess, category that we see. Interestingly, I've also seen like where we collaborate with external partners, like researchers hosting demos, right? All of those websites can also be and are hosted on cloud. Open source, another big one, right? Like we are very big into contributing back. And so a lot of that stuff is actually hosted on GCP. And then finally there's things like, hey, how do we do health checks for projects? Like meta projects? Like how do we make sure all of these things run very healthily? I guess that's like, in short, a summary of all the wild things I guess we have seen so far.

So that's a pretty long list of kind of exciting things and different things. And I guess one thing that adds to the excitement is of course scale, right? So, sometimes when people want to learn lessons from us, we share what we do, and they say, well, but you're not Google. So, is there any other secret to scaling? So, how do we think about scaling the right guardrails? How do we think about scaling some other security controls? Because just the other day, I was reading some SRE writing, and they said, if it's a one-in-a-million occurrence, it means at Google, it happens daily. Of course, I knew that.

Yeah.

And I knew that, but it's sort of like a useful Reminder, like it flipped a switch in my head. It's like scale affects things in all sorts of amazing ways.

Yeah, that's a really good question. So before I actually dive into like guardrails, I want to talk about a little bit about organizing things so that we can scale them well, right? So think about like us as a security team, we want to make sure we know what projects we should expect in which location, right? So folders actually offer that, I guess, enables us to do a hierarchy of projects. and different types of projects. And like, we like sort them out to say, oh, is this production? Okay, go here. Right. And have your own folder hierarchy within that. Like that really helps us have a consistent, like set of defaults to apply with. Right.

How do you track that? How do you keep a handle on that? How do you put that into code?

I think it's more of like infra as code type of thing. And a lot of it is just internal tooling, which obviously, you know, it's not public, but I guess the concept-wise, the TL;DR is making sure you first Take a step back before we even think about scaling, make sure you organize your projects well enough into, I guess, functional categories that you can then manage and have initial guardrails around, hey, do teams know where to put them in the first place? I guess that's problem number one before we even go to guardrails. Now I'll talk about guardrails. So GCP enables a lot of things that help us scale. For example, we have hierarchical firewalls. You can have folder-based, stack-based, you name it.

And we also have firewalls, not all of which are called firewalls in Google land, right? That's been annoying to me personally, because sometimes it's a network security control meant for access control, but not a firewall.

Right. Yeah. The naming aside, because I know a lot of names are just overloaded, right? But to talk a little bit about controls, right? So we have, like, making sure we use both folder and tag-based methods to scale the control application.

Folder and tag-based.

Folder and project-based, yes. The hierarchy should tell you basically functionality-wise, but within that, you can go and do groupings. Because it's impossible to do scaling for the amount of projects we have internally. That was challenge number 1. Now, challenge number 2 is how do we secure this and have the right guardrails is making sure you use a combination of things. For example, org policies. You can have baseline org policies at the top, organizational level, and then folder level. And then, at the project level, you can actually have that fine-grained and say, oh, I'm okay with 10 of these, but the 11th one is going to not work, so I'm going to have an exemption there. Similarly, I talked about firewalls a bit. You can apply those. And then, finally, let's not forget about IAM allow and IAM deny. Those are very key tools that we use. They're powerful levers. that we as a security team use in combination with everything I described to say, hey, let's deny by default here, and let's allow here. And, like, again, track those exemptions all in place. So, it's all infrastructure as code at the end of the day. But I guess, Anton, TL;DR, it's complex, and we deal with it every day.

Yeah, I was about to say, there's no magic, it sounds like. It sounds like you do the work right, and the results are right. It's almost, like, a little bit underwhelming because I thought he would say, Kieran would say, well, but we have this tool. You turn this on and you can do a billion things immediately.

No, Anton, it's going to be machine learning. You feed your hopes and dreams into the magic box and security comes back.

Exactly.

Thank you. The ML unicorn comes out. Right. Yeah.

Yeah. There's, there's a bunch of tooling behind this. Obviously I didn't go into like something that's not public and stuff, but I wanted to share the concepts that helped us scale.

No, I really like that. So, speaking of things that are translatable, hopefully, to the outside world, how are you finding success working with application teams or engineering teams, getting project owner teams to work with you and not feel like you're in the way of business? Because so often on the podcast, it really seems like a team needs a family therapist, not another tool. So, what have you found to relate to these people that really works?

Yeah, that's an excellent question, Tim. Basically, we want to find the right balance, just like everybody else, right? This is a cliché, but make sure we find the balance between having the right security assurances for each team while also allowing a baseline set of controls and exemptions to those, right? So it's that balance. And like, if I'm being very, very honest, in cloud, it's very contextual and extremely challenging, right? So this whole landscape has changed how we think about things. I'll give you an example of how we approach this, right? So When we are deploying a control, any of the controls I talked about previously, we follow a very methodical approach to do this, right? So let's say you want to deploy an org policy or any control. We would say, hey, before we even do this, can we actually accurately measure all the violations that we'll run into before we even hit deploy, right? And there'll be a bunch of those. And then we'll be like, okay, can we actually grant some sort of temporary exceptions to these teams? So that they don't break, right? We obviously don't want to cause that friction of like, okay, we are increasing security. Oh, but you know what? Here's an incident, right? No, we don't want that. So we definitely check for violations right away. And then we are like, hey, FYI teams, here's an exception. This is what we are trying to do. Here's a timeline, right? So talking about like, how do you make sure you work with them is like setting the right expectations. And giving them the time to react, giving them the time to talk to us. And then, when finally we are ready, right, we have all the exceptions in place, we would just go ahead and then deploy. And it would be wider because we might do it at a higher folder level, and then we will work through the long tail, if I may. So, again, this isn't perfect. We see things break all the time, but it is still a very, like, systematic approach to reducing risk.

Things break all the time in what sense? Meaning that you roll things back? I remember the SRE episode, right? So things breaking, of course, isn't a disaster. It's just a matter of you rolling changes back, I would assume, right?

Yeah. So we wouldn't roll— we'll try to avoid rolling the entirety of the change back because then you're adding more toil. We have controls over, as I said, infra as code. So we can actually roll back part of it. Say, oh, for this folder, if you're still breaking and we were not aware of this, so we didn't give you an exception, we can roll that part back. And then we can track with a bug or any communication with that team. So, it can go fine-grained after we have deployed it at a wider scale. And I guess, in my experience, that's the way to scale, is you just cannot start with single projects. You have to go at a hierarchical level.

But it sounds like, at least whenever I've seen people apply a process to a vast scale of anything, there are always exceptions. There are always little people showing up out of the woodwork and saying, hey, but can we do it this way for our part? And I would assume if you have a 100-person team and one person wants an exception, it's unpleasant. But if you have a million machine systems and it's a very complex system, how do we scale the exemption management? It sounds like a nightmare. I'm already having flashbacks to some past time, like, yeah, don't patch this system because X.

How do we deal with that? Yeah, that's a question that's near and dear to my heart because I manage the exemption management team directly.

Ah.

Yeah.

You better know these.

Otherwise, we are screwed, right? Yeah. So let me touch base a little bit on that. So imagine exemption management or exception management as horizontal across the platform, right? It can be any type of control. We want to make sure that it's horizontal. You can request any type. Now you're right. It's a hard problem. That's why we have it, like actually a team that I'm leading doing this. The way to scale this, right? I've given this a lot of thought with teams and like the smartest engineers in the room. is to think about the atomic components, right? What are those individual pieces that an exemption wants? So basic CRUD example, right? You want to create an exemption, request an exemption. You want to read, make sure you can see all the exemptions that a project or folder has. You want to update an exemption. Oh, I want to extend this for another like 30 days. Can I? Right? It should not create toil, right? It should be direct. And then finally, like, hey, do you really need this now? Because it's been a while and we are going to revoke this. And we don't have enough humans to revoke, grant, and read exemptions at Google scale.

Of course. Of course.

So obviously, for each of these, we try to then start automating. And that's where all the challenge lies. So when we are thinking about— now let's go back to what I said about deployment of controls. When you're thinking about, hey, what are the violations, we also start thinking, what are the conditions and criteria for exemptions. Like, what can we expect from these teams? So we try to do a little bit, like, think ahead.

You don't call it exemptions, it's code, I hope, right?

Yeah, yeah, okay.

Okay, Jim is smiling, Karan is not, so I'm nervous now.

No, I'm a little bit serious right now because I'm really going into my team stuff. But when we think about that, we are already, like, prepping our tooling, right, to say, hey, like, for when condition X, Y, and Z is met, Allow the team to get an exemption to this policy or this control, right? And so think about that. We have not even deployed, but we are all set to face that load of requests from humans. And so I found that extremely critical for us to scale, right? And now we also have this, I guess, problem around brownfield is like, oh, we already granted, can we now start automating that pile? And so we are always basically firing up our cylinders to say, hey, what can we expect next? And what do we have right now?

Yeah.

Now that's just the automated beautiful piece that's easy to say, right? What about that pile of exemptions or problems that we need a human, right? So we scale this out by both automation and humans. So first we'll try to automate as much as we can through conditions, which I feel like not everybody needs that. I guess Google scale is too special for that. And then we also make sure we have some humans which are actually going in a rotation and looking at this stuff. And saying, okay, do I need to intervene here? Is this playbookable, right? Can we follow a set of processes? That really helps us. And then like finally, long tail is where the complicated places lie, right? Complicated technology solutions, which are like, oh, this needs a review. This is not an exemption right away. This needs a review because we don't know the risk. And so we'll involve like security risk and other things to make sure we are really granting the right ones for the business. So that was a long-winded answer of saying it's complicated, but we have a method to this madness.

But wait a second. So let me ask you a strange question.

Yeah.

It sounds like you're describing a very rule-based process to exemptions, but like exemptions are— there are rules and there are exemptions, but you have a rule-based exemptions process. It kind of explodes my brain a little bit. And I need to understand if you have exemptions process that's managed this code. Why aren't they rules? Why are they exemptions?

With this sort of rule, so you're right, this almost has an analogy in the detection space where you write detection rules and trip something, but that's something that's actionable. In the exemption space, it's the same concept but reverse. We're like, oh, we know this is going to unblock your use case. So here's a rule that will automatically allow you to enable your team to do work, and then we'll track it. So, oh, I should also mention that we track each and every one of them. So again, there's some scaling required to make sure, are you tracking the right ones at the right level?

Okay. That's actually more clear. And I think I understand that you can have a rule-based systematic approach to exemptions. In fact, at Google, it better be a systematic approach to everything, otherwise it wouldn't scale. It wouldn't. Yep. That flipped the switch in my head. So here's a strange sub-question to this question. Is there something in that process that's sort of not usable at smaller scale? Where scale is kind of maybe— I'm thinking more of an emergent property kind of situation where this only happens at scale and it does not happen at smaller scale. Am I crazy or not?

No, you're not actually. So I'm going to almost go on a limb here and say the amount of complexity and thought that goes into that automation piece of having this criteria set and combining that with other Google-specific technology to say, hey, what type of data are we exempting? Like what is the real risk? And like having that as conditions, I think that's pretty unique to us, right? I can imagine people asking for exemptions, like revoking them, renewing them. I get that. But can you do like conditional-based? I think that's very unique to us. And that's something that I think scale drives a lot. Because think about the other case, right? If you don't have that, right, then you run into this massive risk acceptance across the board. Right. Which is because it's just going to drive—

oh God, yes. This is the original nightmare.

That's right. This is going to drive like VPs crazy and like, they'll be like, oh wait, is there no way to drive this down? And the answer is yes. Let's do nuanced exemptions, if I may.

So in that nuanced exception thing, how do you federate the responsibilities in that model? How do you make sure people are asking for the right things? How do you make sure people aren't accepting the wrong risks? Like, what does it look like in practice?

Now we're getting into the organizational challenges of this thing. Okay. Yeah, yeah, yeah. I love this. That's why. Okay. So I think it starts with having the right RACI. Right? So making sure your roles and responsibilities are clear upfront. What that means is, hey, we as a security team, we're going to define the controls, we're going to like make sure we deploy the controls. The risk that we're reducing is across the board, but it's the business that owns the risk, right? We want to make sure that line is like communicated out loud a million times.

But isn't it like so 1990s advice, which ultimately is the right advice, But people don't truly follow it all the time, right?

Sometimes. I mean, I'm almost surprised. Sometimes I see people saying, oh, it's the security team's job, right? I'm like, actually, you own the risk. So, you know, I need a sign-off from your VP. So anyway, I think that's point number 1. I still hammer it. It's 2022. So yes, that's, I guess, to Tim's point, that's, I guess, point number 1. Now, there's also another aspect, right? We want to be empathetic. It's not just like, hey, we want to like keep driving down risk and like, we don't care about the teams. No, actually it's the opposite. We want to enable the teams. And what that entails is, are you making, showing a clear value add? Okay, you're reducing the risk. So what, like, how does that affect us? Can you tie it to their business? Can you show the impact that they get? Right? Can you show that their lives are easier, more productive in the future? Right? So we want to make sure we factor that in, in our communications during deployment of controls, all of that. And then we also want to make it easier finally to have the right level, and this is very important, right level of stakeholder involvement, which means for low-risk, you know, low-risk things, don't go to a VP. You don't need to go to a VP, right? You can actually do it somewhere in the middle of the hierarchy. So are you scaling your decision-making as well is super important. I think the combination of that really helps establish that healthy, like, Okay. Here's the security team working with the business.

This makes sense, but I think that I still want to kind of insist on maybe a slightly more crisp answer on the whole federated responsibilities, right? Because sometimes I would assume the exemption management process is central, right? So it's sort of like set by presumably Alphabet-wide or cloud-wide, but the point is centrally. So how do we decide which responsibilities go to teams and which are central? And then who decides that?

So there's a shared understanding, right? A lot of it, again, I don't want to speak on Google's behalf, right? But I will say a lot of this, if you're trying to federate in general, right, I would start off with clearly defining who's owning and who's doing what in the first place. And now within our company, right, we want to make sure that we own it. We own the tools, we own the controls, and we deploy it right now. This is, again, it doesn't start off at a team level. You're right, Anton. It starts with actually the leaders of orgs and VPs sitting together and saying, By the way, my org wants to reduce this across the company. Do all of you agree? If yes, then you're going to get communications from my team, and you're going to start accepting those risks. And we do this across the board. That's what I want to emphasize. I guess that's the most important piece. Now, there are more nuance, to give you a more crisp response, right? There's more nuance to who's doing what at a lower level. So, for example, right, if we want to say, hey, can we rotate a key, right? Yes, it helps us reduce the risk, but it's actually the business and the teams doing that. So we make that clear when basically it's about control of code or projects. You're actually— the teams are owning it. They have control. We want to give you the advice, but we are not going to do that on your behalf because we are going to break stuff. So we're going to do it with you, but you're going to do it. So there's some explicit stuff, there's some implicit, but that implicit decisions are also driven by ownership of Like resources, if that makes sense.

All right. One more question about organizational stuff, because it's been on my mind lately. Burnout's a perennial problem for security teams. We all deal with toil and alert fatigue and the problem of chasing people who don't want to be chased. How are you keeping your teams happy and engaged? It's not easy. So what are you doing?

It's not, it's absolutely not. And as somebody who manages people directly, this is like top of mind, right? I will share a couple of ideas, right? What I do on a daily basis, what I do on a quarterly, yearly basis, right? I guess check in with your folks every one-on-one, right? Even before I get into like, oh, what's going on on a team level, a business level, I want to make sure they are doing okay as a person. That really impacts what they're doing day to day, right? So that's like a personal check-in I do. On a much larger, like, longer time frame, I want to make sure I hold like regular career conversations, right? Again, this is not tied to burnout, but it will actually help me solve burnout. In those conversations, I try to figure out what are your real passions? What do you want to do, right? Where do you want to go? And I try to then make plans for them to say, hey, if you want to achieve goal X, here are like 4 steps. And some of this is in your role, some of them is outside. And for the things outside, because, you know, in the role you may get burnt out, when you reach that stage or before even you reach that stage, here's a bunch of things that I want to encourage you to take up on the side. So you're engaged, you're figuring those things out as you go. And you're seeing progress in your career. It's not just about like your day-to-day, right? That's very, very, very key, right? And I've gotten good feedback about, hey, I love doing this, can I do this more? And then we start changing assignments, right? And then less people get burnt out by it, and this scales and this cascades across teams, across orgs. So it's a very small thing which is not tied, but it directly ties up later. That's point number one, right? I think in general, also make sure you show people, hey, the work you're doing, here's a direct business impact. Can everybody on your team show how it impacts the business? If they can answer affirmatively yes, then they see a larger goal to what they're doing day to day, right? Approving exemption requests, deploying a control, all of this is like security engineering work. But do they know how it's working and how it's helping Google, Alphabet, right? That's very, very important. And I think finally I've seen just great value in asking people how you want to be recognized. Because like, you may work around the clock and get things done, and you've done a massive, like, great job at reducing risk. But sometimes people are like, oh, I want to be recognized publicly across teams. I want to see it in my compensation. Or I want both, right? And it's up to like leaders like us to make sure, hey, remember this person? That's how they want it. And so that's very critical because they're gonna come back and say, oh, I did this amazing job, but I still don't feel recognized. And then they'll be like, oh, I'm doing this again and again. I'm not getting that thing from you, affirmation from you. And then they'll definitely be burnt out. So there is this like obvious components to avoiding burnout. And then there's like a very personal part. And I try to hammer both because you wanna keep your teams happy.

But so basically you can say there's no magic, but actually there is magic because if I look at DNR teams, that report that for them, it's also one of the big secrets how we have people on essentially SOC-type responsibilities doing the job happily for 10 years, which is kind of like unheard of pretty much anywhere in the industry. So you're describing a model where a few small things organized rationally and intelligently do produce magic.

It does. And I was part of DNR 4 years before this team. So I've done that work essentially at Google. And I know exactly why I kept doing it, what kept me going. And I was like, yeah, this model works. I'm going to try to replicate the same thing in the infra side of things.

Perfect. So, we're nearing the time, and I wanted to ask— we wanted to ask our final questions, which are any recommended reading on any of this, guardrails, scaling, anything that you prefer, and one tip for the other teams, perhaps not at Google, scaling their security safeguards.

Yeah, let's start off with the tip, I guess, because I think it's said so many times, but I don't think it's, I guess, really absorbed. Even before you address scaling and cloud security, just take a moment to acknowledge the beast of a problem technically and organizationally that you're trying to solve. It should not be just swept under the rug that, oh, it's a problem, it's going to take you a year. Uh-uh. It takes years to do. Recognize that and then get buy-ins from your leadership on, hey, what's your restorements like? Like, we have these many risks coming out. Do you think you can accept these? Because we can't really sit for 5 years and mitigate this. It's going to change. So, have those conversations early with folks at the higher levels and set the right expectations. So, because when you execute, you are in that firefighting, oh, I'm going to deploy this control. I'm going to exempt this, and I'm going to review this. And the bigger picture starts getting lost unless you go back every quarter, every 6 months, and say, hey, here's the progress we have made, remember? And we have touched all of these risks. So I guess that's like one big tip I would start with, along with everything that we all have discussed, like the last 30 minutes. Reading. Yeah, I guess personally I found a lot of reading that I guess you folks have done in this podcast, outside that. I guess, Anton, you have published a bunch of stuff, but I think I saw the Cloud Security Foundations Guide. I believe it was on services.google.com, if I'm not remembering, if I don't remember correctly. I felt that was—

you actually read it, all the 61 pages of it?

You really did? That's impressive. I found it. I'm like, wow, this, you know what, this is good research. So, I want to mention that one, actually. I was like, read that, listen to people coming on this podcast, especially talking about cloud security, and go implement a plan. You don't need— there's no rocket science. It's just find a methodical, scalable approach to this problem. And right, just execute and learn and then execute again. Like that's as simple as it gets.

Perfect. Well, with that fairly optimistic ending and a day that hasn't had magic but has been very actionable, thank you so much for joining us. Thank you.

This was awesome being here.

And now we are at time. Thank you very much for listening and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website. Cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.