EP176 Google on Google Cloud: How Google Secures Its Own Cloud Use

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Tim Peacock, the Senior PM for Risk Analytics, Cloud Threat in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on our community page. Anton, we've got a returning guest, all-around awesome guy, somebody who needs to take more vacation, and just fantastic Googler, Seth Vargo, today. What did we talk about with Seth? Because it could have been anything, But this was special.

Yes, this was one of the more magical episodes in recent history. And I don't use this term very frequently, right? Because even if it's a good episode on SOC or DNR, I think it's really good, but I don't say magical. This one was quite magical.

What is it?

You know, you would never guess. Like, if you have to guess, do you think Google uses cloud? Do we like cloud? Do we not like cloud? What do you think people would guess?

Well, it seems hard to imagine how How could Google, with all of its sophistication, use cloud? Surely it needs to use its own systems only, right?

Right. That's if we are more like a large enterprise versus cloud-native, because if we were cloud-native, then we'd be using cloud all the time, right?

But why would we bother to use cloud? We have all these great internal systems.

Because it's ours.

Hmm.

You think we should ask Seth?

Yes, exactly. We should ask Seth.

Let's ask Seth. Listeners, with that, let's welcome today's guest. I'm delighted to introduce today's guest. We've got returning guest. Last time he talked to us about onions versus garlic. We've got Seth Vargo, principal software engineer responsible for Google's use of, well, public Google Cloud, which is pretty cool. So Seth, maybe I've already answered the question with my intro of you, but Google uses public cloud, right? And which one? I assume it's ours.

That's a great question, Tim, and I think you're going to actually be super surprised at the answer.

I'm shocked.

It turns out that the answer is yes, Google uses Google Cloud, but that's actually not the only public cloud that we use.

Say more.

You know, you may be familiar with products like Looker, which, for example, run across multiple clouds, or even BigQuery, right, our data warehousing solution, which runs natively on AWS through our BigQuery Omni product. So—

That's pretty cool.

While I would say the corpus or the volume certainly leans towards Google Cloud, we've also got A small number of workloads that do run on other clouds. And we've also got acquisitions that might come in running on a different cloud provider. And, you know, we need to meet Google's security and privacy bar while they can still meet their customer needs. So it's actually really the public cloud, not just Google Cloud.

So you're saying when we acquire someone, we don't immediately turn off their existing deployment on Azure and magically put them on GCP. They actually, they go on using it. That makes sense.

Okay. This is trending in a dangerous direction because you said Azure. That's an assumption. Maybe it's Amazon. I think Azure is kind of a low bar in this case, right?

Well, I mean, there are other lower bars than that.

But so we do do it. That makes sense.

Yes.

So we use cloud, okay, as a way to sort of benefit from its advantages and whatnot. But how are we like other clients of GCP and how are we not like other clients? I mean, I don't know. In my head, we clearly have similarities and differences from a quote unquote regular GCP users. So can you shed some light on the security side of that?

Yeah, so I have to back up a little bit, because I think it's pretty common industry knowledge that Amazon moved almost all of their workloads to AWS. And at Google, we took a slightly different strategy, because we have purpose-built infrastructure. We call it Prod, capital P Prod. That specializes in running global multi-tenant services that scale to billions of users. These services you're familiar with, like, you know, Gmail, for example. At the same time, we've invested heavily in that ecosystem. And it is specifically designed and deeply ingrained in our DNA, in our processes, and our security and privacy tools that optimize for developer productivity. So I would say unlike maybe other cloud providers that moved maybe like less mature or more vintage production stacks onto the cloud, we already have an incredibly mature private cloud. And our strategy for adopting the public cloud is actually a hybrid approach or a multi-cloud approach, depending on what terminology you want to use.

Hmm.

It's our goal to enable engineers to leverage the best-in-class technology, regardless of whether that's on our purpose-built data centers or on a public cloud like GCP. As you can imagine, that introduces some interesting security and privacy challenges. Notably, much like every large enterprise, we can't just open all of GCP up to our developers. Just like your average enterprise doesn't give every single developer the ability to interact with every compute network and storage component, we have some governance processes in place, and we have integration or You can think of it as a platform that sits on top, very thin, that sits on top of GCP, that provides things like governance, abstraction layers, and API compatibility. Many of that leverages native GCP functionality, particularly like the org policy service, which is one of the ways that we control scope. It's one of the ways we manage risk. And it's one of the ways in which we create an environment in GCP that is comfortable enough for us to run consumer workloads.

So that last part is really interesting because org policy is one of the things I talk to users about using all the time. Say more about what you're doing with org policy and why that's worked out for you, because I imagine there's such a big gap between what you get with Borg or even a layer on Borg like Pod or Bok versus just Google Cloud. So how did you bridge that gap?

Yeah, so for folks that aren't familiar, like Borg is our kind of like production scheduler and our compute environment internally. And Bokk and Pod are, you can think of them like application frameworks plus plus, that kind of, it's how we deploy infrastructure and code internally in our production stack when it's not running on GCP. So in terms of how we leverage the org policy service, today we lean heavily into the resource hierarchy.

Hmm.

So we have different threat models, just like any mid-size or large-size company. Everything ranging from experimental, which is, you know, go, here's the cloud console, go click buttons, go do whatever you want. But there's, you know, restrictions on billing, you can't get public IP addresses, you can't expose this stuff to the internet, all the way up to, you know, production workloads that are interacting with sensitive data types, like health information, PII, etc., that are extremely governed, extremely locked down. And we leverage the resource hierarchy to enforce different organization policies at different levels, depending on the threat model. So our approach to GCP is, I would say, a threat modeling first approach, where we're looking at the use cases, broad classes of use cases, identifying the specific threats, and then curating an environment that matches that threat model.

That is a very reasonable approach, it sounds like.

It is. But at the same time, it's important to understand the scale at which Google operates on Google Cloud. I have some metrics.

Oh boy.

So let me pull these up. I come armed.

We're not gonna have lawyer cats dropping from the sky.

Uh, no, these are actually approved by lawyer cats.

Wow, cool.

Okay, I'm checking my ceiling. No lawyer cats falling yet.

So we've actually got 42 million Google Cloud projects that Alphabet owns and operates.

That's like a lot.

It's a big number.

That is, that is a lot. It's actually nearly an order of magnitude larger than our next top 10 largest customers combined.

Combined? Oh, that's a good number.

The scale at which we operate, the number of use cases that we're supporting on cloud is extremely vast. And that both is an exciting challenge, but it is a challenge because many times we actually run into limitations of GCP, the platform, that other customers would never hit. For example, we've actually tested the boundaries of the resource hierarchy in terms of depth and breadth. And we've had to make some trade-offs, some technical trade-off decisions that no other customer would have to make. But because of the size and the scale that we operate, you know, we've invested in a program right now where we're actually able to shard workloads into a different GCP organization where we can control a different threat model at the organizational level instead of the folder level.

You know, it's funny you should say other customers won't reach because to me, I've been at Google almost 6 years now. One of the things I've seen is what seemed big back then seems small now, and what seemed exotic then seems everyday now. So I imagine these aren't things that other people will hit today, but you're actually probably paving the way for people to do it in the future.

Absolutely. Like, the technologies and the foundations And the problems that Google is hitting right now are absolutely going to be problems that even small startups are likely to encounter, you know, 5, 10, 15 years from now. So that's why, you know, in many ways, Google is a super mature enterprise that only uses, you know, GA products and services that are covered under an SLA. But at the same time, we're hitting architectural barriers that our customers are not yet hitting, and we're paving the way and partnering with GCP to make sure that when a customer comes along that has a similar scale, they don't have to do something crazy and bespoke because we can support it natively. I don't know, Tim, if you remember way back in the day, we tried to turn on Cloud Security Command Center for the google.com organization.

I do remember.

And I forget the exact number, but I want to say it lasted for like 6 seconds before the whole thing fell over because—

We might have had some scaling challenges.

Yeah.

And, but that was actually super informative. You could probably speak to it more than me, because it really drove a bunch of OKRs and initiatives to get SCC to the point where it can scale.

Yes.

And as we just saw during Google Cloud Next, what was that, last week, 2 weeks ago, time is an illusion. You know, we're expanding SCC to support even more workloads. And like, I would argue that none of that would have been possible if it wasn't for that one time that we turned it on. And it fell over. And then it was all of a sudden a recognized priority that we actually have to care about building this for scale.

I think that's more true than you know, the degree to which that drove conversations on the team. Speaking of security, how has our use of cloud, Google's use of cloud, informed, other than scale, our security products roadmap and how we're securing Google Cloud?

Great question. So Google's been at the forefront and requesting a lot of features that many of you are using today. I think a canonical example is probably IAM deny, which is the, you know, IAM, Identity and Access Management, as you're familiar with, is based on the resource hierarchy and it has an inheritance model.

Which is awesome and super powered compared to other public clouds.

It is certainly different and offers more capability than other public clouds. However, that comes with challenges too. Which if, for example, you want to restrict something at a higher level of the stack or a lower level of the stack based on the default inheritance model, that was really difficult. And, you know, folks on my team worked really closely with our cloud identity team and our access management team to design, build, and eventually launch the IAM deny functionality, which allows us to inject a rejection policy in that hierarchy. And it sounds like a very small feature, but remember, we just talked about the scale at which Google operates. The number of IAM policies that we evaluate on a per-second basis is huge. I can't disclose that number. I don't have permission for that. But it's really big. It's a very big number. And the result was like, we really needed to build a feature that could work for Google scale so that it worked for every other customer as well. That's part of the reason it took us, I would say, a little bit longer to build IAM deny than we would've liked. But the result was an incredibly scalable solution that would work for Google and would work for external customers.

I love that example. And that's such a, like, you know, in some ways you might look at IAM deny and think that's a totally irrelevant feature because you could just not give people the thing in the first place. But it's also this incredible peace of mind for security teams to be able to affirmatively deny the access. And so, I love that example. That's a great example.

Yeah. I was about to reach for some kind of 1990s firewall rule example, but I figured it would date me, so I'm not going to go there. But I think that it's like— Yeah.

It's a good feature. It's a good feature.

Yeah. It's quite incredible. But wait. So, one thing that came to me when you were describing this is that I sort of assumed, perhaps naively, that our cloud use would be kind of like cloud-native, cloud-born, whatever we say now, style of use, but some of the things about hybrid also makes me think of traditional enterprises and of course the size. So are we more like a cloud native or are we more like a traditional edge company in terms of our cloud use? Like that's, I'm curious about that.

So it's a fantastic question and the answer is both actually. We have critical parts of the business that are very enterprise driven. Our corporate engineering team, for example, looks very much like a traditional kind of enterprise business. We run third-party software on a virtual machine, maybe managed by Chef or Puppet, that is deployed via Terraform, right? Like, I would say very traditional, I'll say 2000s, to date myself, because that's the way a lot of that software is designed to be run. That's kind contractually, in some cases, how it has to be operated. But we have other parts of the business that are leveraging containers and serverless and built for scale, built for purpose, scale to zero, scale to a million. So we really actually have all of it. This goes back to what I was saying earlier, where depending on where you are in the threat model, and, you know, the type of data you're interacting with, and the criticality of the business use case, it determines how, like, enterprisey you need to be. At Google, we really value experimentation. We believe that every Googler should be able to use these tools, whether it's on our prod environment or in our GCP environment, to learn, to experiment, and to see what's possible. But they shouldn't be able to do that on user data or YouTube data, for example. It's very constrained. It's as if they signed up with a credit card off the street, not as if they're building a production application.

Hmm.

So we have these different segmentations so that we restrict access to data to still enable maybe more experimentation-style use cases. While as we move further down the stack, as something goes from proof of concept to reality, that we have more controls, we have more governance, and frankly less choice as you get further up the stack. The example I love to use is GCP has, I want to say, 7 serverless products, depending on how you want to count.

Yeah, that sounds about right.

As a company, I think, and as any large enterprise, we're really going to pick one for our production installation and put all of our apples— arrows, apples behind that one.

More apples behind fewer arrows.

Makes sense to me. I would say in these lower-level environments with experimentation, go ahead, use Cloud Functions, use Cloud Run, use App Engine. But as we move into the higher-level environments, there's less choice. And it starts to look a lot more like a PaaS than like an IaaS, for example.

That's interesting.

Yes.

What actually did you choose as Google's serverless of choice?

Oh, that's questionable territory there.

Yeah, yeah, I think that I'm checking my vents for lawyer cats.

Oh yeah, maybe I'll cause all of the other PMs for the other serverless products to come chase me.

I hear them meowing. It's the lawyer cats in the ventilation.

Well, okay, then we'll move along. We'll move right along, listeners.

I'm just gonna say that we run our code everywhere. How about that?

Ah, very good. Very good. So moving right along, I want to ask, other than this hierarchy and this necking down of possibilities, you get higher up the production neediness stack. What else have we done to make this work at scale? And what Google-informed advice would you take from that and give to customers?

So I think there's probably 3 things that I would want customers to leave with. The first is there are many, many, many ways in which Google is actually just like you because Google is a startup, a small and medium-sized—

Wait, wait. Sorry. Sorry for interrupting. This is actually— I need to focus on this, zoom on this line because in some cases, we have to tell people that nobody is quite like us, but in this particular case, it's very important to know that they are in fact like us.

Yes. So the first part here is like, Google is both a startup, a small and medium-sized business, and a large enterprise all in one. And we have all of the challenges that come with being all 3 of those kind of simultaneously. At the same time, Google is a massive global company that is targeted by, you know, state-sponsored, nation-state-level attacks that I would say your average small and medium-sized company is not going to experience. You know, unless they're operating in a very niche space. That's a really difficult zone to operate in. Because we want to act like a digital native, we want to move quickly, we want to deliver value to our customers, whether that's our GCP customers or our consumer customers. But at the same time, we want to protect those users, we have an obligation, both moral and ethical, to protect those users, making sure that their data is secure from both insider and outsider risk. So when we look at things, you know, the number one thing I would take away is that Google is driving features and functionality in GCP that we need in order to secure our use of the public cloud. And that's coming down and being made available to customers as features that they can use as well. But because we want to move fast, and we want to accelerate the rate at which we can deliver features. We're doing so in a less traditional, like, waterfall 2-year planning. It's really, you know, how do we be agile with this? The Custom Org Policy Service is actually a great example, moving on to my second point, of where we've kind of walked the line of enhancing GCP natively versus doing very specific Google stuff. And being able to leverage and author our own custom org policies gives us a lot of flexibility because we have some internal policies. We have our own identity system that doesn't really look like anything else on the planet. So we have policies about how those identities are propagated, how those identities make their way into cloud systems for real humans and workloads. So we can write custom org policies hooking into our internal systems to make decisions about what is permitted or disallowed. But in many cases, we've also then been able to partner with GCP after we get our thing working and saying, hey, we think this is actually a general-purpose organizational policy.

Mm-hmm.

I like to call it getting it promoted, right? Can we get this promoted to a first-class org policy? So that's the second thing is like, there's a lot of things that I say, like incubate within Google, that then become first-class products, services, or features directly in the Cloud Platform.

But are there practices as well? Presumably, there are some things we don't— not just tools that may become products, but is there anything interesting on the practices side? Maybe we do things in a particular way that others can learn from? I'm not sure exactly what I'm asking, frankly, but I suspect that— and ultimately, most of our practices are, in fact, automated, so are they really tools?

So, I guess what I'll say is that on the Practice aside, as a company, Google invests really heavily in automation and scale, we have to. So we leverage infrastructure as code tooling for everything. So our org policies, well, they're configured via Terraform, our deployment infrastructure also configured via Terraform, when we need to make a fleet change, right, that is actuated via infrastructure as code. And, you know, we're using the same tools and technologies that most of our customers are using, with our interactions on GCP. In terms of the process, I think, you know, we have a somewhat traditional structure, at least I think it's a traditional structure where we have kind of offensive security, defensive security, we do penetration testing both as a customer of the cloud provider as well as a producer of the cloud provider. So I think all of that probably looks very similar to what you would see. We've obviously got Mandiant that's doing, you know, more ecosystem-level threat analysis, and then feeding that back into Google's use. We've got TAG, which is also analyzing a lot of this work. But, you know, we have, we have security engineers who are focused on how do we build well-lit paths that are impossible to be insecure. And then we have security engineers who wake up every morning and just try to make my life as difficult as possible.

We have a few of those here, don't we?

In a good way, in a good way, right? I will say that impossible to insecure and secure by default are actually different bars. And at Google, our bar is impossible to insecure, or you can say insecure by exception maybe is another way. Whereas secure by default is actually a lower bar.

Yeah.

Secure by default means if you click the button, it's default, but you can always click a different button and make it insecure. Whereas impossible to be insecure or insecure by exception means that you really, really, really need to go out of your way. And there's like 50 roadblocks and potentially some executive sponsorship before you can do something that we perceive to be insecure or not in compliance with some policy.

I think that's what Phil calls inevitably secure. I think that line came up.

Inevitably secure. I like that.

Yeah. And I think that's— it's certainly not just secure by design and by default, because you can secure by default and end up insecure. And secure by design, it's kind of about it's foolproof, but they made a better fool. But if it's inevitably secure, it's kind of like fate. It's inevitable it will be secure. And I think that's a really high bar.

Yes.

Well, speaking of better fool, I had to go through quite some VP exceptions to run a honeypot on Google Cloud. So I am familiar with your process, Seth. I suffered through it.

I mean, to be fair, running like a damn vulnerable web app or a honeypot certainly feels like the kind of thing that should be by exception only.

Yeah, probably. Okay, so I hate to do it, especially on the note of running honeypots, but I have to ask our traditional closing questions. One, do you have a tip other than this amazing threat modeling-based, hierarchical-based, org policy-based, insecure by exception tip for people? And then 2, do you have recommended reading? And that reading could be anything from Mary Oliver to The Art of War to not Anton's blog.

That last one was oddly specific. So I think As far as a tip goes, oh boy, this is hard. I would say that the intersection of security and large language models is an interesting space and any reading that you can find. I asked Gemini, which is our agent internally, hey, what is the plaintext value of this MD5 hash?

And you know what?

It gave me the right answer.

No.

Yeah.

What? Really?

I mean, it's just a giant reverse rainbow table. So I think tips and tricks are Really investigate how, if you're a security practitioner or security adjacent, how can tools like Gemini and Vertex AI accelerate your security journey? You've got hundreds of thousands of log lines. Can you do things like anomaly detection? You've got things like very glorified regular expression matching. So that would be my tip is just start playing. We've got the AI Studio playground that I think anybody can just click a button and get access to Gemini. That seems like a great place to really experiment and see what Gemini can do for security.

Oh, you know what? I just asked Gemini to give me an MD5 hash for something. It gave me one, and then I asked a separate Gemini session, hey, here's this hash, what created it? And it gave me not what the original one was. And now I don't know whether it was lying the first time or the second time or both.

That's what makes it fun.

I didn't say to trust it. I just said to experiment.

Use it.

Yeah. Just to be super clear. And then in terms of reading, I mean, besides the de facto standards of my blog, you should—

Nice.

Yeah. See what I did there? You should definitely be reading the Attention Is All You Need white paper. It's actually fantastic. It's a bit old, but I've been rereading it, especially with a lot of the buzz and explosion around LLMs, and it's still super relevant. If you're not a white paper person, it can be a little bit dry, but it's really a great paper.

But you're not suggesting we ask Gemini to summarise it though, right?

I am not not suggesting that you could ask Gemini to summarise it.

That would be wonderfully self-referential. Seth, with that loop of reference, I have to say thank you for joining us today.

Yeah, thank you for having me. Super excited to be here and be back on the show.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at @antonchuvakin and @davidmalan.

Anton_Chuvakin and _TimPico.

Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.