EP106 Beyond BeyondProd - How Do You Zero Trust Your Workloads?

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you like our content and want it delivered to you piping hot every Monday or whenever we have a special episode, please do hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, this is another one in our long list of tragedy songs. I mean, Zero Trust episodes. We're talking about Zero Trust's sort of not evil twin, but workload twin today, right?

Yeah. Yes, correct. And of course, we're going to annoy our guest with the question about, but how do you define Zero Trust?

Yeah.

Because I think we swore to ourselves that we're going to ask every guest who promises ZT something to ask him, but what do you mean by Zero Trust?

Do you think that means guests will stop coming on to talk about Zero Trust, or do you think there are just so many people out there so motivated about it, they'll come no matter what?

I think it's the latter.

Yeah, I think you're right.

I think that people are motivated to say something about Zero Trust. And of course, our guest today would say more than something. It'll be fun.

No, he's got a good answer, actually. I think we both appreciated his answer.

Yeah, exactly.

This is a fun episode, listeners. If you remember a few episodes ago, we had the truly peerless Sandra on the show to talk about binary authorization. This is a bit of the look at the internal version of binary authorization. And so perhaps with that, and without giving any other spoilers about today's episode away, let's turn things over to today's guest.

And our guest today is Anoush Sabouri, a group product manager at Google Cloud. Welcome to the show, Anoush.

Thanks for having me here.

And we have some good and bad news here. This is another Zero Trust episode, and we've done a few, and we prefer to start these episodes from asking the guest for their definition of Zero Trust. When you say Zero Trust, what do you mean, Anoush?

And this is great because someday we're gonna edit all these different answers together, listeners, and you can look for our special just definitions of Zero Trust.

That sounds great. Yeah, let me start by talking a little bit about the previous model before Zero Trust, the traditional perimeter-based security model, and why it fell short of the promise, right? So what used to be the case was that customers used to group things around a physical boundary, perhaps a building or a network, and trust everything that ran inside that, right? And that model fell short with a modern workforce where the users just start working from outside those perimeters, maybe from a coffee shop. And later on with pandemic, this actually got exaggerated with people working from home, right? Similarly with the applications, it used to be the model that an application would be a monolithic application running in a, on a dedicated hardware with a fixed IP address. But then with the move to cloud, those applications start getting distributed and then you cannot guarantee where the application is running. So having a physical boundary as your trust boundary becomes infeasible. And that's where the idea of zero trust came into play, right? The idea was that you should assume that your system consists of multiple components and there's no inherent trust in any part of that component, and that you have to continuously verify the trust in each of those subcomponents, right? And I can say like, that's the kind of core idea of zero trust, which means trust nothing. And then it could be applied to anything including users, devices, or a workload. And if I were to summarize, in my opinion, Zero Trust consists of 3 pillars, right? The first core part of it is having an understanding of strong identity versus the traditional identity based on IP, right? Which used to be the case where your machines were hosted on a specific IP address. And the second one is that dynamic context. This is important because, as I mentioned, you have to continuously verify the access. So for example, the time of the day, is it business hours or not? The type of device you're coming from, is it a managed device or not? The strength of the credentials. These are important part of your access verification. So context is very important. And lastly, in my opinion, the third pillar of Zero Trust is this centralized choke point that allows you to have consistent policy enforcement. So to sum it up, Zero Trust is nothing but to always verify the trust in the system and the components of the system.

Didn't you have a fight with— well, a minor fight with John Kindervag who thinks of himself as an inventor of zero trust, about trust nothing, right? I kind of assumed you have to trust the zero trust system, but he convinced me correctly, I guess now, that you really do trust nothing and you validate pretty much every component. So when you say trust nothing, Anuj, do you really mean nothing? No piece of software is trusted in this system?

Well, trust starts from somewhere, right? It starts from a piece of hardware that you have securely provisioned in the system. And then based on that, the trust is transferred to the other part of the system. And so there's always a root of trust in the system, but the idea is that you own that root of trust and that you don't rely on something that didn't come outside your control.

Okay, so here Anoush, I think, did get to what I think is a canonically Google-y answer of like, we'll trust some piece of hardware that we have thoroughly baked and tested and made sure we trust that thing, which in prod land is usually a Titan chip, right?

Correct, yes.

Yeah. So I still don't like the term zero trust. I still think it's really goofy that we've got this world where we're like, ah, trust nothing. Trust this zero, trust that zero, trust that zero. Zero plus zero plus zero equals trust. Because the summation doesn't make sense to me, how you get trust from zero trust. So the answer of, well, you get trust from that piece of hardware, that starts to make some sense to me.

But I think Anousheh's take with 3 pillars does make sense. And if somebody wakes you up at 3:00 AM, Tim, and says, what's zero trust? And you say strong identity, dynamic context, central policy enforcement. That would be clear. It would not be confused with just MFA. I've met people who confuse zero trust with MFA.

That's fair.

I've met people who confuse zero trust with not having a firewall or VPN, which is frankly kind of dumb, right? Well, VPN replacement is a use case, but it's not the same as zero trust. So, I think the 3 pillars is a good answer. So, I think let's keep the 3 pillars. For now. But we're not here to talk zero trust, right? What are we here to talk about?

No, we're here to talk about beyond prod. So, we're going to shift gears, having first established a baseline of zero trust, which we think we can trust. Let's talk about beyond prod because we talk about beyond corp a lot. What's beyond prod?

Sounds great. And before we move on, let me just add one more discussion point about the previous topic. The root of trust that I mentioned in terms of what Google uses is also something that can also be expanded to the customer to verify, right? And so the ability to get guaranteed that the type of hardware is trustworthy is also part of the whole principle, right? Being able to verify the claim of the cloud provider. But let's come back to your question about Beyond Prod and how it relates to Beyond Corp. I think for me to give an answer on that, again, I want to go back a bit in history and tell a story on how Google came up with the idea. Back in 2011, Google started looking at how we can simplify the story for our employee to access our core resources in an untrustworthy network, right? And it was motivated by potential attacks against our own infrastructure, but also had productivity in mind, right? And so we built this solution that basically removed the traditional network-based perimeter around access policies into really a context-based policy. And later on in 2014, we published a paper called Beyond Corp that summarized our VPN-less access for users to our resources. Later on in 2017, we published a companion paper called Beyond Prod that basically extended the ideas of Beyond Corp paper into workload and production systems that are cloud-native, right? So, if I were to sum it up, you can think of Beyond Prod as this version of Beyond Corp that applies to workload versus Beyond Corp, which applies to users and devices. So, that's basically what it is.

So, is it like a BeyondCorp for workloads where there are no human users, but basically machines talk to machines. It's a kind of BeyondCorp for robots. No, that's too glamorous and not exactly accurate.

But I mean, in a way, what you said is there's truth in it. You can think of it as exactly expanding those concepts into workloads. You're absolutely right on that.

Okay. So we've brought up the word workload. What's a workload?

Maybe I can give you an example. Think of an application that does a certain task. Think of it as, let's say, a shopping application that you want to use to find products and be able to pay for them and all that. This application contains or consists of multiple services that are responsible for different things. For example, part of the application could be responsible for showing a product catalog. Part of it could be the one that does credit card payment. processing. These are services which are— you can think of it as network representation of different parts of the application. Each service itself is governed and served by multiple workloads, potentially a pod in Kubernetes or a VM in Compute Engine. You can think of the workload as the most coarse-grained part of an application that does a specific task. Every application will consist of multiple services, And each service will contain multiple workloads.

And while we are talking about this, I'm assuming this is not just made up by somebody and Google actually has something that's sort of running in production. Like, this is something we use, right? This is yet another product idea inspired by how we do things, right?

That is correct. And I can give you a context of, like, how we are thinking about doing this in Google, how, like, Google brings this idea to life, right? So, the way we think about Zero Trust for workloads and how we can bring those principles in play by having a thorough look at a binary lifecycle. and what it does through its lifecycle from initiation all the way to deployment and when access happens. So starting with the development part of it, one of the core things we look at is to make sure that the tools, the sources, the libraries that are used to build that application are trustworthy. And that we also enforce that and verify this before the application gets deployed through a product called binary authorization. So that's the core part of the trust is that to make sure that the code is, or the binary is trustworthy. And then the second one is once you have a trustworthy application is to make sure that it gets rolled out in a secure way. So things like dry launch, being able to stage the rollout and being able to roll back things that may not be right and ability to apply policy without downtime, right? And then once you have that, the next thing is to make sure that it gets deployed into trustworthy hardware, right? And this is where we have initial conversation where you start by looking at a machine that has a root of trust in it and is guaranteed to be hosted hosted in a Google data center, right? And so the trust starts with a piece of hardware that is deployed in that machine, and then that trust is transferred into the upper part of the stack. So the trust in the hardware translates the trust into kernel, and that translates to trust in the operating system. And so once you have that foundation that this operating system is trustworthy, then you can put your trusted binary on top, right? And now at the end of this, you have a piece of code that you can trust that is running on the Google hardware, running in a Google data center with the provenance check, right? And next, once you have that, then we start establishing trust between the workloads, right? This is where the strong identity comes in, where you have a version of TLS, a mutual TLS between these 2 workloads to make sure they're authenticated and all the communication are encrypted.

Hold on. What does strong identity look like for a workload? Because surely the Borg job can't touch their Nubby. So what does that look like there?

That is an absolutely great question. So the way to think about this, and there is a standard version of doing this these days in public, is it kind of a very similar to DNS. Format, but using a different notation. So basically we group workloads as part of what we call a trust domain, which serves a specific type of purpose for that set of workloads. And then from within the trust domain, you have sub-organization like namespaces, which allows you to say these kinds of workloads can talk to each other. So you come up with a grouping of how to map the workloads into certain tasks that they do, and they're only trusted to do those types of things. And so this is created as part of the workload registration or creation. And then that maps into an identity format that our identity service can issue a credential for these workloads, technically reflected through a certificate, but then they carry that identity with them through that certificate as a credential. Now the certificate is tied to the device that they run on. And so that gives us the kind of the almost like a strong credential format that we discussed and allows them to use this certificate to create that mutual TLS. So the identity that is based on that grouping is reflected into a strong credential that is later on used to mutually authenticate these 2 things together. Does that make sense?

There's a lot of layers of abstraction in this.

Yes. Yes, for sure. Yes.

I was about to say something very similar, but Tim said it better, frankly. So I'm going to grab a crayon and like try to draw it with crayons. So there are workloads, right?

Yeah, there are workloads.

And we use them for like stuff. So, we need workloads to know that other workloads are who they say they are?

That's correct, yes.

Okay. So, Tim is— you don't see it, audience, but Tim is making a really pained laughing face.

Audience, I'm at a loss for words at Anton's simplification of there are workloads and they need to talk and know who other workloads are. I'm just dying here. Like, I can't, I can't. Anoush, Please help, please.

Yeah. So you can think of it as, imagine there is a registration for these workloads that allows you to know for a given service that they run under what namespace they can acquire. Right. And so that's part of, as part of the workload creation, that is part of the enrollment process. Right. And so they can discover what other workloads exist that can serve their purpose. Right. And as part of that discovery, they can also find a way to also know who is the credential issuers for them and how to trust them. There's a centralized service that allows this workload to know what identity they belong to, and also more importantly, who can they trust and talk to. That centralized service is a way to look up other services and be able to understand who they can talk to. That's one of the core benefits of this centralized way, is that ability to govern these kind of communication and create policy between what workload can talk to what other workload in that manner. For example, imagine you have your frontend web server that wants to talk to backend that does credit card processing. You can go to this centralized way and express those policies saying, for example, workloads under this namespace belong to my prod environment that do frontend web server. Workloads in this namespace are the ones that are doing my backend credit card processing. You can author a policy that says this frontend web server can talk to the backend under these circumstances, to basically create an order.

But can I not just be stupid and do it by IP address?

Well, so the problem is, what is the IP address? You don't have that, right? With the cloud, the workload can get deployed in any environment and you don't know, right? The whole point of elastic cloud is that your workload can end up anywhere with any IP address that are not exposed to you, right? And that's why the identity is important because it roams with the workload. It doesn't tie to any specific endpoint. Right.

But we are achieving much more here than just a new type of an identifier, as I understand this machinery, right? Like when I read the BeyondProd papers, it's very clear to me that there are security benefits on top of just reliable system of identifying, well, workloads that will do stuff.

Exactly. Yeah. So for the workload to be able to declare that identity, it's not that it can show some static secret, right? It has to prove everything that we discussed, that it's running on trust hardware, that it's basically still a trusted binary, right? So all that is reflected into by the time the credential issued is part of that. And so that's the difference, the main difference with this mentality, the thing we discussed, continuously verify. Just because you acquired this identity and credential a few days ago doesn't mean that you can represent the same contents and get the same identity. It has to again be verified. And so you can think of the workload roaming across different hardware at all times But for it, it can assume the same identity, but it has to go through the same checks and verification again, right? And so I think that's the core difference with this model is that decoupling from what IP address it runs on, if it makes sense.

Okay. So I want us to shift gears a little bit and I want us to talk about what is this good for? What threat models are we addressing here with this technology? Because I kind of get the threat models we addressed with zero trust. You address the threat model of not paying more than you were paying your VPN vendor, things like that. But what threat model are we dealing with here?

Maybe I can start with going back to the shopping app example. Let me motivate the problem slightly better, right? The shopping app example, if you remember, had, like, 3 different components, right? A product catalog, credit card payment processing, and potentially a login page, right? And each of them may be talking to different backends, right? So, your product catalog can be talking to Bigtable for the list of products, your credit card payment has to talk to a bank, and your login has to talk to IDP. And each of these components can be running in different compute environments, right? So when you look at the matrix of the components talking to each other over network, your access paths are exponentially growing, right? So as you can see that, I mean, when the access paths are exponentially growing, the number of access policies you have to define is also growing, right? And being able to consistently apply this policy across different environments becomes harder and harder. And if you cannot catch up with them, that means that you have data exfiltration problem. And so one of the main things that this solves is that consistent policy enforcement that reduces the data exfiltration problem.

And the other problem, presumably also workload impersonation, theft of resources. I don't think— to me, it doesn't sound like exfil is your first problem, but I think we can argue about that later. But if I want to steal compute to do my coin mining or my other stuff, or if I need resources to encrypt your data so I can ransom it, Absolutely. Perhaps there's more of the resource theft and of course the exfil and other stuff, right?

You're absolutely right that it comes with all of the benefits like you mentioned. That's basically what strong credentials gives you. The kind of my response was more from what customers tell us, like when they look at concepts like this, that's the top of the mind, but you're absolutely right, it has all other benefits as well.

Probably exfil because this system allows encryption between workloads to be a lot more reliable, a lot more logically clear, and a lot more transparent. Like what you're doing is encrypt this to this, as opposed to like some kind of point-to-point system that would be tricky. So I think it becomes much more clear once we discussed it. But here's the— so BeyondCorp and Zero Trust original idea dates back to 2010. BeyondProd, the original BeyondProd papers were published by Google last year, late 2020.

Okay.

We'll find the link. But the point is, why is now the right time? Why is this a now problem?

Why now?

in 2010, nobody thought about, hey, zero trust for workloads, what a great idea.

The paper was published actually in 2017. But yeah, in reality, what happens was that the move to cloud has expedited this idea, right? From a few minutes ago when I discussed like that shopping app, as the app gets modernized, right, it gets exponentially more complex to do access policy checks, right? And so that has been one of the main drivers for reconsidering this. And also new compliance regimes, right? It's making it harder for a CISO to have assurance that they're still complying with these ever-changing regimes that keep coming their way, and they want to have a consolidated story that allows them to get these complaints out of the way. So I would say between these two are the top drivers for reconsidering Zero Trust and beyond product for these systems.

I don't know, I think this is all pretty cool, but my question that I want to close on before our actual closing questions is, like, if I'm a company and I've got a jillion other things going on, How do I think about slotting this kind of work in with everything else I might be doing, like rolling out YubiKeys to my team or investing in a more modern vulnerability management framework or migrating to cloud?

This is a great question. And the way we see this is actually part of the journey to modernizing the application, right? So I think the prime time to consider something like this is as the customers start thinking about how do I move into things like containers, microservices, and, you know, how do I deal with my security architect over there. Solutions like this are a great fit to have a cloud-native security architect for those applications. But the beauty of it is it doesn't stop there. It actually expands to traditional lift-and-shift application, even on-prem application too. There is a really gradual approach that one can take to deploy this across the environments. Honestly, these are very complementary to other stuff that you just mentioned. You can also think of it as defense in depth. Meaning that if you roll out YubiKey for your users, right, you want to think about, okay, so I solve and secure my user access to the application, but there is delegation happening in the chain of calls, right? So like, how do I transfer that trust from my user authentication to the application down to the multiple chain calls it has to make to the backend services? And stuff like this would definitely help with that. And even the vulnerability analysis is also a great complement to this, right? Which could be combined with some of the centralized policy to say like access to a workload is only given when the vulnerability So, I think I see this very complementary to the other security initiative as the customer thinks about moving to cloud, in my opinion.

More complementary is scary to a lot of people who have to decide on the budget because if it doesn't replace anything, there's often no money for it. So, I don't want to call you on this, but I think I agree that it's complementary to many existing tools.

Anton, I don't want to call you on this, calls you on this.

Yeah. So I think the, the, the part that I want to call out, what does it replace? That's a great question, right? So what we've seen from the customers is that they are looking at modernizing their authorization system that were built with things like IP-based and network perimeter in place, right? And so they're looking at how they can put something together to do this, right? And the one thing that they come quickly to the realization is that doing this at scale is a very hard problem. So the kind of the work that this replaces is those in-house built solutions that tries to modernize their authorization system in a cloud-native environment. So there is a definite replacement of technology that it does. However, it's also from another concept that our team brought up around securing user access and vulnerability. It could be considered as complementary to those.

Okay. Now the whole thing makes more sense. And I think that it's legitimately sometimes for some people it replaces obviously confusing old authorization system that doesn't fit the modern realities. But for some people, it's a layer they needed, say, in a new environment. So given the time, I wanted to get to our favorite— well, actually, Tim's favorite closing question, namely resources, reading materials, maybe highlight one resource, and also give maybe one tip to people deploying this in production in their companies. And I think that I don't want to put words in your mouth, but some guests, for example, said, think of the— whether it fits your culture. And I don't know, just give us one tip for the audience who is deploying this and a resource or two.

Sounds good. As for the resource, you know, this week is Cloud Next. So we just had a session that talked about this with a demo. So I think that would be a great place to see some of the concepts we introduced here and how Google is thinking about doing this at scale. In terms of the tip, I would say, Don't build this in-house. The scale is a hard problem. And, you know, and more importantly, getting the security right, right? Google has done this for years. And what we're doing here is bringing what we've done in-house to market. And I think allowing hyperscalers doing this tend to be the right bet normally. In terms of the call to action, what to do, you know, though, obviously we are getting started on this. We want to get customer data and input. So, you know, you can reach out to us at beyondprod@google.com to get more information. And we want to hear your thoughts on, like, what problems you're dealing with and what kind of solution would you expect. So, that's my 3 things that you asked.

Perfect.

That does make sense.

And again, thank you for answering and thank you for being on the show.

Thanks for having me. Appreciate it.

And now we are at time. Thank you very much for listening and, of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, Or wherever else you get your podcasts. Also, you can find us at our website cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikian and TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.