Scaling Google Kubernetes Engine Security

Security. Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Tim Peacock, a product manager for threat detection here in Google Cloud, and Anton Chuvakin, a reformed and tamed analyst and a member of the cloud security team here at Google. You can find this podcast wherever podcasts are distributed and at our website, which we are just so very close to launching someday. In the meantime and into the future, you can follow us on Twitter, twitter.com/cloudsecpodcast. Our guest today, couldn't be more excited. We've got Greg Castle, the Anthos and GKE Security Tech Lead here at Google Cloud. And so today we're going to be talking about Kubernetes, securing your GKE environment, some of the interesting investments we're making behind the scenes, And all things there. So with that, Greg, we know that not everybody knows how to Kuber-a-net. So could you give the security practitioners who've been living under rocks for the past half a decade what the quick overview of Kubernetes and GKE in particular are?

So Google has really kind of been doing the containerized thing for quite a long time. Most of the sort of outside world was shipping dependencies and shipping VMs and installing a whole bunch of software and kind of shipping a VM to make it production. system work. And inside of Google, we've been using a containerized approach for a long time. And the idea there was you have just your dependencies, you get a fast start time, you can pack a whole ton of sort of diverse workloads onto a virtual machine or a physical host, and you get to explode sort of these monoliths of software into lots of little microservices that can execute independently, they can release independently, the teams can develop independently. And I think for a lot of people, That's a big shift from, like, going between those 2 worlds. So, we're seeing people who are used to shipping software on VMs, where all the dependencies had to be installed for every possible piece of software that would run on that VM, now start to think about containers and microservices and trying to get this agility out of that new environment. And it's sort of like a steam to electricity shift. It's a big change. And so, if you're kind of sold on containers as an idea and a thing you want to do, Once you've got a whole bunch of containers, you need something to help organize them. So like, how do you know which containers are going to run where? Like, how is their traffic going to get routed? How does all your storage attach? How do you replicate for availability? All of those kinds of problems. And that means you need an orchestrator to help you with all of those new container microservices that you've now built. And so that's where Kubernetes comes in. It gives us the basic structure for that orchestration. And it gives us a whole ton of sort of pluggability and extensibility to plug it into a modern production environment. So, all those touchpoints around networking and storage and those kinds of things plug into Kubernetes really well. Well, it does expect you to provide all that stuff. So, like the production network, the storage, the logging and monitoring, you bring the container runtime, the OS to run it on. And that's basically where GKE comes in. So, GKE brings all of that stuff for you. So, Kubernetes gives you sort of this blueprint for how a factory could look if you were going to build one. And then GKE brings all of that stuff. GKE actually builds you a factory. It's fully functioning. It's ready to sort of pump out widgets from day one.

Nice. So which aspects are special with GKE? What did we do different there from a security standpoint?

If you think about sort of traditional host security and sort of the new world of container security, there's a whole lot of new layers that we need to care about. So there's namespace separation in the kernel.

What is a namespace?

A few different things here. So like there's the sort of container concept of namespace where You're sort of like isolating things into boxes on a host using the Linux constructs. And then there's the Kubernetes concept of a namespace. And so the difference there is the Kubernetes namespace is kind of a group of things in the API. So I'll give you the security part of the sort of like the namespace answer for Kubernetes. So if you want to write sort of network rules about who can communicate to who, then you could put, say, like all of your frontend stuff in one Kubernetes namespace, and all of your backend stuff in another Kubernetes namespace. And then you could write rules about how those 2 things can talk to each other using something like network policy. You can also use Kubernetes namespaces to limit the resources that are used by pods. So you can have these resource quotas around CPU and memory and storage and that kind of thing. And that can sort of put a box around like how much of the underlying VM and the underlying infrastructure resources that this thing is allowed to use. And then there's also sort of an RBAC angle of this, resource-based access control. This is the authorization layer that's built into Kubernetes. And so namespaces help you there. And you can say, hey, Greg is allowed to get pods in the Gmail frontend namespace, but he isn't allowed to access the Gmail backend namespace, for example. Sort of 2 different types of namespaces there.

That was useful actually to clarify, because again, some of the on-prem security professionals who come with background of securing OS and maybe VMs, I've seen a fair bit of people confused And of course, when they come to Kubernetes and container security, they start saying, where's the host? Where do I put the agent? If I run on the host, what do I see? So they kind of start with their on-premise OS server mindset and maybe a bit of a VM veneer. And then they come and start asking questions, which I think container sec people start frowning and cringing and making faces. If you are encountering this type of situation, like, how do you typically explain the key meaningful differences between Kubernetes and container security from traditional host security? Because I can imagine an orchestrator doing things to Windows security or Linux security. So how do you get to the sort of DNA or the essence of a difference?

The OS is just responsible for a lot less is one of the things. It just needs to run containers. And so if you come from this background of like, well, I'm shipping all of my stuff in the OS, I've got every dependency that all of my production code needs to run, I've got all that stuff installed. And then I have this sort of big, fully featured OS that needs to run all of that code. It really gets really simplified down quite a lot to just being able to run containers. There's a lot of good security opportunities. And so we took advantage of a lot of that when we were building the container-optimized OS. So we did a whole lot of hardening and a lot of minimal construction of how that OS works. So it's just enough to run the container. And then another thing that's quite a big difference that can be a little hard to wrap your head around too is that when you're thinking about multi-tenancy, it's a very natural thing for security folks and for people who've been running production infrastructure for a long time to just Saying, okay, well, if I can just group by VMs, if I can save my high trusted things on these VMs and my low trusted things on those VMs in the same cluster, then I have like this VM separation that I'm sort of like used to. And it actually doesn't work that well inside of Kubernetes having just that VM separation. It's still a great boundary between sort of like the host that is running the virtual machine. But you've got a lot of sort of very privileged agents like the kubelet, like a lot of networking and monitoring agents that run. inside those host VMs that interact with the API server and the Kubernetes control plane. So it's actually less isolated than you may have thought.

Hmm.

If those VMs are inside the same cluster. Tim Warclair and I actually did a whole KubeCon presentation on this. You can be surprised in this space if you're sort of expecting to isolate things by virtual machines. And so one of the shifts there that we're trying to encourage is thinking about pod isolation. And so a pod is sort of the unit of scheduling inside of Kubernetes where you can run a container. And it makes a lot more sense to have the isolation at that level. And so what we've done for GKE is taken advantage, again, of some internal technology called gVisor that gives us this sort of strong sandbox that we can wrap around a pod that completely reimplements the Linux kernel and protects you from a lot of sort of container breakout vulnerabilities and gives you like that stronger isolation mechanism. So you can run basically trusted and untrusted on the same VM if you put them in these gVisor sandboxes. It gives you a much stronger boundary there. So I think those things are other new things. So the OS needs to be a lot smaller than you're used to, and there's a lot of hardening that can be done there. And then the VM and the multi-tenancy part of it is quite different. And actually one more, host firewalls are actually quite different too. So we're used to being, hey, VMA can talk to VMB, and the services between those things, we understand what they are and we write rules about the IPs from a VM A and VM B and what's allowed to access what. The Kubernetes scheduling doesn't really kind of care so much about VMs unless you explicitly tell it to. And so these containers are going to get scheduled all over the place in different places. And so there's a sort of a higher layer of network policy that's built into Kubernetes that makes more sense to use than host files in most cases.

So that departs from server A talking to server B pretty far, it sounds like. We're not like Hey, this is kind of server A to server B, but slightly different. No, no, we depart from that land kind of almost altogether when you're describing this fabric, right?

Yeah. And you can go even further up the stack. That's the Istio layer, the service mesh kind of layer. So if you want to think about sort of, this is the way Google internally does it. If you think purely about service A talking to service B, rather than like the infrastructure layer on which those things are sitting, if you're trying to reason about like, Is this service allowed to connect to this other service? It's a really natural way to be able to reason about authorization, and that's why we like it at Google. And so again, Istio is sort of like the open sourcing of technology we've been using inside Google for a long time, where it's a lot easier to reason about who can talk to who at the service layer than it is about which VM IP can talk to which other VM IP, because you have to know exactly what's scheduled on there. So you can do the host firewalling kind of equivalent. You can do that at the network policy layer. So you could have some sort of broad rules at the network policy layer, and then you can have really targeted rules per service to do your sort of network isolation.

That makes a ton of sense. So going back up maybe a couple layers of detail, what would you say are the key differences that a practitioner should keep in mind about securing something on GKE versus securing an on-prem Kubernetes deployment? Surely gVisor comes into that.

There are some differences in terms of our product offering. Some things are available on GKE that aren't available sort of on the newer platforms, but also like it's a pretty different beast. So when you're talking about on-prem workloads, you're talking about a lot of integration with existing systems. So you've probably got existing web application firewalls, you've got existing storage solutions, you've got secret storage, you might have like HSMs. So all of those integrations are built into GKE cloud using our sort of cloud-native solutions for those things, the cloud storage, the Cloud KMS, the Google Networking Fabric. But when we're talking about on-prem, all of those things have to be integrated with the on-prem environment. So, you know, we're looking at, you know, integrations with HSMs for storing secrets and similar, like on-prem network fabric, on-prem storage. And so it's really those integration points that are quite different when you're on-prem.

So with the hardening that we've done, where did we start hardening and where did we go? How did our hardening of GKE evolve?

The overall vision here is that it's really a move-in-ready security. Let me give you an analogy that's a real estate analogy. So, like, if you move into a house and it's a fixer-upper, you have a ton of things to do. Like, you have to renovate the kitchen, you have to replace the flooring, maybe you want new curtains. You have, like, this big to-do list of things. And so the idea is with GKE that we just eliminate that to-do list. It's move-in-ready. You don't have to do a bunch of renovations. And we do all the hardening. We write all the security policies you need. And you just manage the exceptions and you look at the alerts. That's sort of like where we're aiming and where we're trying to get to. So you really shouldn't need this very large security team handling just like the basics of the platform. You can have that security team focus on much more targeted work, and the basics are all taken care of for you automatically. There's a lot of basics. So there's the OS. So I talked a lot about container-optimized OS. So there's the hardening of the OS, there's the constant patching and maintenance of the OS that is really quite a lot of work. There's the Kubernetes configuration. If you look at some of the news stories about how Kubernetes clusters have been compromised, it's typically really basic mistakes like putting the Docker socket open on the internet or putting the kubelet unauthenticated port out on the internet available for anyone.

But these sound like pre-cloud mistakes with the OS. So this to me is an interesting observation, which would eventually lead us to the next question about threats. This sounds like very 1990s problems.

We've built with Kubernetes these complicated container orchestrators, and a lot of people are fairly new to securing them. And there's a lot of options inside of Kubernetes. And so, it really does take some expertise in Kubernetes to get that Kubernetes configuration right. And there is expertise there, and taking it as is and expecting to be able to sort of bootstrap a team on this large system is really a pretty tough ask. Actually, just at the Kubernetes layer, if you think about things like cert issuance and rotation for the kubelet and the API server, the TLS TLS key provisioning, all of that stuff is kind of like exercise for the reader inside of Kubernetes. And it's part of like the basics of like what you need to do. We really tried to, in GKE, do as much of that stuff automatically by default. And we've taken it sort of even further with Autopilot. And the idea there was to like take away even more knobs than GKE has, and really like make it as secure by default as possible, give you like GKE's very opinionated view of how that cluster should run, including from a security perspective, but also from many other perspectives around manageability, like auto scaling and auto upgrades and that kind of thing.

Which reminds me of what we started talking about very briefly, as far as users' mistakes and other stuff. Like, in your opinion, when we do this security, when we do the hygiene work, the background work, like, what are the realistic threats we have in mind? Like, what's the threat model that we were assuming? So maybe it's like I'm trying to ask 2 questions at once, not only the threat model, but what are the realistic threats to container deployments that we considered Misconfiguration is, I think, the biggest, most obvious threat.

There's still dangerous options inside of Kubernetes, and it's very easy to turn those on if you're trying to debug something and just trying to make something work. Like maybe you're debugging an identity integration and you turn on basic auth. You put a weak password in there and then you forget about it later. It's like really easy to do something like that.

So the threat is not that. Threat is somebody finding and exploiting that, presumably, right? That's what you mean. Like it's easy to find then?

Right, exactly. Being able to take advantage of those misconfigurations is scanning that stuff from the internet, looking for those open Docker sockets, looking for those open Kubernetes kubelet exposed ports is how we're seeing the real-life compromises happen today. So yeah, I think misconfiguration is a big one. Patching is also quite a difficult thing to keep up with Kubernetes, and vulnerabilities get fixed all the time, and it releases quarterly. And so keeping up with those releases and keeping up with those patches is actually quite a lot of work. If we think about those things in terms of outside scanning of the cluster, and then we think about the workloads themselves, we basically assume that any workload that's running on the cluster can be compromised at any point. These are big, complicated production systems. They have a lot of diverse workloads running on them. If you're building a cluster, you need to assume that that pod or that container might have malicious code running in it at some point. Then that gets us to these questions about sandboxing, but also about pushing forward the hardening of Kubernetes itself, and also NGINX sort of doing additional hardening. But if you get execution on a host, what can you do with that? If you're root on that host, can you get all the secrets for the entire system, or can you only get the secrets for the subset of workloads that are actually scheduled on that node? And so there's been a lot of work inside of Kubernetes to sort of push the hardening of Kubernetes forward to limit the impact of a single workload compromise too.

That's the first one there that really strikes me as super novel. You know, exposed ports, bad passwords, that's old school stuff we've been doing for a long time, but compromise a workload and get access to secrets, that seems interesting and new, and there's lots of cool opportunity there. Sure.

So like, frankly, now I'm a little bit shy to ask the last question because initially when I read up on this, and again, I don't have that much depth in container security, I sort of assumed you made it more usable and more secure at the same time. And that was my impression, but you were talking about complexity so much that now I'm I'm nervous and I'm thinking, did we manage to make container Kubernetes security a more usable system for developers and also more secure?

We've been talking about like challenges a lot. We haven't talked a whole lot about advantages. There are some like really interesting and unique things that come with containers. So you get to be able to really minimize what you have inside the container. We've had a lot of success with the containers we run, moving them to these distroless static containers that Google maintains. And so what we've got is Kubernetes and GKE is basically a whole lot of Golang code. And so all we really need a container to do in most cases is run some Go code. And so we don't need a shell, we don't need a full OS. And so we have like these really minimal environments that both make it way easier to patch, but also make it super hostile to an attacker trying to execute in that environment. If you look at sort of most exploitation paths, so not having a shell and not having an OS, not having a package manager, just makes it way more difficult to be able to operate inside that environment. You've really got to tailor your exploit to that environment. And so we have this opportunity where, especially for compiled languages, to get to these really minimal install environments where we expect them to get compromised in the threat model sense, but the environment that those attackers are then operating in with their shellcode and trying to get execution and establish a foothold is just so hostile that it's actually quite difficult. There's that kind of advantage with minimal containers. There's also a lot of the workloads we run are stateless, and Kubernetes and the design of the system basically expects these to come and go fairly frequently. So we don't expect these to run forever. And so we have the option to kill them regularly, for example. So I know some customers have been exploring, hey, let's just kill workloads regularly. If people do establish footholds in them, they're going to have to really automate and work hard to operate inside that kind of environment. Those are some of the advantages, just containers and orchestration that we get that offsets, I think, some of the complexity.

Yeah.

But I think also too, for developers, if you can secure that basic infrastructure, secure that Kubernetes infrastructure, and that's these problems that I've been discussing at the infrastructure layer, if you've got some expertise to do that, then the developer's life is being fairly drastically simplified. They only have to care about exactly what's in their container, They only have to build it with the trusted CI and CD system. And then a whole lot of security processes kind of happen automatically. And compare that to the older world where maybe they had to worry about which IP addresses on which VMs they need to write network policy about. They can now write network policy about their service instead, assuming they're using a service mesh. So, I think, like, some things have, like, definitely gotten easier on the developer side through the advent of containers. There's a lot of security advantages there.

Hmm. Well, with that, we are very much at time. Greg, thank you so much for joining us today. You called in from Australia today, right?

That's right. Yeah.

I think that makes this our first transcontinental podcast episode, which is super fun. You can find this podcast on Google Podcasts or wherever you get your podcasts. You can follow us all on Twitter. The show is twitter.com/cloudsecpodcast. My co-host Anton is @anton_chuvakin. I'm @TimPeacock.

I'm Mr. Gcastle, M-R-G-C-A-S-T-L-E.

All right, Mr. Gcastle. With that, listeners, please tweet at us, email, argue with us. If we like or hate what we hear, we can invite you on the next episode, which I promise is more fun than arguing with people on the internet. And with that, we will see you next time on the Cloud Security Podcast.

Security. Security.