EP60 Impersonating Service Accounts in GCP and Beyond: Cloud Security Is About IAM?

Security.

Hi there. Welcome to the Cloud Security Podcast by Google.

Thanks for joining us today.

Your hosts here are, as ever, myself, Timothy Peacock, the product manager for threat detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and much beloved member of the cloud security team here at Google. You can find and subscribe to this podcast wherever podcasts are available, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you like our content and want it delivered to you piping hot every Monday afternoon, please do hit that subscribe button on your podcasting app of choice. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, this is, I think, a super fun episode for a lot of reasons. One, it's one of the more in-the-weeds technical episodes we've done in quite some time.

That's correct. And also, there would be a couple moments when I interrupt the guest and say, wait a second, this sounds dangerous.

There is some honest-to-God danger in this episode, which is kind of fun. And I think this episode also features the easiest-to-understand explanation of how identity and permissioning work in Google Cloud.

Correct. And it has also a very fun argument about how to teach cloud to on-premise people. a particularly confusing area, identity in the cloud. So to me, this is also very fun.

It also, listeners, uh, you won't see this, but both Anton and I made very confused faces during this episode. So you'll hear us talk about that, but you won't get to see it, unfortunately. We'll have to get screenshots of that and put it on the Twitter, won't we? So perhaps with that, let's welcome today's guest.

I'm delighted to introduce today's guest, Dylan Arey, co-founder of the Truffle Security Co. Uh, Dylan, I've heard rumors that security in the cloud has a lot to do with identity. Could you tell us why identity is important for cloud security? What's going on there?

Yeah, no, it's a great question. You know, identity in a nutshell is what allows access to cloud services. So basically you have identities in the cloud and those identities get permissions or roles that facilitate who can do what, who can access what data, and who can run what compute resources. So all of that is kind of built into this idea of identity. You have both machine identities and user identities. And then on top of that, you have permissions that get layered on top. I think what's particularly interesting about Google is this idea that rather than identity being user-centric, like people traditionally think of it as like, what does a user have access to? Those types of questions actually aren't answerable in Google Cloud. Instead, identity is resource-centric. So who has access to this resource? And that took a little while for me to get my head around, but basically the cloud is built in such a way where an individual user or a machine identity can be given access to lots of different resources, but the owner of that machine identity can't necessarily see those grants. But the resource owner can always see all of the different identities that can access their resource. So as a database owner, you can say, who can access my database? But as a machine identity owner, you can't answer the question, what does my machine identity have access to? Instead, all the different resource owners out there have to be able to answer the question in summation for you. And so I think when it comes to Google Cloud in particular, that was the toughest barrier for me to mentally get my head around. But once that kind of clicked, everything else sort of fell into place from there.

Wait a second. So our guests aren't seeing the video, of course, because it's a podcast. But I am looking at my own picture, and I'm looking at Tim, and we both look kind of confused. I think I know what you said, Dylan. But imagine an on-premise security person who is maybe working in a SOC or maybe working just securing systems. How do you explain it, what you just said, to somebody who isn't really well-versed in the cloud? Is there any way to explain it to an on-premise person?

Yeah, that's a really good question. So in a traditional environment, you might have something like Active Directory that determines what identity is and what groups and permissions an individual identity has access to. For example, within Active Directory, you might put individuals into a group that allows them to have access to a particular computer network. And then within that computer network, you might have things like databases and things like that. So you may only want your database admins to fall into this particular AD group. And then you would easily be able to log into Active Directory and say, oh, well, these database users have access to this database group, which allows them to access this database. In Google Cloud, it's a little bit different. Instead, you have owners of databases. So everybody can kind of own their own database. And then you have each one of those owners have the ability to add users and identities and groups to those databases. So rather than being able to say, oh, let me look up Tim and see what databases Tim can access, Google Cloud actually doesn't have that ability like you might have in sort of the Active Directory world. Instead, you can say, well, let me look up this database that's the credit card number database and see who can access this database. And so we can get clear answers to who can access the database, but we can't get clear access to what are all the databases that Tim has access to. So it's very centric around the resource and not centric around the identity.

So it's easy to figure out who can touch a piece of data, but it's hard to figure out, given identity, what it could touch.

Yeah. So if you have a piece of data, you can figure out who can touch it. But you can't say, well, what data can Tim touch? That's a much harder question to answer.

Surely it's answerable though, right? Somehow?

Actually, by design, it's not answerable. And the reason why is all of the different data owners can grant you access to their data, even outside of your organization. So you can think of a large Google Cloud customer like Snap, for example, could arbitrarily say, I want to give Tim access to our production database. And the permission to be able to view that role binding is one that you wouldn't have by default. So you wouldn't even be able to see that the binding happened, but you would still be able to be given that permission. So when I say, what data does Tim have access to? I wouldn't necessarily know that Timo has this random permission over to be able to view the Snap database. And even if I have the ability to create roles and and one of these roles is one that I just created, I still don't have control over all of the permissions that get assigned to— or identity, I mean. If I have the ability to create a service account or an identity, I don't have the ability to see all the permissions that identity has been granted, because those grants can happen to resources that are outside of my control, outside of my organization. And I wouldn't have the permissions to even list or view those. And so that's kind of when I say it's very resource-centric. If you have a resource, you can see who can access that resource. But it's not identity-centric. So you can't say, what does this identity have access to? But you can say, who has access to this resource?

Got it. Well, my friends at Snapchat, please don't grant me permissions on any production databases. I don't want that. That's more trouble than it's worth.

Yeah. And I think that it makes sense. And I think that a little bit of a meta conclusion on this is that it's not like the system has more complexity. It's the system is different to what you're used to. If you are used to servers and boxes and, you know, getting a shovel and pushing VMs from place to place, like this system is just different. It's not better or worse. It's, to me, a little bit different. I think that's the summary. And let me briefly put a cynical hat. Tim, yes, I do have one of those. And the cynic in me says—

Only one?

Ah, okay. Let's avoid— I'll avoid the question. So, somebody who's more skeptical about cloud security once told us that in the cloud, we are one identity mistake away from a breach. And it's stuck in my head as kind of a line because Of course, if you make a mistake with identity, but nobody makes a mistake with firewalls, your database isn't exposed to the outside on-premise. But we are in the cloud. And if I make an identity mistake, maybe because I'm an on-premise dude who doesn't understand cloud, what will happen? Is it as bad as he said or not?

Well, that's a really good question. I think one of the big differences is that all of the APIs that you talk to are on the public internet. So if you do have, for example, a key that maybe accidentally gets posted in a place where you don't want it to be, and that key has access to certain resources, it is not the same as a password for an on-premise service leaking out. Because that password can't be used unless you're on the private network, but the key can still be used to talk to a public API. So I think what you mentioned before, it hits the nail right on the head, is it's not better or worse, but it's just, it's different. And it's different in ways that have both pros and cons. And it's really important that we understand all of those nuances. And so while it's true that you can definitely misconfigure firewalls, it's, it's also true that having all of the APIs on the public internet comes with its own set of risks. And we just have to learn the new language and learn to adapt to this new set of risks, which is different than the traditional on-prem access controls that were the norm 20 years ago.

That actually makes sense. And so we have a next thing we wanted to touch is, of course, the whole service accounts angle. So for somebody who isn't deep with GCP, what's the 30-second story on a service account? How is it regular from a normal IAM account? I understand there's no human behind it. That part I do get. But what about the rest? What's a service account?

Yeah. So let me define a couple of roles in GCP. So I mentioned the idea of a resource. Resources can can be collected together into groups of resources, and we call those projects in GCP. Now, to get access to resources or to projects, we grant 2 types of identities access to either a resource or a collection of resources. The first type of identity is a user. So that would be something like Tim. So Tim would be our user. Tim could also fall into a collection of users, which are called groups. And so on the equation, we have Tim or a group of Tims, and then we have a—

A horrifying thought.

And then we have a role binding. And on the other side of the equation, we have a resource or a collection of resources, which would be a project. So that, that works all well and fine if we need to find out how Tim interacts with the cloud. But a lot of times it's not a user interacting with the cloud, it's a machine. a machine identity. And that's where service accounts kind of enter the picture. If we want a laptop to be able to talk to the cloud, it could either use Tim's permissions, or we could create a machine identity that has its own set of permissions separate from Tim's permissions. And that allows us to create an identity that's scoped down to the specific use case that we're working on and doesn't necessarily have access to all of Tim's Google Drive and all the extra things that Tim has access to. And so we call that type of identity a service account. And those service accounts don't have to just run on a laptop. They could also be directly assigned to a compute resource in the cloud, like a Cloud Function or a VM or something like that. And the permissions for those Cloud Functions or VMs are contained within this thing called a service account, which are different than the overall set of permissions that a user or a group of users might have.

Got it. I found service account very confusing early on because I thought they could only belong to services, but of course they could belong to laptops, they could belong to a server, they could belong to a machine somewhere. So it's perhaps a confusing name for people that are new to cloud. And listeners, one of the other things I want to add before I ask the next question is I want to remind our listeners that Dylan is not a Googler. Dylan, actually, I don't think you've ever worked at Google Cloud. You're just a user who became the founder of a security company focused on this problem. And given all of that, this is still maybe the most cogent explanation of how our identity system works that I've heard in my 4 years of working here. So thank you for that.

I might correct something that you just said there. I did found a security company, but actually it has nothing to do with this problem. Oh, it's unrelated. But I did do a lot of research, particularly into Google Cloud as an independent security researcher with my co-presenter, Alison Donovan. We did quite a bit of research with the Google team, came on site a number of times presenting to a number of Google PMs and ended up doing the most viewed Black Hat talk in 2020, specifically on this subject, on GCP IAM.

Wow. So with all of that, I expect to have another incredibly cogent answer for this, and hopefully this becomes the most viewed podcast episode of 2022. What's a service account impersonation?

Yeah, I think this kind of leads itself into a lot of the content that we talked about at Black Hat. Basically, it's this idea that one service account or one user can actually, through cloud APIs, give itself the permissions of another service account. So if you, for example, have one of these set of 3 possible permissions, right? So remember when I said there's a— on the one side there's an identity, and then in the middle there's that connector, which is like a role or permissions. And then on this side, there's resources. Those resources— and this is where things can get a little bit tricky to understand— but a resource could be something like a database, but a resource could also be a service account. So a service account can actually live on both sides of this equation. It could both be something that gets permissions and something that receives permissions. So service account impersonation is what happens when we take an identity and we grant it a certain permission to a service account or a collection of service accounts that allows that identity to take on all of the permissions of the service account that we've attached it to. And there are 3 specific permissions that allow us to do this. There used to be more, and we'll dig into that in a second. But the 3 that exist today are the actAs permission. And what actAs allows you to do is it allows you to take a service account and attach it to a resource that you control. So if there's a service account out there that has, let's say, an owner permission, and I don't have owner permission, but I do have an act as role on that service account, I can take that service account, I can attach it to a VM that I control, and then that allows me to take on the owner permission that that service account had. So we can use this to elevate our permissions. We can start with something that doesn't have a lot of permissions and all of a sudden become owner if we have that. The second is called the token creator role. And what the token creator role allows us to do is it allows us to export a key, kind of like a password, an API key for a given service account, and then use that on behalf of the service account that we exported it from. So if there's that owner service account, we can take a password or an API key that can do everything that owner account can do, and then we can use that to do everything that the owner account can do. So we can elevate our permissions the same way with that key export.

Okay. So let me ask you this. It sounds kind of dangerous.

So—

Wait, there's a third piece of danger we should hear about, and then we'll talk about why it's dangerous.

Okay, fine. I'll shut up. Yeah.

What's the third piece of danger?

The third one is called workload identity user. And this actually came about when we introduced workload identity to things like GKE. Basically, the idea is that it used to be back in the old days, uh, when you use GKE, the nodes had service accounts attached to them, and all of the workloads that you ran would use the permissions of the node. That had a whole host of issues. So the people at Google Cloud created a better system where each workload running in GKE could have its own permissions. But in doing so, we also have this new permission that allows us to assume the identity of those different workloads. And we need that because our node needs to be able to spin up these workloads, but it does create that other path of And before we jump into your next question, there's the 4th one that I want to touch on. This one's near and dear to my heart, and this is the fact that there used to be a whole lot of permissions that allowed you to access an identity called the default identity, the default service account. And actually there's a couple of default service accounts. They all exist when you create a new project, they get spun up in your account, and by default, they have a lot of these dangerous permissions built into them. So it used to be, if I wanted— sorry, Tim, to keep picking on you, but I wanted Tim, for example, to be able to run a Dataproc job. I would grant him a Dataproc permission. And at its face, it would look like it couldn't do anything other than Dataproc things. But because that then allowed him to create a Dataproc compute resource, by default, that That default identity got attached to it, and that default identity has both token creator and act as applied to your entire project. And so this allowed Tim, when I just gave him Dataproc, to all of a sudden jump to be able to spin up a new resource and have that default identity attached to it, and then all of a sudden get access to all of the service accounts in the project. And when we pointed this out leading into our DEF CON talk, Google actually changed the rules. They sent out an email and they said, we recognize that this shouldn't be a thing. We shouldn't be able to go from Dataproc to editor to all the rest of the service accounts in the project. And so they sent out an email saying, this will no longer be the case. You will no longer get the default identity attached to your Dataproc, Dataflow, and Composer workloads. Instead, you'll need to attach a different identity to it. And those rules got changed. So now we don't have 4 issues we need to worry about in terms of impersonation. We only have those 3, which are act as, token creator, and workload identity user.

So there's a couple of cool things in there to pick on and drill into. First, what a great example of vulnerability disclosure, like, working out positively. Thank you on behalf of all our users and on behalf of Google for that. But 2, I want to pick a little bit on this distinction between act as and token creator, because with act as, the logs show that you're acting as the service account, right?

I think you might be flipped there. With token creator, If there's a, uh, I think, and the rules may have changed, by the way, because the team is constantly updating and improving and adding extra security to the platform. It's one of the nice things about working in the cloud is that auto-updates and is always constantly getting better. But when we were looking into it, token creator in particular, when you created short-term or long-term tokens, those got logged pretty well. Like you could usually—

Mm-hmm.

Call it 99% of cases, see when a token got created and you'd be able to use that to say, oh, well, Tim just made a token for this thing. There's a, you know, a potential exploit opportunity there, and that Tim is now maybe perhaps operating on behalf of the service account. But what was less transparent at the time was if Tim created a VM and attached a service account to that VM, Mm-hmm. that didn't, at the time, that didn't get logged super well. And because of that, that created an opportunity where an attacker could come in, take some identity, create a new VM, attach a service account to that new VM that just got created, and then start doing things on behalf of that service account. And they could just keep playing this game. So we open sourced a tool that basically took that to its extreme, where you would start with a service account that had act as on a project with a bunch of service accounts, and you would basically create one Cloud Function for every single service account in that project. And you'd have a, you know, a whole host of Cloud Functions. And then for all of those service accounts, you would say, do any of these have permissions into other projects? And if so, spin up Cloud Functions in those projects with all of the service accounts and all of those projects. And it recursively—

Oh no.

Recursively spidered out like that. And the, the only thing that would show up in the logs is just, oh, well, Tim's just making a bunch of Cloud Functions. It wouldn't say that he's attaching all these extra identities and adding more and more keys to his empire. And so the last time we looked into it, act-as was actually not quite as logged as we would've wanted it to be, but token creator had pretty good logging and attribution around it.

Got it. Well, that is alarming, and I don't like the idea of these spidering out service account Cloud Functions. That just makes me uncomfortable.

I'm sure it has some legitimate use, just like, you know, memory injections into live memory process on a desktop has legitimate use, right? Right? Right? Right? Okay. So on a more serious note, how do we detect this? Like you mentioned, I get the whole thing about logs, that are sort of unrelated that have been produced or vaguely related. So how can I see if my service accounts have been impersonated? What's the detection strategy? What are the rules? What are the approaches I can take? I know maybe it's not such a great question with a clear answer, but, like, where would I look, Dylan, to look for detections?

Yeah. So I think one of the things that we just touched on, the fact that token creation is pretty well logged, allows us to anchor on that in particular. And if we see all of a sudden an identity is creating a whole bunch of tokens whole bunch of service accounts. That's a pretty suspicious thing that we wouldn't normally necessarily see. So that's one thing to anchor around. Another thing to anchor around is that the thing that I mentioned, we open sourced the spidering. Uh, we intentionally wrote detection rules for that tool itself.

Hmm.

While it's not generic to all the behavior, at least if someone just out of the box runs that spidery Cloud Functiony thing, you can at least detect to see whether or not they're using that specific tool and that we included that in the repository and you can see what Stackdriver filter to add to detect on that. I think the number one thing that we tried to impress on the team is like, when it comes to logging, figuring out who's doing what, when, and when these service account impersonations are happening, like, that's so important. And we have seen some progress there, but I also think like we're not at the end state there yet. I think there's, there's more work to be done to improve our logs so that we can better get a handle on some of the act as permissions and things like that. And I think we'll expect to see that improve over the next year or two.

Yeah, that is one of the beauties of the cloud and the control plane of the cloud in particular is you can't not patch it. It just happens for you, which is great. So on the same sort of line of questioning as Anton's after is how do we secure our organization against this? Because despite being the threat detection guy, I actually believe that an ounce of prevention is worth a pound of cure.

Yeah, absolutely. And so I think that the most important thing for an individual organization maintainer that they could possibly do is enable an org policy that was again created in collaboration with us in going through this exercise. This is a new org policy that came out in the last year or 2 called the Automatic IAM Grants for Default Service Accounts. And so what this does is, if you remember when I mentioned that there are some default service accounts that get created, those default service accounts still get attached to VMs and Cloud Functions and all kinds of things. By default, often without you even realizing it. And by default, they have act as and token creator permissions on all of these service accounts in your project. And so that's a super, super dangerous thing, especially for a large organization that might have a lot of service accounts. And so the, the easiest thing to do is just to remove that default grant so that when you create a new VM or a new Cloud Function, you start at square zero in terms of IAM, and you don't start with act as and token creator attached to those resources. So that's probably the number one thing is disable that, that default grant. And you can do that through an organization policy that was created in collaboration with, with our efforts.

Tell listeners what an organization policy is though, because not everybody's familiar with that piece of cloud magic.

Yeah, no, that's a good call out. So an organization policy is basically a set of rules that everyone in your organization has to abide by. Whereas before, when you create a project, the default is you could do a lot of things, like you can add people outside of your organization and you can attach firewall rules and stuff like that. If you go in and start pushing buttons and knobs inside the org policy, you can disable a lot of those things or make those things impossible for new project owners to be able to do. The thing about default service grants is it's not actually an active action anybody ever took in the first place. It was just something the cloud kind of like put on you. And so by disabling it, nobody has to have that put on them. Basically, no one has to have these default service accounts attached to their services. It can create a little bit of friction because before, all of your VMs and Cloud Functions, they could do a lot of stuff. You know, they could talk to buckets and they could perform act as, and they could do all these things.

So.

So all of a sudden, taking that away can create a little bit of friction. But when it comes to securing against this particular impersonations problem, it's the easiest and highest impact thing that you can do for your organization, kind of worth the friction that it adds. And then the other thing that I'll mention is that when it comes to these impersonation roles, they have a particular purpose. Like, if you want to be able to attach a Cloud Function to a VM, you need Act As. So it's not like we could just get rid of the permission across the board. But what we can do is oftentimes these permissions are done at the project level. In fact, that's where the default grant is done as well. It's done at the project level and the UI, when you push the IAM button, that's talking about project-level IAM grants. It's a little bit more tricky, but when you dig into the individual resources, you'll find there's a separate set of permissions there. And that's actually how we do our resource-level grants. And so what I mean by that is rather than saying all of our, you know, this Tim identity has act as on all of the service accounts in the project, that's what I get from the IAM page. Actually, I just want Tim to be able to grant this one service account to his VMs. And to get to that, we have to go to the service accounts page, dig into the service account, click the roles and permissions of that service account, and then there we can set our act as permission. So that's the difference between doing a project-level grant, which is to a whole collection of resources, versus an individual resource grant, which is just to the one thing that we're after. And so when it comes to impersonation, if you need impersonation, and there are plenty of legitimate use cases for it, highly, highly recommend defaulting those to as small as possible, just the resource rather than a whole project. Because when you do it to a whole project, that's when you start to get in trouble. You start to pull in extra service accounts that we don't know what permissions they have and things like that.

Got it. That makes a ton of sense.

Okay. That actually makes sense to me.

Yeah.

I was about to say that this, I can see how you can combine it into a more secure, secure approach to permissions, but I also can see how people would trip over it and fall face into the mud. Yeah. I guess this kind of depends on the knowledge. By the way, in some other episodes, we did tell people, learn the cloud before you migrate. And people laughed at us. for this being trivial advice. It is not trivial advice. You do need to learn cloud before you go there. So now we are almost at time. I wanted to ask our 2 traditional closing questions. One would be easy. Where can the audience learn more about this? Any reading materials? Of course, a link to a Black Hat preso would be helpful. And the second is one practical tip on how to deal with a threat, with an issue.

Yeah. So when it comes to this particular problem, which is the privilege escalation and lateral movement within Google Cloud, the number of resources on it is actually pretty light.

Oh.

But there are some really luminary people that I can point to. The first is somebody named Spencer who used to work at Rhino Labs, and there are a bunch of great blog posts on Rhino Labs around cloud security stuff. I think Spencer is at CrowdStrike these days. The next one is actually someone who works at GitLab. His name is Chris, and he's done a whole bunch of stuff on Google Cloud privilege escalation. And there are some great write-ups on the GitLab blog post on that. And the last person I'll call out is Kat Traxler, who used to work at Best Buy and also has done a lot of speaking in this space. And she's done a lot of blog posts. I don't think she's at Best Buy anymore. I forget which security company she's at, but that's probably the third resource. So there's the Black Hat talk we gave in 2020, which has A whole bunch of fancy video editing because I was bored during the pandemic. So check that out. There's Spencer's content on RhinoLabs, Chris's content on GitLab, and then Kat Traxler's content that's kind of a little bit all over the place. But I think she was at Best Buy when she did a lot of it.

We'll look them up and put the links. Thank you very much for this. And any practical tips, like your favorite tip when people deal with this?

Yeah. So I think if there's one thing that I can impress on people, It's that default grant org policy. If you can disable the automatic IAM grants for default service accounts— I know it's a mouthful, but that's the actual name of the org policy— if there's one button to mash, it's that one. You know, if you don't know anything about cloud security, if you're switching over from on-prem and you're moving into Google Cloud, mash that button in your org policy, and it'll improve your security in myriads of ways that I can't even get into here.

Perfect. Thank you very much for this. I very much appreciate you being on the podcast. I think it was fun. Looking forward to future discussions, perhaps.

Yeah, absolutely. Thanks so much.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvakin and Tim_Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.