EP199 Your Cloud IAM Top Pet Peeves (and How to Fix Them)

Security.

Hi there.

Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior PM for Risk Analytics and Cloud Threat in Google PsychOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us, and the rest of the Cloud Security Podcast listeners on our Google Cloud Security Community page. Without any further ado, let's turn things over to today's guest.

Hello, everybody who is listening to us in the livestream or in the recording. Today we have 2 esteemed guests, Michelle and Sita, and they're both big fans of IAM. Are you guys?

No.

No? Okay, so there's a mistake here, right?

Well, I've written about how much I hate IAM, and I always, when I give talks, I always talk about how much I dislike it, but I know it's important. But it's sort of like flossing, you know?

You don't like it, but you do it? Yeah.

Yes.

And I think your very famous I Hate IAM, But I Need It Desperately blog is definitely required reading. for people listening to this episode because, well, first thing, Michelle needs more views on her blogs, and second, it's a genuinely fun blog. So, let's get to our favorite subject, as we just established. I wanna start from a somewhat conventional question. That is, people sometimes say, in the cloud, your one IAM mistake away from a breach. And it's been kind of a cliché, and it comes up and down, and people like it, people hate it. So, What is your reaction to this line? Like, when you hear, in the cloud, you're one IAM mistake away from a breach, are you in favor? Do you dislike it? Do you have some kind of nuanced answer? Who goes first? Let's see, alphabetically.

Okay. Yes, you are one mistake away, mostly because really cloud security is identity first, right? I say this a lot, you know, the boundary in cloud is one of identity. That's really where everything starts. And if you don't get that right, It just sort of starts to cascade into more and more mistakes that are more horrific in terms of over-provisioning or putting people in the wrong group and assigning the wrong roles. It just— and I've seen it in personal experience. I worked at a place where that was the cause. It was an over-provisioned account. And yeah, that was what was leveraged in the breach. And it was embarrassing. But, I think it can be a little arcane, and let's face it, it's not very exciting. It's not sexy. The SecOps stuff is much more interesting. Talking about threat intel is much more interesting than IAM. I know, I know, I know. So I think that's why. But yes, I would agree that that is accurate. That's an accurate statement.

Okay, so Sita, what's your take? I really hope you would say, no, no, no, she's just totally wrong. Okay, or maybe you'd agree, so.

I wish, but she is spot on. But I hate your question, Anton, because it's so true. We are just one mistake away from a breach in cloud IAM because, you know, partly it's because of 2 things. One is cloud, because in on-premise, you don't have the same effects of a cloud. Let's say a service account credentials got leaked in on-premise. An attacker has to be in the private network, somehow get the credentials, and then do the damage. But in cloud, we have everything in the open. Everything is public. You have the credential, you assume the identity, and then boom, there's disaster. So that's really why, because it's easier to have a breach because of IAM. And also, if you see, funnily enough, IAM breaches are caused mostly due to misconfigurations, not due to any vulnerabilities. And there was a report which said 74% of the breaches that happened in IAM were due to manual misconfigurations, and that's a huge number. So, Yeah, you need to have good IAM hygiene in cloud to have a really good security posture, and that's non-negotiable.

Can we talk about that? So, there's a great report from Gartner. I just have to bring this up to sort of, you know, stick it to you a little bit, 'cause, you know, former Gartner. More than 95% of infrastructure-as-a-service accounts use less than 3% of the entitlements they're granted. I love that.

That is kind of mind-blowing, though. This is depressing and mind-blowing at the same time.

Yeah.

I think that for this just sheer percentage and sheer coverage, like, my previous scary number from cloud was the percentage of containers with SSH access, which I assumed to be zero. Like, in a normal sane world, that's zero, right? Zero is the right number. So, I kind of— but to me, the 95% and 3% is Similarly scary/bizarre. It's like a horror movie noir kind of data point.

It's scary, but you know, it's not actually quite surprising because no one cleans up tech debt. We all know it. We've all been there. And IAM tech debt is the worst.

I've never seen any— every time you try to— it's like I was talking to Sita the other day and I said that IAM is kind of like— IAM cleanup projects are like IPv6 projects. Everybody goes, yeah, we need that IPv6. We need to convert all our IPv4. And then everybody goes, nah, we're just gonna do translation at the edge. It never comes up again.

Hmm. I mean, so let me use this opportunity to sort of gently push against the whole 1 AM mistake from a breach, because what about auxiliary layers? What about NetSec? What about things like EPCSC in case of Google Cloud? And yes, you can gently push against my push by saying they're all controlled by identity. But these are more than one mistake, then you have to make 2 mistakes. You have to have a mistake in the primary access, and you have to have another mistake in configuring, say, VPCSC. So, are you sure that in a well-designed cloud environment, you are still one IAM mistake away from breach? No.

Okay, we're assuming, yeah, we're assuming a good design, but how many people measure twice and cut once, really? Especially with IAM, They just want to get in and, like, techies are going to tech, right? They want to be hands-on keyboard. And IAM architecture, it's a design. It's like you pull down your org chart, you do persona mapping, you make sure you understand what everyone's job function is, you map it to a group. There's a lot of actual paperwork. And nobody really wants to do that. Nobody wants to sit down and do that planning because... You know, resource hierarchies and organization layouts, it's not exciting.

So, you're kind of going with it never happens. You're going with, well, architect it, what do you mean that never happens? Like, that's your answer.

Yeah, it kind of is. Like, I'm just telling you what I've seen in organizations over and over again is that you get there, and I was at one organization, I'm not gonna name names, of course, unless you buy me a drink later. I've been in places where it was caused by a compromised account, you know, an identity, and they talk about how they're gonna mitigate it. And I was in a meeting and I have this really good friend who loves IAM. My friend Nagesh just loves it. And I remember being in a meeting with him and they were like, oh, we're gonna put these new firewalls. We're gonna do this super segregation and segmentation down. They're gonna be super small. I went, well, what about identities? Oh, that's out of scope. I went, oh, no, no, no, no, no, that's not out of scope. And we start arguing and then Nagesh just goes, Michelle's right. And that's when I called CNN because I, of course, wanted it, you know, the world to know that somebody said I was wrong.

I hope you're joking about CNN, but let's—

But yeah, so it's, that's just my example of how common it is. I mean, Sita and I have also experienced this together when we work together that I'm, you know, constantly nitpicking on when we set up accounts ourselves. So, you know.

Yeah, it's not always the fancy stuff that ends up causing the problems, but it's more like the overlooked stuff. Recently, there was a vulnerability that was discovered in a public Docker container in Docker Hub, which had a leaked admin access account to really powerful core Python repository. And that was Scott. And it's so, you know, beyond me to imagine the consequences we would've had. But it's always the stuff that we overlook, the small ones, that ends up causing the really bad, dangerous effects.

And it's our fault. I mean, the roles are bizarrely named. That's why you have to plan as well. 'Cause you look at, if you ever look at IAM, I mean, it's like, I used to be a Sendmail admin, and reading the sendmail.cf file, I know I'm dating myself, right? But, you know, you read that file, and the statements in it are very arcane, so it's pretty simple to mess up. That's why the planning is so important. It's not that people want to do the bad thing, they just don't have the time, right?

It's just, and it's too hard, to be honest.

I think that makes sense. And I think it also kind of brings the real world back in the story. I wanna handle one more popular cliché connected to IAM, and it's this whole identity is the new perimeter. Like, I think I've heard that, probably in the early 2010s. I don't know, like, I'm thinking that it's another one that just floats from report to report, from paper to paper. So, Saseeta, what's your reaction to this whole identity is the new perimeter? Can you break that down in simple terms to people who are maybe not deep in IAM?

For sure. I think at the core of it, the concept is really simple, because identity was the next logical step thing to do. So, every interaction with with a system or a data resource or platform. Everything's tied to an identity. And earlier, we used to have physical machines to do access controls. Thankfully, we don't anymore, and we don't have to deal with it. But yeah, we've been dealing with physical parameters as boundaries, and we were so sure as long as there's a firewall, no one could harm us and we were safe. But that's not the case anymore. We started building in cloud, building remotely, and we were everywhere. And the concept of a perimeter became very blurry as we went on, and we needed something new. And if you think about it, we've always used identities, be it on-prem, be it in cloud. We've always used identities to access something, and that actually became our logical next step to become a new perimeter. And it's very fitting in our case.

But when you hear this, you don't think, I don't need firewalls, you don't need network access control.

Absolutely not.

Because this— I've met people who treat this as, IAM is the perimeter, not the network boundary, not the firewall, not the network access control. Like, you don't have this negation of the old perimeter take in your interpretation.

Security is like an onion. That's why it makes you cry. You want layers, right?

That's right. I think you mentioned—

Yeah, you want layers, you want stuff to work together. Yeah.

I think it's Michelle's previous episode, I think, where we mentioned the whole onion versus garlic security model, which I think I need to bring up to the audience. And obviously spice up. Yeah, sorry, I'm terrible at puns. I need Tim. Tim's not here today.

I was a firewall admin. I did all those things. I had to set up bastion hosts. I used to have multi-user shell systems, like, with no firewall, you know, when you're only protecting it with a host-based firewall. I mean, so I've been there, and really it's about an evolution, right? We always had identity, but now we're using it in a more powerful way, I think. Is that what you're saying, Sita?

Well said, yeah.

Yeah, perfect. So then, at this point, we are kind of close to the question that I've been obsessed about for quite some time, and I'm using this opportunity to sort of like, I guess, absorb knowledge from you and hopefully from the audience if somebody asks the question. And this thing is, the point of the obsession is this. A lot of people say in the cloud, you may make mistakes in certain places, but you must do IAM right. So, it's almost like, sounds like good advice, given what we just discussed, one mistake away from a breach, but it's also very hand-wavy. And I've asked multiple people and got kind of like hand-wavy answers from more of them. So, when you both hear, in the cloud, you must do IAM right, what comes to your mind? What's your 3 AM answer to this?

In my experience, first of all, it's planning, right? You're basically recreating your org chart in the IDP, in the identity provider. So, you want to make sure that you really know what that is, right? Most org charts are kind of fuzzy. Well, how are you gonna recreate that in an IDP? The other thing is you need to know where your identities are. In hybrid environments and traditional on-prem, say you're finally migrating to the cloud, you could have identities scattered all over your organization. You need to understand what your system of record is. This is nothing new. I remember going through this back in the olden times with LDAP.

Mm-hmm.

And hey, what system are we gonna use that's the authoritative data source for that identity system of record. I think that's the most important thing. Understand where they are and then sort of map out that org chart and how that's gonna look and craft that on paper or, you know, maybe digitally in a mind map. But, and then get, then you can go from there. I think that's the most important part. The nuances of the technology are gonna vary some. depending on the identity provider you use. But that's the most important thing to start with in my mind. Is that helpful?

It's helpful, but you didn't just say planning is critical, because of course, most people would agree that planning is critical, but like, a type of planning you described focuses on these persona types rather than doing access by actual person. So to me, is the persona mapping part of the answer, or is this— there are also other critical parts of the planning?

Yeah, the persona mapping, you're also gonna need to figure out who is going to manage IAM. Do you know that this is the biggest war that happens in organizations? I've seen organizations either, it's like hot potato, no, I'm not it. Or I've seen some organizations fight over who gets to have IAM. You know, does it belong in security? Does it belong in cloud and platform engineering? That's a big discussion, you know, because that means who's gonna do the regular maintenance? Who's gonna handle cleanup? Who's gonna handle schema extensions? Like, where you're mapping, when you're doing federation, you know, maybe you use AD, and you wanna federate with multiple clouds. How are you gonna— who's gonna be in charge of extending the schemas, and then mapping using the common language, right? CEL, I think, is what you use, yeah.

This sounds like work, though.

Yeah, it's a whole architecture and engineering exercise, yeah. It is a lot of work.

Oh, I thought you have this magic wand that I think my 5-year-old would use to, like, And then it's like done?

No?

Yeah, yeah, everybody wants this. Everybody wants to do it.

Oh, never mind. Let Sita try.

Yeah.

So Sita, what's your answer to doing IAM right? And please don't include things that involve York. Otherwise, you're fine.

Too late.

But I have to completely agree with Michelle on having a solid org mapping. I think that's one of the most important tasks you could do for your IAM strategy. And maybe if I have to add to it, I would say don't cut the IAM red tape. It's absolutely there for a reason. We all know IAM is hard, especially when you're in a multi-cloud environment or a hybrid setting, when you have your identities everywhere, multiple providers. It is difficult and notorious to manage, but you have to pay your IAM due diligence because That essentially defines your perimeter in the cloud. And how secure or how restricted your perimeter is depends upon how strong your IAM mapping or a game plan is. So, you have to invest time into getting that right.

Oh, zero trust. Wait.

Olga, you just had to go and say it, right?

I just had to say it. How many times have I been asked, we want zero trust. I ask the correct question, what's the state of your IAM repository? What's your system of record? What's, you know, how is IAM architected? And they go, oh, no, it's terrible. It's in terrible shape. Like, you're not gonna get zero trust then, right?

Okay. That's a very fair point. And I think that it's ultimately a more useful confusion than equating IAM with zero trust. Like, if I get— if you get good IAM, then you have a big stepping stone, but you're not done with zero trust. There's other stuff to be done.

Right.

Okay. So, like, I almost feel like we should not get distracted by ZT, but my next question, our next question here is kind of also pretty general. One thing that I've encountered recently is when people really start to question the least privilege approach, which to me is kind of foundational in traditional security. And I'm thinking about both least privilege, separation of duties. These things came from, I don't know, '70s or '80s, but can you explain, do they still matter in the cloud? And if yes, maybe a short and possibly fun journey into do they matter and how do they play with IAM discussions we are having?

Yeah. I mean, least privilege and separation of duties, these are the core principles of identity, right?

Even today, even in the cloud.

Yeah, sure. You don't wanna over-provision. You don't want— you want the right roles. Are you a cloud engineer or are you a software developer? You're a software developer, you don't get direct access to production, period, end of discussion. You don't get to use production data in development, end of discussion. And we know your identity. That's how we're restricting that. Because that's how you reduce risk. That's how you mitigate the possibility of pivoting across, you know, when you look at that, the MITRE ATT&CK framework or the intrusion kill chain, you don't want that ability to pivot across the environment and get to the keys to the kingdom, right? You know, the wonderful PII or credit card information or something like that.

Yeah, it's actually providing the right permissions. but also the right-sized permissions.

But you're saying that there is a way to do it even in rapidly changing, very dynamic, very multi-party cloud environments where there are developers, multiple teams, this entity, that entity, customers, employees. You're saying that it's not a hopeless quest to go for least privilege.

Yeah, but you will need to be able to have somebody who takes that as a responsibility, a team or a person within your cloud engineering team. You need to be able to, You gotta own it. You gotta step up and own it. You know what? IAM is like a bonsai tree. You know, like bonsais are very notoriously difficult. You've gotta maintain them and clip them all the time, and you have to hire somebody to come in and take care of them if you go on vacation. That's how you sort of need to think about IAM, that kind of care and feeding. You know, it's like a pet.

Right, but careful with pets, we're in the cloud. Yeah, yeah. So that's—

I know, I know, I know, I know. Think about it like an animal. Okay, you can't just walk away from it. And I think, in my view, I don't know, Sita, what do you think? Is that— that's what I see, that people just walk away and they think it's just going to take care of itself. But what do you think?

I think we have to circle back to the Tech Debt topic because if there's one thing that's notorious for not getting fixed, it's the Tech Debt. And IAM is even worse because we have to now talk about entitlements and managements, and that's a whole other topic by itself. How do you make sure the permission that you assign for someone is still valid after a year? That's a big question. You have to look at the access patterns. You have to constantly keep track if they do need the permissions. And that's where I think we may be talking about the other concept, just-in-time access.

Which is a much more modern concept to me than just do this privilege for everybody forever, right? Yes.

And that's brilliant because, yeah, maybe let's take the case of Someone who's doing the compliance work, maybe an auditor, needs access to your dashboards or databases to do compliance. And that's for a very limited period of time. And you'd want to provision them with the permissions for maybe like 2 months, set a timer, and then forget about it, and the permission removes itself rather than, you know, you're going to having to pull, remove the permissions, and doing the stuff manually. So yeah, I think that's a brilliant one.

How many times have I seen a Cisco support account in somebody's firewall? I mean, that's— it's a concept, right? You're asked by a support person, oh, can you You give me access, and nobody ever takes it out, right? And that's the account that gets compromised, right?

It's like the long-lost secret.

But I think, I feel like in some environments, the secret is becoming more lost over time. That's the source of my nervousness, is not that it's a lost secret, but it's also kind of more lost over time. At least that's for me, is one of the signs I've seen is when people say, oh yeah, everything's so dynamic, I'm going to lose least privilege, I'm just going to over-provision. But then, then they end up in the news, as we discussed, right? Yeah, that's— they throw it out. Okay, so on this depressing side, we have about 5 minutes left and we have one very, very fun question that I always have to ask and it's the pet peeves. I think that that's another one that goes to both of you and let's go in the reverse alphabetical order and start with Sita. So what are some of your IAM pet peeves, presumably learned from clients, picked in a dark alley when you're walking in the park? I don't know, it doesn't matter the source. Don't name clients ideally, but— I am pet peeves.

For sure. I think if you're talking about pet peeves, we can all take roundtable turns and then go on for days.

Yes. Limit to 6 hours. Oh, sorry. 6 minutes.

I think maybe the top 2 for me would be not doing MFA. Because, you know, Michelle has this funny saying, you know, MFA is like exercise. It's good for your health, but you still don't do it. And that's like basically it. beyond us. Because, you know, you could have the best possible passwords in the world, but what happens in case of a breach? It doesn't have to be a first-party breach, maybe a third-party breach. Your password gets compromised. You need to have a backup. Like, when a building's on fire, it's not enough if you have a fire escape plan. You also need a fire extinguisher. You can't question, why do I have this? You need to have both for emergency. So yeah, doing MFA, not doing MFA is probably one of it. And maybe the other one is long-lost secrets, like we spoke about, like setting something, forgetting, provisioning, over-permissioning, and then that stays on forever, conveniently for some attacker to, like, exfiltrate your sensitive data over time. There was a particular case a few years ago where an attacker tried to exfiltrate sensitive information for years together after the initial compromise. So yeah, we'd want to avoid that kind of situation by tying it back to the Just-in-time privileged access and also maybe break-glass mechanisms. That's another great one. Like, just because someone has admin access, they shouldn't be able to use it. You need a justification, a paper trail.

Oh, that's my pet peeve. That's my pet peeve.

Oh, we're leaving pet peeves. Let's do it.

Yeah.

Okay. My first major pet peeve, and it sounds so boring, I sound like a schoolteacher here. I should do this. Failing to plan, right? I think because nobody— it's so boring. And when you start in cloud, you want to go and you want to try things out and you want to play with things and get things working, but nobody wants to slow down. Wait a second. It's like, I'm the bad news. I'm the grump. No, we have to plan. But that's my pet peeve. I really like to see, like, I'm really in love with resource hierarchies and organizational structures and replicating that. I wanna see that. Like, I'm a nerd, for example, for the File System Hierarchy Standard. I wanna see that laid out in identity, right? So failing to plan, nope, because nobody ever wants to come back and fix things later. It's really hard because you inevitably will mess things up, right? And then I have seen this happen way too many times, giving founders the root or the super admin account, right? Because they don't want to be locked out, right? They think, oh, but I need to be—

That's a startup, small company problem, right?

Yeah. Well, I've seen, I once worked at a software company where we acquired A smaller software company, and the founder left, and they had lost the root account. So, that will—

I want to laugh, but I'm not laughing because this sort of unfortunately makes sense. That's the thing.

Well, and I've had another problem. I mean, this is just, this isn't necessarily cloud, but it was with a, you know, one of those digital ad plugin companies. And the, CEO had super admin rights to the console. This guy got phished and they had been trying to compromise a major media publication and they couldn't do it, but they were able to do it through the ad plugin company because this— they were able to phish this guy. So it can be very impactful when people don't really walk that out and think, oh, those are the consequences that can occur. So those are my pet peeves.

Cool. I love those. And I think that this— I like how they are both nuanced but also broadly applicable. So, thank you very much to both of you. So, Tim usually asks our favorite closing questions. And of course, this is one tip and some reading materials. I think that let's do maybe one tip each and one link. I mean, for links, I'd probably use your blogs. These are amazing.

Yeah.

Not only I Love, I Hate IAM, and of course, IAM is a Mountain blog. But anything else? One tip, one piece of reading for the audience.

I would say least privilege.

Oh!

Because it is super— I know, right? That's everyone's favorite. Assigning the right permissions and the right-sized permissions for a very specific or a sensible duration to the identities is super crucial to having a very good secure posture for IAM in the cloud. And reading and thinking. I was about to say the blogs that we wrote, but he stole it.

Which is fine. Yeah, that's fine. Yeah, I mean, that's actually okay. I mean, I'm sure you have your favorites.

Yeah, yeah, yeah, our blogs.

Yes, we have blogs. Yeah, that works.

Or my 101 talk. There's still a recording on the Next site that I did with Elliot Abraham, the wonderful Elliot. Yeah, that's wonderful. And my tip is, if you're gonna change, if you're going to make major changes to IAM, when you do a cleanup, make sure you have a dry run. Make sure you create some kind of org or secondary practice org where you can try them out and you can replicate how it might break applications by having like a practice site. You really wanna do that. If you've gotten to that place, and I'm sorry you're there, I know it sucks. If you've gotten to that place where now you're cleaning up a mess, you need to have a practice space. Don't try to do this in prod, even with all the tools in the world. When, you know, we have some great ones in Google, I'm just saying, to troubleshoot inevitably. Like, I have been there. I have made the whoopsies where I've like worked at university and taken out all of parking for the entire university for like half an hour.

Oh, no.

I have done that. So, you know, you don't wanna be there. Don't be me. You know, have a practice environment.

Perfect. Thank you very much. And I love this one because I think that one is definitely safe. This one is worth the whole price of podcast on its own because this to me is a cool one and people sort of assume, and then they end up like, what? We assume the content validator works, YQA, and we'll be like 9 million systems down. Sorry for jokes about the current events. Thank you very much for being on the show. This was amazing. This was the energy. That we love, and I think the audience loves, too. So thank you very much.

Great.

Thank you.

Find us on the blog.

Yeah, to read.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikian and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.

Security.