EP177 Cloud Incident Confessions: Top 5 Mistakes Leading to Breaches from Mandiant

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior PM for Risk Analytics and Cloud Threat Detection in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts. as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on the Google Cloud Security Community page. You can find the link to that in our show notes. Anton, this is another one of our livestream from the floor, fulfilling a childhood dream, but this one feels a little bit more like a security nightmare. But it did inspire what I think is one of the more interesting conversations we've had in a while, 2 days later.

Yes, correct. This episode has led to a lot of new ideas and insights. And yes, it had a couple of scary moments. And they were scary in the how the past gets you in the future. Even if you are really, really modern, really, really cloudy, really, really coherent in terms of your approach to security, then sometimes a particular piece of— I shouldn't say malicious— particular piece of annoying technology from the past comes up and bites you. And I think it starts with an A, and the second letter is probably— what is it?

Somewhere between a C and an E. Whatever letter is in between those two is what it is.

That's correct. It comes and it really bites you. And then everything that you've built in a modern manner crashes on you. And then you call Mandiant and they save you.

That is what happens.

I think that's a good pitch for Mandiant.

Yeah, it's a great pitch for Mandiant. Very good. I think if I were to summarize this episode, I would summarize it with a final line from The Great Gatsby. So we beat on boats against the current, borne back ceaselessly into the past.

That's so beautiful. And I could not quote him better than you did. So this was just incredible.

Well, with an F. Scott Fitzgerald quote, let's turn things over to today's guests. Ladies and gentlemen, welcome to our second livestream of the day. Delighted to have you joining us here. We have 2 guests. And one microphone for this session. We've got Omar El Aden and Will Silverstone. They're both Mandiant consultants, and we debated whether we would give you a fancier title or a shorter title, but we'll just go with Mandiant consultant.

Just Mandiant consultant. Yes.

Listeners, I want to apologize in advance if our audio quality is bad or our video quality is bad due to a misreading of Hamlet. It's been a childhood dream of mine to do a show from within a show. So we're doing a show at the show, but hopefully it works out. I want to start out with you guys here. This is your first Next, right? So what are you guys, what are you up to? What have you seen? What are you stoked about? How does it feel to have so many cloud people around? How's it going?

Yeah, it's awesome being here in Las Vegas. It's actually our second time at Next.

Second time, okay.

We attended Next 2023 in San Francisco, which was also awesome.

And I think the best part about Next and being here is you have people from all different areas of focus when it comes to cloud, not just security.

Totally.

A lot of different perspectives, a lot of different opinions. And so it's just so awesome being here.

Will?

Yeah, no, agreed. And it's nice to be around a lot of cloud people. We spend a lot of time doing incident response of on-prem environments, so it's nice to get the cloud point of view sometimes. So it's great to be here.

That's awesome.

And I think the— my inspiration for inviting you was actually your M-WISE presentation from last year where there was a lot of highlights. Some of them will show up during this session. But one thing that came up to me is that you deal a lot with incidents that are kind of traversing from cloud, on-prem. So it's not like you deal with cloud incidents. There are incidents that involve cloud and on-prem, kind of like hybrid incidents. Why is it hard? I guess I know the answer, but why is it hard?

I don't. I wasn't there.

Yeah.

Okay. Tell us more about this whole incidents that span cloud and not cloud.

Yeah. So yeah, I think probably the biggest challenge that organizations face is that they have their cloud security teams and they have their, you know, non-cloud security teams, and they're not really thinking about the integrations between the two. So, you know, they're not really thinking about how a compromise of your on-premise environment can then lead to a compromise within your cloud environment.

Yeah, and I would add on top of that, that I think one of the common misconceptions that we see is organizations using their cloud as an expansion of their on-prem environment and not really leveraging the speed and agility that with cloud, and that's why they're applying the same method, the security methodology, same program, same tooling even, for the— that they have on-prem to their cloud environment.

So wait a second. So we make jokes on this show all the time about how some people treat cloud as a rented data center, but you're telling me that they actually treat it as a rented data center which they then connect to their regular data center. So everything they have in the legacy environment that's nasty and unpleasant rushes to their cloud environment.

And it's exacerbated by the fact that they treat their cloud security teams separately from their rest of security teams. If only people hadn't started a podcast called The Cloud Security Podcast that suggested it was different.

I think that's very self-serving of you, Tim.

It's not my fault? That's good.

I'm glad it's not my fault. Actually, it's mine, but—

It's our fault. I'm sorry. So what does it actually look like when an attacker is moving from on-prem to the cloud? How does that happen? What should people be looking for? What are some like, Juicy stories to make that real.

Yeah, I mean, if you look at it foundationally, the common integration between on-prem and cloud environments is either through kind of network connectivity and identity services. And commonly the issue that we run into during incidents is organizations using the same identity store that they have on-prem, using that in the cloud. And what that creates is a central focal point of failure, where if credentials from that identity store is compromised, an attacker is able to reuse those credentials to access not only the cloud environment but also the on-prem environment.

But they naively think it's a good thing, right?

Correct.

That's an important side point, that they're not doing it out of spite. They actually think it's great.

Yeah, and also from the perspective of using the same credential to manage both environments, if there's one infrastructure team that's responsible for managing both environments. So more of it from ease of use and ease of kind of a day-to-day doing your job. But I think our recommendation is always for cloud, we want to use cloud-only local accounts that are not tied to whatever your central identity store is. Will, anything you'd like to add?

Are you popular with those teams when you give this advice?

Yeah, I think the focus is usually on, you know, privileged accounts where you're talking about Not every account needs to be completely isolated, but focusing on your privileged accounts, 'cause there are a lot of benefits to integrating everything within a central identity provider and then being able to give out access from there. But understanding that if you provide access that way, that a compromise of an on-prem account can easily lead to a cloud compromise, and then really making sure those privileged accounts are isolated as much as possible.

That's very practical, implementable advice, but it sounds hard. And this feels like a very, very classic security usability trade-off.

But it's also coming from, you're not just giving this advice in the abstract.

Yeah. You didn't pull this out of thin air.

You're giving this advice informed by our IR, which basically makes it really hard to argue with. I may think, no, I want my single active something, whatever it's called, identity store everywhere. But, and then you like, yeah, that's why we had like 50 incidents like that. So sure, do it, but that's likely what will happen.

And also a different perspective is when you look at cloud ransomware-based attacks, most of those attacks occur on-prem and propagate to the cloud. So what happens most of the times is attackers are compromising, abusing a tool for example, that functions and services both environments, and then they can maliciously use that tool to mass deploy that encryption, for example, that propagates across cloud and on-prem. So another reason to look at it not just from the perspective that we need to separate our privileged identities, but also that some of our critical infrastructure and security tooling needs to be separated so that we prevent that propagation effect.

And that's something that I think comes up a lot in like theoretical cloud attacks is ransomware in the cloud. Usually it's like encrypting a cloud storage bucket or, you know, encryption of S3. But what Omar just mentioned there is definitely the most common that we see in ransomware in the cloud. It's, it's that it's a domain-joined virtual machine that encryption was applied in Active Directory and just propagated to the cloud environment. Fascinating.

It's almost like the On-premise mistakes come to the cloud and the attacks that in theory shouldn't be possible or likely in the cloud become very possible, very real, and very likely. So, it's like they're always disarming the cloud advantages.

Yes.

Maybe I'm over-philosophizing this, but that's what it sounds like. No, I think that's right.

I want to pick at this a little bit more and get back to something you said earlier about the siloed teams. Is this also a risk of siloed tools? Like, on one hand, we want to separate our identity providers so there's not, you know, crossover failure, but do we want to, by the same token about the teams, unify our tooling so there's consistent visibility and things aren't happening in silos? Is that one way to think about this?

Yeah, yeah. I mean, I think from an EDR perspective is a good example. Organizations don't necessarily have a process to make sure they have full deployment of EDR across their cloud virtual machines like they would in the same process that they have with their on-prem environment. So that's probably a perfect example of that.

So wait a second, you asked about tools. Didn't you say that the separation is good, but then now you're saying that unified tools is better?

Unified monitoring, it sounds like.

Yes.

Yes, but I think you have to separate it, right? Where when we're talking about separation, the focus points of separation is really around the network integration between your on-prem and cloud, as well as your identity services. When it comes to security and infrastructure tools, we call this at Mandiant trusted service infrastructure, right? It's a centralized tool that services the remaining infrastructure. It's fine to have unified tools. It's actually maybe better for your team.

Yeah.

However, we need to account for those trusted service infrastructure tools and their protection, and we cannot blindly them. And as mentioned, we're starting seeing a trend where attackers are abusing those tools to mass deploy, for example, ransomware at scale. So we need to account for the protection of these tools, and we can't fall in the trap of blindly trusting all of our tools that we leverage across our infrastructure.

So it's almost like the identity better separate, but say detection and response monitoring better together. And this is a subtle and probably a really important point.

But it also sounds a lot like what we're doing with SCC Enterprise.

Really?

Bringing together your in-cloud detection with your Chronicle security operations, unifying that. It actually sounds a lot like that.

For the DNR.

Yeah, for the DNR piece. Yeah, yeah, yeah, yeah. And Will actually talked about this in our MY presentation last year. We want to talk about the control plane and data plane logging components.

Yeah, I mean, From a logging perspective, I'm sure you guys see this all the time, from a cloud logging perspective, the logs that are required to do a full investigation are not enabled by default in a lot of cloud providers.

Right.

So having those kind of data plane logs, those access logs to storage buckets that aren't necessarily enabled by default, having those, but also having your on-prem logs to be able to see, you know, on-premise identity is compromised, logged into a cloud environment and access data in S3 bucket, the full picture of that attack and having, you know, logs centralized that way has a lot of value.

Huh. Okay. That's actually interesting, but let's try to broaden it like beyond identity, behind, beyond networking. So do you have like 5 of your favorite mistakes that people make in these type of environments? I think you are, again, you are not to copy your M-Wise Prezo into the podcast, but what are your 5 favorites circa 2024?

Yeah, like top 5 bad things, by the way. We're not looking for top 5 good, we're looking for top 5 bad.

Yeah, I mean, the top one is going to be around identity. I think we covered that enough. So maybe that— I mean, not having segmented identities, weak MFA enforcement is going to be another big one. We could probably use the rest of the podcast to talk about that. And then the other one around identities is credential management. So if it's service accounts that have long-lived credentials associated with them, having some process to either move towards getting rid of those or having some management process.

So it's like the opposite of zero trust. What do you call it, the opposite of zero trust? Infinity trust? Trust everybody? It sounds like that, like it's the opposite of zero trust.

Right, because we see, you know, you lock down your access to your cloud environment, but you have these long-lived secret keys from service accounts that can be leaked to a code repository, leaked anywhere.

Open every door.

And it bypasses every control you just put in place there.

Yeah, that sounds bad.

Yeah, and I would just add on top of that, kind of key mistake that we see oftentimes is organizations not prepared to respond to an incident.

Failure to prepare is preparing to fail.

Exactly. Like, they receive a notice indicating, for example, leaked service account credentials have been identified somewhere. But no one's taking action. And by the time they start thinking about taking action, that's the same time that the threat actor was able to leverage that same account to go and perform malicious activity within the cloud service provider. So taking that into perspective and building predefined playbooks, just because your remediation and hardening efforts on-prem is just so different in the cloud.

And listeners, one thing I want to point out is we've just added a default-on org policy constraint that if a service account credential is seen on a public GitHub repo, we will kill that service account by default starting in July of this year.

Right.

We should put a link to a blog about it because that's actually kind of important.

It's huge. It's one of the biggest positive changes I've seen in my time at GCP.

And I want to go back to something Will talked about a little bit on multifactor authentication.

Yeah.

Because oftentimes there's a misconception around what MFA method is being used. And we like to break this into kind of 2 separate buckets. One is not only that we want our organizations to enforce strong phishing-resistant MFA across really their entire user base, not just privileged accounts, but then also the operations aspect of MFA needs to be locked down. When I say operations, meaning from where and how end users and privileged accounts are allowed to register or modify an existing MFA method. And that really goes back to seeing attackers performing password spraying attacks, getting into an account, and they're prompted to register their own MFA method. By locking down that registration modification piece, you're prevented against that standard, I guess, attacker technique.

Right. It's the MFA registration and the self-service password reset. We often see, you know, organizations don't really have an inventory of all the ways that a user can reset their password. Could just be one self-service password reset portal, or there could be, you know, 4 or 5.

This reminds me of some, as they say, recent incidents, right?

Yeah, I don't think we want to say anything more about that.

Not to say anything else, but it reminds me of some recent incidents.

Yeah, there's a ton that come to mind that use just that. And so, you know, locking down where you can register MFA from, how you can reset their password, all the ways that you can take over an account. Yes.

Okay, so I want to shift gears a little bit and talk about attack surface. We've spent, you know, decades thinking about on-prem attack surface. How are people missing the boat when it comes to cloud attack surface? What should people be thinking about that they're not thinking about? How do we put that into perspective?

Yeah, great question. I think the biggest misconception that we see also related to attack surface is organizations looking at their attack surface purely from a Network perspective. External network.

They assume it's a scan, a spray and find IPs kind of thing?

Correct, exactly. Using some sort of an automated scan that looks for services, managed technologies that are exposed on the internet and could be used as a vector against an organization. In some way, that's actually a positive thing because we want organizations to be continuously scanning their egress IPs, public domains, etc., to look for what's But that's not just it.

It's a yes and.

Correct. Yes, we want organizations to account for their identities as part of their attack surface, taking into account the trusted service infrastructure tools. What I indicated earlier around infrastructure and security tools that service your entire infrastructure should be part of your overall attack surface reduction program. Also taking a look at endpoints, and if a non- If a non-privileged user's endpoint is compromised, will an attacker be able to dump credentials from memory that can be used for lateral movement or privilege escalation? So kind of to summarize, taking into expanding your attack surface program to beyond network and making sure that it encompasses identity, trusted service infrastructure, and endpoints.

It's a lot trickier than scan this range, right? So it's kind of like, what's your advice for this pillar for the identity as an attack surface? How do you find it? There's no, Scanner for accounts.

Yeah. Is there an off-the-shelf tool for this?

I wish.

Yeah.

But yeah, it's an inventory issue, I would say, is one of the things, especially not from identity side, on the trusted service infrastructure side, especially as a lot of these tools are, you know, SaaS-based and can be accessed from the internet. Is your credential vault, is that a SaaS-based tool that can be accessed from the internet? Can we at least, you know, lock that down so it can only be accessed from specific network locations or specific devices and not have that be— that portion of your attack surface be accessible from the internet.

But also when it comes to identity, and just to expand beyond that, also kind of take what we talked about earlier is segmenting your privileged identities in the cloud using local cloud accounts.

Network segmentation, but identity segmentation.

Correct.

Say more about that. What's identity segmentation look like?

So the quick and easy way we always recommend our clients to do is if you're using GCP, for example, use GCP-based local accounts for your administrators so they're not tied to your identity store. If you're using any other cloud service provider, apply that same methodology at those accounts.

That's interesting.

Apply strong MFA, restrict how they can reset their passwords, and collectively all these controls together do reduce kind of your identity attack surface. That's practical advice.

It actually makes sense, and I like that. So So the network is still a pillar of attack surface. Identity is a pillar. Is there more, a third leg of the stool or something?

Yeah, if we look at it kind of the third and fourth pillars, I would say, you know, third would be your trusted service infrastructure, any security and infrastructure tools that can deploy scripts, for example, across your endpoint servers and virtual machines. And the fourth pillar would be endpoints and making sure that we're accounting for our endpoints, specifically our privileged administrators' endpoints that are being used to perform administrative functions as part of our overall attack surface reduction program.

So I think I understand the first, second, and fourth, but I'm not completely sure about the trusted services. Is it essentially APIs or not? Or what is that?

No, it's like your endpoint management, your EDR.

If you're thinking about, you know, the Active Directory tiering model, It's expanding what that Tier 0, which is traditionally just like domain controllers and PKI servers, and expanding that to include your mobile device management platform that can be used to, you know, push scripts out to endpoints, your EDR tooling, your remote access support tools, all that kind of stuff also.

And a quick IR example kind of from the front line is we're seeing attackers target accounts that have privileged access into a victim organization's MDM solution, mobile device management solution. And then from there, once they're able to get to it, they mass deploy malicious print across every single endpoint managed through that MDM.

I bet the security team was so happy they invested in all that automation.

But it's a resilience— well, resilience for the offensive side, basically. Their access is much more resilient as a result.

Yeah, that's awesome. That's turning people's tools against them in a really tragic way.

Exactly, tragic way, yeah.

I mean, that part is— I almost opened my mouth to say that sounds like fun, and I'm like, no, it really doesn't.

It's really weaponizing tools, platforms, and native cloud services against the victim organization.

And we've even seen this in public breaches. Like, this came out in the Uber breach, right? Compromising the privileged access management tool and then escalating from there. So this makes sense. This isn't like some deep secret.

This is, uh, But it's a pillar that people may not think of when they think of attack surface. I think that if you have somebody like a mundane traditional view is network, somebody who's enlightened would think about identity and then probably exhaust a large population at this point. But thinking that there's trusted service part of attack surface and then talk to endpoints. Endpoints to me, I would argue against, but I wouldn't do it on a podcast. But I think because ultimately they're connected to a network, so it feels like it's If you do a good job with network discovery, you should cover endpoints, right?

Correct. But the common mistake that we see is there's a lot of default configurations across endpoints that attackers have used to obtain credentials. One small example, there's a setting in legacy Windows systems called WDigest, and if WDigest is enabled, it allows clear-text memory to— clear-text credentials to be stored in memory. So we've seen attackers dump credentials from memory by abusing WDigest, right? So hardening those devices, specifically the ones used by administrators.

Okay. That is not reduced to port scanning. So I think I can see why it's not about the, do I have the port? Yeah.

Okay.

That's an OS setting.

That's truly an endpoint as a pillar. That's kind of mind-expanding slash eye-opening.

You just changed Anton's mind live on air. That is, you win a big prize for that. Good job.

I think the way that that relates back to cloud, if we bring it back to the Cloud Security Podcast, is I think cloud security teams spend a lot of time focusing only on cloud, obviously. So think about all those checks that a CSPM tool would do across all the different misconfigurations. And the things that we're talking about here are kind of backdoors to, you know, you can put all the encryption you want in place, you can do all your misconfigurations But if your mobile device platform is compromised, if your on-premise Active Directory is compromised, it's an easy way to get back into a cloud environment that bypasses all of those misconfigurations.

That makes sense. So let's actually— I wanted to step on the identity button one more time because the previous time somebody changed my mind, not on the air, was actually you guys at the M-WISE Prezo when I had this epiphany of sorts when Even if you do whatever cloud you do, whatever modern or legacy environment you have, what gets you is often the AD. So even if I am a more or less enlightened GCP client or use some other cloud, what may lead to an incident is Active Directory. So it's like the same line of thinking that you just expressed, that they sync on-prem credentials, rights, privileges to cloud. And so their modern, beautiful, shiny cloud becomes exposed As opposed to that 1990s stuff, badness coming from that.

Right. I guess I would expand that on top of AD to really identity providers in general. AD is not built for security. I think everyone kind of understands that. So that's very often the most—

Can you say that again slowly for the people in back?

AD is not built for security.

Very good.

But so it's very often the weak point. But really, you know, if you think about any third-party identity provider that organizations are using to access their cloud environment, that's very often kind of the pivot point that an attacker can use.

I think it goes back to the main point of do not use your cloud as an expansion of your on-prem.

Yes.

Cloud is different. It needs to be different.

But it also needs to be considered as part of it, and you need unified observability and trusted infrastructure. That's complicated advice. Like, there's a lot of nuance in this that I hope people listening don't lose because that's critical.

I mean, that's a good example where the answer is not magical, or the magic is that you should do a good job with this, following that path. Like, it's not like you have this secret short path.

No.

Yeah.

A secret short path is the one that leads to a ditch, which we just briefly described.

Yes. Okay. So, what's your best preparedness advice? We touched briefly on, you know, failure to prepare is preparing to fail. What's your counter-advice to that? How do people who are new to cloud, how do they do this right?

Or maybe another way to say it, what's the best way to not meet you?

Yeah, how do we avoid talking to you guys?

So I think it goes back to kind of the key talking point throughout the entire podcast today, identity. Let's start with focusing and narrowing down on our identities. At a bare minimum, privileged accounts, let's separate those from any identity story that we have. Let's make sure that we're enforcing strong phishing-resistant multifactor authentication methods. such as FIDO2 security keys. Let's lock down from where and how those administrators can reset their passwords or modify their MFA methods. Let's continuously scan for what's publicly exposed or what could be publicly exposed and build playbooks so that we know how to react once we receive an alert. Anything else you'd like to add?

Yeah, I guess just from the logging perspective, like we talked about, making sure that you're going through and inventorying Those logs that are not enabled by default, and determining for your organization: Do these have any investigative value for me? Or is it logs for a database that's holding the most sensitive data for my organization? I should probably have data access logs enabled for those. And then I think from a detection perspective, a lot of the cloud providers have really good out-of-the-box detection capabilities. Exactly. And then kind of taking a look at those, and then on top of those, are there any? Custom detections for specific processes for my organization. Maybe we use a specific deployment pipeline to deploy infrastructure, and we put a detection on that for any infrastructure that's deployed outside of that pipeline. Things like that.

Wait, you mean the Terraform job that's got the permissions to do everything? We should protect that?

We we could consider protecting that. That's probably a good idea.

And I guess the system that can modify it then, and the system that can give commands to the system that can modify it. becomes a trusted services chain, which is again now my newly found epiphany.

It's a great epiphany.

But, you know, I want to make a depressing point that it reminds me of compliance, because if something touches card data but something touches that system and then something touches that system, it becomes kind of a chain. Oh yeah, fine.

You've made me very sad all of a sudden. Guys, this has been such a fun chat. I hate to do this, but it's time to ask our traditional closing questions. Do you have one tip? And I feel like we've just kind of condensed on one around identities. And then 2, do you have recommended reading for people? And that can be anything from like Dune to Mary Oliver to the SRE book, just not Anton's blog.

Yeah, Anton's blog is kind of under the embargo, I think.

Yes, I think it's the swear jar. You got to put a dollar in the swear jar if you say his blog.

I would say the wrap-up recommendation is use phishing-resistant MFA methods. I think that would prevent almost all of the cloud-based incidents that we've talked about here so far. And then I think reading material, there's a really good newsletter, comes out weekly called, I think it's CloudSecList.

Yeah.

Yep.

Marco's list.

Yeah, Marco's list. Yeah.

Right. So that's a great resource.

You're coming on the show, buddy.

Yeah. Keep up on any, you know, what the news of the latest cloud security trends and all that is. So that's a good one for me.

I would say, I'll actually give a documentary that's not related to security, but I think—

Sure.

It resonates a lot. There's a documentary on Netflix, it's called My Octopus Teacher, and it walks you through how resilient and how the octopus adapts to how changes, surrounding changes and changes in its lifestyle. And I think it really resonates with how we as a whole, as security teams, need to be, just given that the threat landscape continuously shifts and evolves with different and net new attacker techniques. And we, from a defense perspective, need to be kind of as fluctuating as an octopus.

I love that.

I love that because it's not about like build harder walls, bigger bricks, but it's like be more like octopus.

Be like an octopus. Be resilient. Be adaptable. Be able to squeeze yourself through a crack this big.

Maybe that wasn't that funny.

Wait, not that last part. Okay, guys, thank you so much for joining us today. This has been terrifically fun. Listeners, I hope you enjoyed. Tune in tomorrow. We've got 2 more of these before Next is over.

Awesome.

Thank you for having us. Yeah, thank you very much. It's been a pleasure. Thank you.

And now we are at time. Thank you very much for listening and, of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.

Security.