EP153 Kevin Mandia on Cloud Breaches: New Threat Actors, Old Mistakes, and Lessons for All

Security.

Hi there.

Welcome to the last episode of the year for the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Tim Peacock, the Senior PM for Threat Detection, now at Chronicle, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with your hosts and the rest of the Cloud Security Podcast listeners, not on Twitter, not on LinkedIn, but special announcement teaser for the end of the year, we've joined the Cloud Security Community page. We have our own dedicated discussion section. You can come argue with Anton and myself over the internet in a forum and the rest of the listeners. We're really, really looking forward to the discussion, especially around today's episode, because Anton, who do we have today?

Actually, it's a bit of a secret, but it's of course out now. It was Kevin Mandia.

Kevin Mandia. What I loved on this episode is you tried to hold Kevin Mandia, CEO of Mandiant, accountable for something he said in what, 2001, you said?

I don't remember when it was.

20 years ago.

2000 to 2001. But the shocking part isn't whether I tried to hold him responsible, but he continued as if it was yesterday.

Yes, he knew what you were talking about.

It blew my mind a little bit. It was like very motivational and fun at the same time.

It was motivational and fun at the same time. And maybe Kevin's the kind of guy who needs no introduction. And so with that, check out our community page. And here's today's guest. Today we are joined by Kevin Mandia, still titled CEO of Mandiant. Kevin, Thank you for joining us. We're delighted to have you here. We wanna look back at the year that's shockingly already gone by. What were some of the surprising cloud breaches in 2023 and what do we learn from them? How are they different? What's new? What did we learn this year?

So Tim, after 17 years as a CEO, I've learned how to take a question and then say whatever I want, right? First off, and by the way, Anton, I know you do lots of 1990s references. Well, I'm gonna bring you back to the '80s for all of my references. So hopefully people still remember Back to the Future. Probably that was made in 1990. You know, it doesn't surprise me we're starting on breaches. So what I wanted to do is just take a moment to say, why do people ask Kevin Mandia about breaches? It's interesting 'cause I go back to 2004, we started Mandiant with the belief that it was strategically important to respond to breaches, but it was never meant to be our core business. Believe it or not, maybe I just executed really poorly. We were an endpoint company. a self-funded endpoint company. And we funded it, but I always believed that security could not be done with software alone, period. And that you had to create a learning system that I did not see the prior vendors, Symantec or McAfee, make. That learning system of let's go out and own the moment, systematically own the moment of the breach at Mandiant, because that was critical to just making better security products. So that's why we did it. And then somehow we became, you know, the governance folks always go, man, you're getting scared, we're going to get a bunch of FUD. We were never designed to do fear, uncertainty, and doubt. We responded to every breach as security professionals, and we wanted to make it a discipline solely so that we had first mover knowledge on the new and novel attacks, which I think is critical to any security product, whether it be SIEM, Endpoint, network, hell, you make it up and throw 4 letters in front of it and put it in the cloud as a new product category.

Just don't put X in front. Don't put X in front. No, please, no X in front.

Not XDR. Is that right, Anton? Come on. How about X? Well, you can't do XXX either. It has a bad connotation. So the bottom line is you got to go X security. The bottom line, I wanted everybody to hear that. But that being said, We became known as the incident response folks. So we get the question on breaches. Now, back to your question, when you asked most surprising was the key thing, Tim, that I picked up on.

Yeah, please.

To me, I go right to Storm-558. I don't know how to deal with the preceding zero to that group, by the way. I guess it's a 4-digit number. Storm-0558, and that's the Microsoft nomenclature for it, and I hope I got that number right. To me, that came out publicly in July, and that was the accessing of the consumer grade key for the email product at O365. You know, hats off to the attackers and how they figured that one out, that there's a, you know, to quote the blog, an identity mistake kind of thing. I'm forgetting the exact blog quote, but there's a, you know, the identity anomaly that allowed someone to use a consumer key and access, hey, the real email on the enterprise side. And then hats off to the defenders that actually had custom rules to detect that attack. To me, Anton and Tim, that surprised me. Yes. That was the best attack. If you wanna call O365, that to me is a cloud-based attack.

Yeah.

Yeah.

Yeah.

We, we buy that as cloud on this show.

Yeah. Well, by the way, when you say cloud attack, okay. Somewhere there's a gray line as to, you know, somebody told me, hey, the MOVEit is a cloud-based attack. I'm like, maybe. You know, maybe depends on where the resource was. Anyway, the most surprising thing, Storm-558. That was hats off to the attackers, hats off to the defenders who detected it, and Microsoft moving quick to just deal with the use of those forged tokens. So that was the one.

And if you look at that, what to you stands out as that being especially different? You know, what should we learn from that? What's the key takeaway?

The key takeaway?

Or a few takeaways.

Yeah, first off, you know, I've heard a lot of different ways on, so how did the adversary get that attack done? And one of the things I've heard, and maybe it's true, maybe it's not true, is that they were able to dump memory on a developer's machine to get the token that they needed, meaning the threat actor targeted an engineer and got the token by dumping memory. Let's say that's true. Well, now you need memory that masks tokens, you know, or something like that, or something that's safer for your cloud-based environments. And so that's one thing you could take away. But it's always a great reminder when you see an attack like that, make no mistake, it's— I don't blame vendors for those attacks working.

Yeah.

I recognize it's a good lesson for all of us to see the efforts made and the R&D and innovation done by the offensive side because they tend to impress you in time. And that's— I do believe, you know, at this point you ask the question and everyone listening is like, so what was the question? But I do believe that's a good thing for all security professionals To have one incident they personally were impacted by, they need to respond to, that is by a high-order adversary. And you go, wow, that was creative. That was pretty good. That wakes you up.

But wait, so this is a classic story that's occasionally a very hard fight because when somebody gets breached, there's this chorus of little peanuts who scream, what a bunch of losers, they got hacked. But I sometimes point out that Hey, can't you imagine a case where defender was really good, but the attacker was just a little better? And to me, that's this case when the defender was good, but the attacker was just better.

Well, and I think the advantage does go to the attacker, right, Anton? I mean, every defender's got to show up on Patch Tuesday or whatever day you got to patch something and find all your installs of every vulnerable app. We're prone to human error operationally. That simple. And an attacker, a lot of times they're successful due to some of the mistakes we make or some of the times where employees are just absent from work. Whatever it may be, advantage goes to the offense.

Yeah. So this is a properly depressing take.

Sorry.

Go ahead.

Yeah, no, it actually makes sense. I think that we do need this golden depressing threat. Otherwise, we may lose our security professional cards if we are too positive, right, Tim?

Because—

I hope that's not how that works.

I think, Anton, the threat never goes away. Like, people always say, well, yeah, I mean, to take that through, there's always been crime. And right now, the best way to monetize— probably the safest way to monetize any crime is be 10,000 miles away from where you commit it, extort people. You have a somewhat pseudo-anonymous or anonymous currency in Bitcoin. And that's a great deal for folks that are like-minded to do that. And you're never going to get rid of espionage. And I always think— you wrote a book, I believe, called Cyber Warrior years ago, right?

Security Warrior, yes.

Yeah. So wherever there's conflict, even ideological conflict, you're going to have— in command and control, you're going to have a cyber warrior showing up. So there's always going to be an offense in cyber. We can push it to get better all the time. And then I often wonder, if we push it too far, Anton, do we start having personnel security challenges?

Yeah.

If you can't break in from 10,000 miles away and it just became easier to send people to get jobs, You know, our massive international workforce, we're going to have to start doing better personnel security. So it'll never end, but that's okay.

And listeners, what's fun, and this is a sneak preview for coming in the next season, we had a whole conversation and a whole episode coming about the intersection of cybersecurity and state security coming next year.

Yeah, that, that part. So let's switch to cloud. This one—

Back to cloud.

Well, back to cloud for a moment. Yes. But I think that I'm already excited about the cyber Intersections episode, so to say. So, the next question is kind of the who question. And I've met enough people who do more like security in the trenches who say, we don't care about the who. We just care about stopping the attacker, detecting. But in case of, well, we can afford caring about who, right? And sometimes, we often care about the actors and who they are. So, as far as clouds, compromises of things in the cloud infrastructure, Do we see any difference in who is going after these environments? I don't think this breach you described required a separate team on the offense side. I think it must have been the same people who compromised Active Directory on-prem or something. Or is it not? Are these different people? Are these different threat actors who go after cloud?

So that's a good question. You know, I think against O365, you do get the cream of the crop. You're going to get state-sponsored. And I think in Workspace too, because That's where our communications are. That's where critical data is. If you're a diplomat and you want to know someone's position before you go in and negotiate, you're going to go after O365 and email. You're going to go after Workspace and email. So take those off the table. What about cloud infrastructure and just in general cloud? Who's coming after that? I would agree with you, by the way. I never cared about who was attacking us early in my career. I'm talking '96, '97. I always felt like if somebody walked up on the street and started punching you in the face, Tim, you wouldn't be, like, blocking punches going, who are you? You would do something about it. However, replace the who with risk. It allows us to show up and help a customer by saying, oh, that's this group. Here's the zone of probable outcomes day one. It allows you to maybe, during those 3 or 4 days of the fog of war, if it lasts that long, because there's always a little bit of a fog of war at the beginning of an incident.

Yep.

Allows you to assess risk much faster, and the playbooks can be tailored to that group. So that's why who matters. I think there's a lot more commodity stuff outside of O365 and the data. I feel like in the cloud, it is different than hacking on-prem. And I feel like— I almost went this way, Anton. If somebody came to our red teams and said, hey, hack cloud, we would just go to Bitbucket and GitHub and look for access keys. I mean, that's the first thing we do is look for exposed credentials. If we wanted to hack the cloud, I think we might today still choose, let's hack them on-prem and find a document or find the Windows binary or find something that has the keys in it that aren't rotated and go access the cloud that way. So I feel like I would look for credentials that would get me into their cloud. And I think that's what the attackers are doing. Which is a long-winded answer like my first answer, which is, I think that cloud-based intrusions are mostly credential-based. If there happens to be a vulnerability out there, there's always, you know, hackers hack for 2 reasons. They either target a person or a company or information, that's kind of focused, or they target based on what the hell they've got in their bag. And we've got these vulnerabilities that we can exploit, let's just go rip it. I think that's not happening that effectively in the cloud. And I, I will tell you, most cloud adoption, people are looking at things like, do we have exposed S3 buckets? They're looking at the generic configurations and getting them somewhat right. I don't know. To me, it feels like credential harvesting and commodity threats hammering away all the time at your cloud infrastructure.

I had a depressing moment at mWise, and I think M in mWise stands for, like, you. Yeah, I don't know about that.

Hey, I didn't name it. It's a That's the word wise in it, so I don't know if I apply, so.

So there was a presentation from consultants, a couple of main consultants, and they said this beautiful presentation about cloud and the attacks. And I asked them, what really gets you in the cloud? And I expected some kind of magical answer. And they say, well, actually Active Directory does.

Yeah, yeah.

And I'm like, ah, ah. But like, that was their answer.

Anton, I would tell you, like, so I started looking through our forensic reports, and you should, you know, read some of them. Storm-558 is a real adversary with unique capability hacking the cloud. Let's take that out of it. There's a few others, but I feel like it really is commodity threat-based right now. That's what we're seeing out there.

And that tracks with what I've seen defending GCP customers at scale. I will regularly see a campaign letting it rip against a particular vulnerability where I see customer after customer getting popped on, say, the same container URI. Again and again, because somebody's got a new trick up their sleeve. Aside from mishandling your credentials, which has been a challenge since the dawn of time, what other mistakes or risks do you see people taking with their cloud that they maybe in 2024 ought not to be doing?

Sorry for being a bit of an ass briefly, but like, do you think that syncing your cloud access and IAM to Active Directory is a mistake? Is it too controversial? Maybe not.

Yeah, to me, one of the fastest detection mechanisms you have that you have a major problem is somebody syncing your Active Directory somewhere that it's not you. You know, that was one of the first pieces of evidence we had in our own breach. The reality is, I think the weakness in the cloud really is credential management. It's the identity providers and the fact that we're having too much exposed and leaked credentials. And what stinks is if somebody told me, hey, Mandiant, you're in charge of making sure our cloud's secure, I would do the most boring Security 101 stuff ever known to man. I'd be like, well, we need to rotate more frequently. We need to have a key not work on 5,000 instances, but one key per one instance. Tough break, guys. Least privilege, you know? And that is such boring stuff that people are falling asleep listening to our podcast, and you guys make these things fun.

We try.

I wish I could give you something, but it really is least privilege, reduction of account, access bloat, you can't be lazy. And that's the reality. And so you gotta start aggressive. And it's hard to retrofit when you started your cloud provisioning lazily, quite frankly.

That is for sure true. If you wanna be exciting about that, you can talk about all the cool AI and IAM Recommender and Policy Intelligence.

You're gonna drink if you say AI.

To me, identity security really is to identify the misuse of accounts in the cloud in a clever, language model sort of way. There's a way to do it. We've gotta have people notice, this is the first time this API has ever accessed this instance, whatever it is, or this account's accessing this resource. There's a way to do it. And that's why, you know, I know that there's some products like CrowdStrike's got an identity security product now that's doing very well because it's starting to do those things. We gotta look at Chronicle and Anton, you work there, you know, like, boy, do people want better. You know, zero days are going to happen. We can't do much about patching something we don't know about, but everything goes to lateral movement and credential use. We just got to constantly keep updating that detection capability in Chronicle, and our customers will be really well served by that.

One of the things we launched actually more than a year ago now is a detector in Security Command Center that does in real time the first time a service account uses a new API.

Perfect. Yeah, and just have like— I don't know if you have this. Now I'm impacting products too. Let's impact products. Just give it thumbs up, thumbs down icons, right? That logon, thumbs up. This one, thumbs down. Let the system learn what's normal and what's not normal in an environment. You gotta start somewhere, and that's a great security feature you just described.

There was a brief point about lazy, and I think that you talk to CIOs and boards, and you talk to, quote unquote, important people. And how would you give this advice to them without telling them, you're lazy, that's why you're so owned? Like, how would you tell a CIO, dude, you need to really focus on— Kevin is laughing, just for the record. Let the record—

He's like, dude, I've been in the board meeting. How would you tell them that your problems are because you're lazy?

Stop being lazy. And they're like, how does it go?

Yeah, that's easy. It's really not lazy. When we were talking about identity management, you know, and the provisioning of identities is so diffuse a company. It's done in pockets all over the place. It's very rare you find, I am the identity man, and I do identity and provision all of it. There are always some pockets of engineers throwing down an LDAP server, or getting into Active Directory, or kicking something out in AWS or GCP. And because it's so distributed, it's not lazy. It was probably the wrong— it's lazy by leadership to not try to impose their will, you know, security leadership.

Yeah.

But they are, so it's not really lazy. It's actually lack of knowledge of what an adversary is going to do down the road. It really is. You train people all the time, but you have these new 22, 25-year-olds getting into the business going, hey, let's just throw down an LDAP server and get our 42 people on it and do our own little project. And that's the thing that gets popped. So how do you rein it back in? Identity does matter. If you decentralize it, You still need governance, you know. In my opinion, in most organizations, when governance has failed, you've got to centralize till governance is successful, and then you can distribute it back out and empower business lines to do their own thing. And identity, for whatever reason, is just so decentralized at most organizations, it's hard to get your arms around. So it's not lazy. So that's how to explain it, Anton. Your people aren't lazy. They want to do the right thing. They don't know how to. And the right thing right now is you need to really think of each identity you provision as a key to your castle, period. Kept it. Yeah.

So I'm very lucky I get to talk about my mom on the show, which is different than talking about your— talking about your mom in therapy.

My answers are so long you skip 6 questions to get right to it.

Yeah, we did skip a few there, but I want to ask this one. So she worked in AT&T's disaster recovery a long time. Their, their motto was quote, preparing for the inevitable, which I always thought was pretty metal. What advice do you have for CIOs and CISOs to prepare for cloud breaches, which I don't want to call those inevitable, but next to it?

Maybe not of clouds.

Not of clouds, just in general.

And I do think the skillset to respond to cloud breaches are a little bit different than on-prem, right? Because we've had a 30-year run of doing on-prem. You can get your forensic tools out, you can access the system. It is different, guys. So, I mean, we do need to develop the skills, and I'm certain we have responders that are great at, I can reverse Windows binaries, but what the hell is the authoritative log file in Azure? You know, that's going to happen. And by the way, we may still be asking that question because there's a lot of different log files, a lot of SaaS-based programs. It's— you have to do a dry run. And I always know the governance guys hate when people pen test or red team. Because it always pokes holes. But the reality is, part of great governance is to test how you're doing. In my opinion, nothing provides more value than actively simulating or doing a purple team on your cloud-based resources and see how you do. It provides unvarnished truth. Most folks, the first time they respond in the cloud, are not going to be good at forensics. will not understand how to access the log files. You may have to go to your identity provider logs. You may have to go to 3 of the 4 Azure log files. You may have to get some O365 stuff. It's different. GuardDuty logs, you know, they'll get there fast, but how do you get the data? What does it mean? What do attackers do? And then again, I think a lot of it is you're looking for a lot of times valid credentials and the misuse of them. is the needle in the haystack kind of stuff where you just need a ton of data and you gotta start processing. Here's the invalid use of these accounts and, or this API key to access things, to access data. So long story made short, test, simulate.

It's great advice.

Have a 2-hour dry run with your executives. And during that dry run, you actually should simulate something that really matters. And boy, I could talk about this the whole time. Because you should simulate loss of cloud resources. Can you operate your business without it? You should simulate, we've had a successful breach, it's worst-case scenario, just to see the comm channels that need to open up and who takes charge, what's your unwritten policy, and what's gaps in, oh my God, we don't have an answer to this. Well, let's go sort that one out before we have the real problem. So that's what I would say, test it. And by the way, if you test it with the CEO involved, you will be testing it till he feels good about it or she feels good about it. And that's okay because I don't think it should last longer than 2 hours, Tim. I really don't. It's test it, see how you did, test it again, see how you did. And if the attackers are successful and you respond terribly, that's good. You want to know that, you know? You don't— yeah, you know, it gives you a prioritized list of things you need to take care of.

So, and testing implies not just like tabletops and simulations on paper, but it really means testing in metal, testing in real life in the field.

Absolutely. That's—

I just reconfirm it.

I would technically test it with really smart folks, and I would use external testers, but I would have internal folks give them a ton of knowledge. This is not a try to guess what we're like from the outside. If you have 5 really smart red team folks and they can't breach you in 5 days, you should feel pretty good. What I would recommend, though, is you simulate user-level access. Just skip it. Hey, they had a zero-day, they're in already.

Sure. Simulate that.

If they don't get lateral movement and get the crown jewels in 5 days, you have world-class defense. And I don't think it's an us versus them. I'm a big fan of purple team or even letting the blue team respond to it in real time. Let them do something so you get a real confidence boost that we can, if we see it, we can do something.

You know, it's funny at Google, we have a special way for the blue team to find out on, you know, say a Friday night or a Saturday morning, was that real or the red team that we caught today? There's a special emergency hatch for them.

Yeah, I get that.

So I had a kind of story that actually explains why in security I actually like detection and response rather than, I don't know, AppSec. And this story, and it's going to sound really strange, but it involves you.

Classic Anton.

Probably not in the '90s, but maybe in 2000 to 2001. I don't quite remember the date. I've heard you speak somewhere. Like, I think it was ISSA in Boston for some reason, because I used to live in Boston. And somewhere in that speech, you mentioned that you've never seen an intrusion being detected by an IDS or something similar. Like, I don't know if you'd recall what you said 23 years ago.

Yeah. It's probably DLP, by the way.

Anton, are you trying to hold our guest accountable for something he said 20 years ago?

No, no, no, no, no. It's a real story, and it's stuck in my mind for all this time because it's almost like I felt like detection is interesting because I thought we had this technology, but apparently this person who does a lot of IR in a lot of interesting places tells us that this detection technology we all love kind of doesn't really work.

Yeah, to me, a lot of it's like that game Frogger, where like some defenses are like the slow car lane and the frog just gets by it right away. DLP, in my experience, never detected a single exfil in my career when I was involved or I read the reports or I responded. Somewhere out there in the 10,000+ incidents responded to, Somebody probably got a DLP signal and went, hmm, that's bad. That was probably it. And again, Anton, realize the breaches we respond to, we have a bias. They worked and the breaches were at a scale and scope where people needed help.

Yeah.

And that's why I can tell you almost never did I have to sit at a SIEM to figure out what the hell happened. We just went straight to endpoint and collection because the SIEM didn't help us either because The only thing we got out of SIEM was the network-based stuff. You know, we needed to interrogate the endpoints to see what happened and what was stolen. But yeah, I think DLP and anything I said was numerically sponsored by at least the incidents we had firsthand experience on. So you might have caught me on a year where IDS was 0 for 50.

Yeah, but that's also like deep in many security professionals' hearts. They sort of know it as well. Like a lot of detection tech, Is not how, like, look at the friggin Amtrans report, how many breaches are detected externally by nobody at the company. It was like, I don't know, whether 63 or whatever. Like, it's not 3, it's 63. And it's not 2000, it's 2023. So, it feels like we, well, you specifically are great at IR, but we collectively suck at detection.

Well, listeners, on this cheery note, I'm going to ask Kevin our closing questions.

Yeah. Oh, okay. Anton, I guess we're at the time for sure.

No, Anton is taking us into a very dark place and we're at time, so I got to wrap us up.

We got better, Anton. Trust me, we're better at this than ever before. We're closing the window.

Okay, that's about enough, Timmy.

That's actually true. That same report says that dwell time is way down.

Yes, that's exactly right. We are way better.

Yeah.

Do you have recommended reading for the audience?

You know, this is the one question I prepped for because I thought about it. Depends on the audience. But I would tell you this. If you're a CISO, you have to read the civil complaint filed in New York, basically the SEC versus SolarWinds and Timothy Brown.

Yeah.

You have to read that. It's mandatory and you have to read every damn page. I am still, by the way, I'm in my second read and I'll reserve my opinions till another time. But boy, will that change how CISOs operate.

Well, on a remarkably cheery note to end our season.

Yeah, great.

Kevin, I can't thank you enough for joining us today.

Boy, do we depress people sometimes.

Oh, gosh. But listeners, the good news is dwell time is down, so we're getting better. Kevin, thank you.

Sam and Anton, absolute pleasure.

And now we are at time. Thank you very much for listening and, of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity, Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @Anton_Chuvakin and @_TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode. Bye.

Security!

Security!