EP156 Living Off the Land and Attacking Critical Infrastructure: Mandiant Incident Deep Dive

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior PM for Cloud Detection, Investigation, and Response in Google Cloud SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with your hosts and the rest of our listeners on our Cloud Security Community page. You can find the link to that in our show notes and on our LinkedIn page. Anton, this episode I think is a delightful follow-up to last week's episode because we go from the theory of how nation states relate to each other in cyber to the very, very real practice of how 2 nation states are relating to each other, both in cyber and in the physical world.

I think you win the understatement of the year prize with that in regards to nations' practical interactions. Unfortunately, as we know, it involves tanks, missiles, and cyberattacks, right? And cyberattacks. So, but yes, it was a very geopolitical episode. And I think, did we ever mention cloud even once?

No, and I'm sorry, listeners, that it's not a cloud episode, but I think it is a worthy entry in the Cloud Security Podcast by Google because we're talking about operational technology and living off the land and the challenge of forensicated environments that have been wiped and actor attribution and all of these things that are relevant for cloud security. And so, you know, we're not going to talk about IAM policies, but we do talk a lot about security. And so this is maybe the cloud security episode.

Yeah.

Cloud Security Podcast by Google today.

It was the one. We were almost like— I think a couple of times, both of us had a possibly urge to say, Sandra, can you say cloud just so that it's a cloud security podcast? But ultimately, the flow was so good that none of us did this.

And the content was so interesting. And so with that, let's not waste one more second. Let's welcome today's guest. Today, we are joined by Sandra Joyce, VP at Mandiant Intelligence. Sandra, I am delighted to have you with us today. Of all of the Mandiant folk I've met since you have all joined us at Google, Your orientation towards the mission has been like really meaningful to me, and I really appreciate that. So I want to talk today about a pretty interesting report that came out last year on a pretty complex incident. And this episode we're releasing right after an episode that we did talking about state sovereignty and cyber. So this is just perfect. Could you give us the 2-minute overview on the incident that you reported on last year and we'll dive in?

Yes. And thank you so much for having me. First of all, it's a pleasure to be here. And I really appreciate your comments about the mission because when you're in security, if you're not about the mission of security, then you might lose sight of what it's all about. And this report that we did is one of those examples of really understanding what it's all about. And I'll tell you a little bit about it. The threat actor behind this one is known as Sandworm. And so, Sandworm is the sort of, let's call it the nickname that has been given to this threat actor years ago by actually somebody who works for me named John Holquist, our chief analyst. And years ago, we were tracking activity that was tied to Russian military intelligence organizations. And Sandworm has been very active since the invasion that started in Ukraine by the Russians. And they've been carrying out all these wiper attacks. They've been carrying out major attacks on Ukrainian critical infrastructure. But most of them haven't been that successful. But actually—

Hmm.

What we saw is one that, that was successful. And so What we outlined here was that Sandworm actually managed to slip through the net on this one. They used what's called living off the land techniques on OT systems or operational technology systems, and they were actually able to evade detection by defenders and maintain access for 3 months before executing their attack. And so what ended up happening was they were able to shut down power plant operations, which was, you know, power to customers, to the Ukrainian population. For a brief amount of time. And that actually was coinciding with active and live missile strikes that were incoming. So a lot of speculation around that was, were they coordinated or not? We don't really have a lot of hard evidence that they were coordinated, but they were happening at the same time. So it's very interesting. 2 days later, they deploy a wiper and wipe a lot of the evidence, let's say, of their activity. And that was something that allowed them entrance to the OT systems that they were disrupting. So that's basically the short version of the story.

That's wild. And that doesn't seem like a coincidence to me. That seems like there's some coordination happening between digital and physical. And that's pretty interesting to see.

Yeah, it's, you know, you want to, you want to always be careful. What do you have evidence for? Right. There's always a question in the, in the practice and the tradecraft of intelligence, right? What do you have evidence for? And this, I would say, is— to us, it's extremely interesting.

So you also mentioned some of the things that I guess most hardcore security audiences would know, but some of the cloud audience we have may not know. So you mentioned living off the land, which I guess occasionally gets contentious because some people treat it as not using malware. But you also mentioned attacks on OT, so control systems, something that a lot of people assume is not common these days. So can you explain this to the possibly cloud audience? Living off the land.

For the uninitiated, yes.

Yeah, please.

Yeah, no problem, of course. So when we talk about living off the land techniques, what we're really talking about is a threat actor is exploiting the native functionality of a system. So they use this to do many things, to move laterally, escalate their privileges, and execute tasks rather than using malware. What's interesting is, in this case, they also used malware, but living off the land purely is that rather than, you know, you deploying this purpose-built malware that can be detected and there are signatures on it, instead of doing that, it's using that built-in functionality. One example of this would be PowerShell.

Hmm.

And that's what was used. And so when we talk about this being used for OT, you know, when you think about OT or operational technology systems, this is— think of things like, You know, water systems, manufacturing—

Valves, pipes, sensors.

Exactly. Things that are running industrial processes. And, you know, unlike IT systems, which tend to be kind of cornered off from the internet, or IT systems that tend to interact with the internet, OT systems are the ones that tend to be cornered off because they're running some of the most sensitive and critical systems, right? They're the things that are like the switches And the you know things that are monitoring nuclear power plant activity, or how much chlorine is going to be released into water supply, or how much chemicals are going to be put here, and they they create that sort of industrial control system. You know ecosystem. That's why when you combine living off the land techniques and OT systems, this is something that gives us a lot of concern.

Sure, because it's hard to detect and the consequences are dire.

And it's also hard to investigate, right? Because there's no— there's no outline with telltale signs and code. And ultimately, they use a wiper in the end anyway. So even some of the history stuff, command history, may be gone, right? So just as a side question, how do you typically investigate cases like this, especially living off the land combined with wipers combined with fairly uncommon systems?

How do you Yeah, how did you even do this?

Yeah, exactly.

How did you do this? All right, so there's a few things. So number one is, you know, we were called in to help remediate. So we, we actually can help. We've had partnerships with the Ukrainian, you know, government, and we're able— we were— we've been there, we're helping them to defend. So it's a bit more of that access that we have. But remember, when they deployed the malware that wiped this, there was actually a 2-day period where they didn't actually wipe them out, you know. So they, they lagged for a little bit. That gave us an opportunity, right? And so a lot of this has to do with perhaps an unintentional lack of coordination on the part of the Russian military on this. But it would have been a lot harder had they wiped it in that same day, right, in that same moment.

Mm-hmm.

But you're right about everything that you're talking about with living off the land. It typically denies us that That most powerful tool of tracking these threats. And so, you know, I think about, you know, we've seen this before, though living off the land is not something that's new, but we've typically seen it in other circumstances. For example, I remember a few years ago, a North Korean threat actor that was doing a bunch of bank heists all over the world. Bank of Bangladesh comes to mind.

That was a big one.

That's exactly— yeah, a lot of people remember that. That threat actor, what they were doing is they would be persistent in an environment for like 2 years. Then when they were ready to move the money, they would actually deploy these wipers, right? And they would wipe, and then it was extremely hard to do the forensics after that because, well, everybody was trying to figure out what was going on with the wiper activity. And in the meantime, they're moving money out the door, right?

Yeah.

So it was a bit of a disruptive kind of thing. That's not what happened here. Here, they were using and abusing the native functionality and then had a space of time, and then they deployed the malware. So very different.

The way you're describing the wiper makes me think of, like, a bad guy running around and throwing a smoke bomb and dashing out the door.

Yeah, it's like, light it on fire after you rob the place or something. Yes.

I mean, it's $90 million, I think. What was the number? I think it was something—

Big number.

Large, nice— what is it, 10-digit number or something? 8-digit number. Yeah.

It was an incredible accomplishment on the part of a scrappy country like North Korea, where it was actually really hard to move the money because sooner or later they realized that these $10 million donations to orphanages in Korea out of nowhere were starting to light up some of the suspicions of internet, SWIFT and all those.

Mm-hmm.

So some of that got stopped based on process, but Yeah, those, those were some pretty, you know, scrappy moves on the part of North Korea.

So I want to roll back to Russia just for a minute here and ask another speculative question, which you can rightfully tell me, stop speculating, but I can't resist. We saw a cyberattack coincide with physical attacks. Did the attacker risk tipping their hand here? And if so, could we, if we'd caught this earlier, have understood what was going to happen in the, in meatspace as opposed to cyberspace?

It's hard to say. You know, when you're doing battle planning and mission planning, it's something that is happening within hours. Yes, you could always, you could say that it's gonna tip off, but really, if you think about it, in the Ukrainian battlefield, missiles have been ongoing. These things have been just happening one after the other. The wiper activity that we've been seeing in the first 4 months, for example, of the invasion, was more than we'd seen in the past, I would say, something like 4 or 5 years combined.

Oh, wow.

So the activity was— is ongoing. Now, that trickled off, like the wiper activity that we saw in the initial invasion period trickled off, it started to go to more peripheral organizations, like, you know, instead of the Department of Defense, it was more parks and recs, you know, trying to kind of go to maybe less well-defended places. But, you know, I would say that they were definitely hiding behind the fact that The assaults were constant, and it wouldn't have necessarily been an indicator of what came next.

Sure. If what comes next is missiles, knowing that missiles comes next doesn't change the picture.

Right. And that's probably the bane of intelligence sometimes, when if you know something that you can't do all that much about, then is there a value in knowing, right? Like, I'm sorry, this is my bias from my previous job, is when sometimes clients would ask about, should I get more threat intel? And we ask them, Well, can you do anything if somebody tells you you're going to get hit by the biggest DoS ever in 3 hours? They'd be like, not really. Then why would it help you to know that if you can't act operationally? There's no agility.

That really is sort of the holy grail, let's say, of threat intelligence, though, is can you do the forward-looking early warning, right? If you can do early warning in a place, you know, as specific as possible, that is a very, very difficult thing to do because You know, the tradecraft of threat intelligence is more about risk and probabilities of things happening because we don't have a crystal ball. But we can say things like, we have seen this threat actor use these techniques before, and we have seen evidence that they're targeting this industry. So if you're in this industry, make sure you're hardened against these tactics, right? So it's more of using the probabilities and making assessments based on what we're seeing, you know, before, what we're seeing, you know, today, and the evidence, and then making those sort of strategic over-the-horizon projections. That is as much art as it is science.

Which, by the way, brings us to the other dreadful question, the A-word, attribution, right? About art and science immediately made me think, well, we have an attribution question somewhere in there. And it's next. Yeah, and it is next. So, I mean, I don't even know where to start, admittedly. I've met enough security professionals who sort of frown, even in 2024, about attribution being like all that valuable to their teams. But in this case, in this kind of higher stakes game, we can jump to conclusions, of course, which country benefits the most from this. But how could you talk about how your team handled attribution, how robust it was, and how sure are we really in this case and maybe in others?

Yeah, I mean, this attribution for us is a very delicate process. And we have the benefit of having something like 15+ years of cybersecurity-related threat actor activity, you know, targeted intrusion data that we can consult and look at. And that's really important when you're doing attribution because it's really this sort of delicate process of looking at clusters of activity and then bringing them in, sort of merging in Clusters of activity, when you see those overlaps in things like behavior, infrastructure, malware, the actual targeting. And when we see that forensic evidence, we can say, oh, there is overlap in all of these different areas. And then, we start to merge in those clusters of activity. So in this case, we had assigned what we call an uncategorized cluster of activity or group, UNC3810. And this was based on some of the malware overlap and some of these other factors. But when we did the broader analysis, we were able to see that there were overlaps in all of the areas that we talked about with behavior, infrastructure, malware, et cetera. And we were able to merge it into Sandworm. And so also against the backdrop of, in a battlefield situation, Russia's not necessarily concerned with hiding who they are, right? We already know who the adversary is.

Yeah, sure.

So Sandworm has a history of conducting these types of OT attacks. If you think about Ukraine in 2015 and 2016, that was Sandworm. They have a long history of doing this. So it's not necessarily this large analytical leap. There's a lot of forensic evidence, and that's basically the process that we go to. It can take a long time.

So on that, on that note about OT, OT is a really interesting aspect of this attack. And we can think all the way back to like Stuxnet and even things before that in the OT world. But here, OT in particular, where do you think people are underprepared for OT? Where should we be thinking more about this? What's the future hold for OT?

She may be on the podcast for another 6 hours, though.

You don't know. That's true.

That's a risk. Yeah. Let me start at the beginning. No, I, I, you look, here's the thing. When most of these industrial processes and power plants were built, they did not necessarily have the security mindset that an IT infrastructure would have. Right. So, therein lies sort of this, you know, the OT defensive landscape. It really does continue to kind of lag far behind the typical defenses of an IT environment. And that is simply because of the nature of how those things were built, created, and where they exist, you know, out on a— in the middle of the ocean, you know, with oil extraction, or they're in a water treatment facility or that kind of thing. So, right now, you know, what we're seeing is that These OT attacks are kind of rare. It's actually really hard. Like, in what we saw with this living off the land technique is these threat actors needed to know quite a bit about the actual system that they were working in. So they really had to understand and have a very high level of familiarity with the operations of this place, which is also an interesting feature of this. It's very hard to scale something like this, like an attack like this. Because of that, high level of technical knowledge that you need to have.

Hmm.

But, you know, over the last decade, we've seen Russia's capability grow in this space. So I mentioned Black Energy in 2014. I mentioned the blackouts in 2015. You know, they started to then automate those attacks and malware like Industroyer in 2016 and in 2022. So there's a lot of evolution here, but OT continues to lag behind. So in saying that, we really need to invest more in that. some of the basics when it comes to cybersecurity hygiene in OT environments.

I feel like if you gave me root on a domain controller on an oil rig, I would have no idea what to do. But if you gave me root in a cloud environment, I would do some damage. So this makes a lot of sense to me that it's genuinely hard to do OT attacks well, and extra scary that they did do one well.

Well, it's also not just technical knowledge, it's site knowledge, right? It's knowledge of technology in this particular place. So, it's almost like, I think that's the angle that Sandra didn't focus on, but I think it's also kind of the scary part is they knew how this utility operated, not just how utilities operate, right?

Right. And I think that there, you know, we think sometimes we might have a talent shortage. Well, bad guys have talent shortages too. So, you gotta imagine trying to recruit for this. I mean, if you're a government like Russia, You may be training and you have groups like, you know, the Research Institute of Chemistry and Mechanics, where we saw that the Triton malware was originated from. So in some cases, they have systematic learning and training by some of their, you know, top talent in this. But it's still a rare skill. And to your point, it needed to be tailored. And that's what this was.

Gosh, it never occurred to me that there'd be a talent shortage on the other side of the fence. That's, that's so funny.

Yep.

Thank goodness.

Right? They've got bosses, they've got bills, and they've got talent shortages. One good example of this is in the ransomware community, in the underground, they have affiliates that they recruit. And, you know, there is a bit of a fight to make sure that you can recruit the right affiliates to join your efforts, right? So they do fight for talent on the other side, as well.

Wow. I'm glad there's market pressures there, too.

Yeah. At this point, I'm like, yeah. But if it's your sole security measure, that bad guys have talent shortages, we are probably safe. I think that takes a depressing spin. So let's go back to the case, right? And I think that you also mentioned that it wasn't just like you showed up on site 5 minutes after the attack. You had— Mendiant has a long history working with Ukraine going to, you know, pre-war years. So could you share a bit more how this came to be and maybe some of the highlights of how we are helping defend the country?

Yes. So, as I mentioned, you know, we have the belief that we need to be really adjacent to some of the biggest threat actors out there. So, Mandiant has always had a presence in Ukraine. We have a presence in Taiwan. We like to be adjacent and be able to learn, right, what the latest techniques are. It's long been known that Russia deploys a lot of their up-and-coming malware or techniques against Ukrainians and in the Ukrainian environment. So, We need to be there. So that gives us the benefit of establishing those relationships, but also it allowed us to be there literally the minute that the invasion started. We were already being contacted and were ready to roll to support. Some of the stuff that we did right off the bat was helping the Ukrainians defend against infiltrations into their civil defense systems that were designed to warn the population about incoming missile strikes. And the Russians were—

goodness.

Yeah, they were trying to turn it off. Right. And we were able to go in and help them to kick them out. So it's things like that where we're able to be there and defend when the time came.

Man, talk about the mission on that one. That's wild.

Yeah, that was wild. Actually, as the invasion started, about 300 Mandiant employees were spun up to support. And many of them, in fact, I would say most of them did it just because they wanted to, and nobody asked them. And I think to your point, you know, mission is so important. Sometimes a Mandiant consultant, for example, is the only thing standing between a Russian threat actor and their entire network getting wiped. And that's just— it's a privilege, and a mission is a privilege, right? And I think that that's probably why Google acquired Mandiant, because Google has that same sense of mission.

Mm-hmm.

It has that same sense of, you know, what is the right thing to do right now? And I think those core values are prevalent in both organizations. And now we're one, and that's a great outcome for both of us.

Respect the opportunity, respect the user. It's the same story. So I hate to do this because Anton's right, we could talk for another 6 hours, but I have to ask our traditional closing questions. One, do you have recommended reading? And 2, do you have advice to help organizations prepare for attacks on their OT?

Yeah, I do. I have both. All right. So there is a book called Sandworm.

Ah.

And it was written by Andy Greenberg, and Mandiant is prominently featured in this because it's the origin story of this Russian military intelligence group, how they were started, and then sort of the investigation of what they have done over the years. Our chief analyst, John Holquist, is, is one of the main characters in it. I recommend reading it to learn more about the Russian capability in cyberspace, which is tremendously sophisticated. And then for organizations that understand that they have an OT environment that can be susceptible, I would say that a couple of things. Number one, make sure that when you're designing a security program, there's a temptation to do the OT part and then the IT part separately. A comprehensive security program looks at both. And the reason is we have seen the IT portion influence the OT portion, even if The OT portion is cornered off, as I mentioned before. One good example of this is in an attack on the IT payment system, which regulated how much power was going to go to customers, caused that power to be shut off in a case, I believe it was in South Africa.

Wow.

So looking at IT and OT as a comprehensive view is one way to do it. The next thing is, you know, making sure that we have an eye on how Not just nation-state actors are looking at these vulnerabilities, but the increasing role that cybercriminals are using to exploit OT. So that's one of those over-the-horizon pieces that's coming that we need to be very attuned to. So, you know, starting with the basics of cyber hygiene is always a good thing, but I would start with that and then think about the other pieces as well.

Well, Sandra, this has been tremendous. I can't thank you and the group enough for the work that your teams are doing and Thank you for joining us.

Thank you so much for having me. It was a pleasure.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at @antonchuvakin and @tamiyazaki.

Anton_Chuvakin and _TimPico.

Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.