EP257 Beyond the 'Kaboom': What Actually Breaks When OT Meets the Cloud?

Security.

Hi there. Welcome to season 6 of the Cloud Security Podcast by Google, now in video format on YouTube. Welcome, Anton. It is great to be back. Your hosts here, listeners, are still myself, Tim Peacock, one of the SecOps PMs, and Anton Chuvakin, senior staff in Google Cloud's Office of the CISO. You can still get this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. You can also get it at YouTube, youtube.com/@cloudsecpodcast. If you want our content delivered to you piping hot every Monday, please do hit that subscribe button. You can also ring the bell, hit the subscribe button, whatever it is YouTube hosts say. We're still learning how to do this video thing, listeners. Or you can argue with us on the Google Cloud Security Community page. Anton, we are talking about things that go boom, trains running off the tracks. This is a crazy way to start the year for a cloud show, isn't it?

Correct. And by the way, there's not much AI in this episode. So for those of you who are tired of us becoming— I don't think there's any. Yeah, fair. I'm just trying to soft sell it. But I feel like this episode is a really useful refresher to people who sent us jabs by saying you should rename the show into AI Security Podcast and all that. No, I feel like we are still Cloud Security Podcast. I mean, if you think we should rename the show, tell us. We rename it into something. I don't know, maybe Security Podcast or Google Security Podcast. I don't know. At this point, I feel like we are still the Cloud Security Podcast by Google, but the topic today is very much about something not in the cloud, not only in the cloud.

In fact, it's things explicitly not in the cloud, I would say.

Right. Some of them are on rails, some of them are in the woods, some of them are on pipelines. Yeah, pipelines.

Siberia.

Some of them are in Siberia, perhaps, but this one was not far.

Was that explosion near where you grew up?

Nowhere near.

No, nowhere near.

I don't think anybody even knew about that.

Well, listeners, if you want to know more about Siberian pipeline explosions, how to prevent them, let's turn things over to today's guest. Hi, listeners. Welcome to season 6 of the Cloud Security Podcast by Google. Impossible to believe that we've done already 5 years of this. I have with us today Waffles, my dog, who's now going to leave us so that we can have our actual guest, Chris Sistrunk. We are delighted to have him join us for the kickoff of season 6. Chris is a technical lead in Mandiant's OT consulting group.

Hi, everyone.

Chris, when I hear OT—

Thanks for having me.

Ah, thanks for joining us, man. When I hear OT, I think of stuff like Stuxnet targeting PLCs, the CIA in the '80s tricking the KGB into blowing up a Siberian gas pipeline. Is that what we're talking about? Is this field like super kaboomy or is there a whole nother class of attacks beyond the kaboom that we should be thinking about?

Well, There's a lot of that kind of hype, the Kaboom, the Black Hat movie, you know, what was it? Die Hard 4.

Oh yeah, sure.

Things exploding. You know, those are really great attention getters, but OT is basically an umbrella term for anything that has a cyber component and a physical component. You know, it could be anything from an elevator, an escalator, to a pipeline control system, water SCADA system, or, you know, electrical utility power grid. So now when you talk about, you know, hey, these Stuxnet-type attacks, they just don't happen every day. You know, it's—

No, that's like a generational thing right there.

Yeah. What was it? The old kooky newspapers. This man will turn a computer into a bomb. You know, it's just not there. And so—

Okay, so what is there?

Yeah, so what we have is all of these systems that are increasingly connected. You know, they have more computer brains, more computer chips, and the managers and the middle to upper management, even the CEOs, hey, they want to know what's going on. What's the manufacturing system producing today? What will they produce tomorrow? How do we get that data faster? Then we can connect, you know, networks to the IT network, to the cloud, to get all this data. And it's really great, but it's also, hey, well, that means it's connected to where it wasn't. And some of these things could, you know, let, if it was not secured properly, you know, a ransomware attack to happen or a commodity malware to show up. So that's, you know, that's kind of A little overview there. Hopefully that answered your the the theme of your question there.

Yep, for sure.

And I think that it's interesting where cloud gets added to this because I I am old enough to remember times when people discussed connecting OT or whatever other systems of that sort to the internet. Forget cloud. And there was a group of people who said no no no no we're gonna have an air gap. We we're never gonna connect blah. to the internet. And today, we are on a cloud security podcast. And of course, a lot of OT control systems are now in the cloud. So, explain the whole thing. Like, how is this change? How does it change things? And what are the blind spots and challenges? Because we kind of went pretty quickly from, oh my God, don't connect to internet, to, oh yeah, that's cloud-based. What's the story here?

Yeah. You know, I grew up as an engineer in RS-232 serial world, right? Old man yells at cloud. All right, we'll never connect it to the cloud. But we are starting to see that actually happen.

Wait, wait, wait. You were in the don't connect it to a computer. You are in the step before don't connect to internet. Don't connect to the computer is what I'm hearing.

Yeah. Well, they had computers. But there are old mainframes that like OpenVMS, DECterms, those kinds of things, not your everyday Windows. Now they might've had some old Windows and of course there still are, uh, but you know, the cloud is just another computer. Uh, so just really big, right? So it's just a matter of engineering. And there's engineering needs for control systems. And now one of them is, hey, we have all this data. Now we have this computing resource that can crunch these numbers to solve a problem we couldn't solve before. Or they want to leverage some application like AI quality control on manufacturing process, for instance. Heck, you could even buy a serial To cloud converter. You can buy these. It's a serial device that has a cellular modem in it. I was like, but that's what they market it as, a serial-to-cloud converter.

I need to pause for a moment and like process that because this is just such a— I think it's if you just discover like a wormhole that leads from 2020 straight to the '80s. I think that's what the thing is.

It is a time-traveling device. Wait, I thought we had that wormhole already and it was your sense of humor.

No, it's the '90s. My sense of humor has the '90s jokes, but this one goes straight to the '80s.

Yeah. Yeah, so, you know, the air gap, it really doesn't exist. It really only really exists in a very, very few, very few places. It's kind of a misnomer. It's not really an air gap. There is a connection somewhere to some other network most of the time. And now it's starting to include cloud in some way. And some, some OT is hosted in cloud now, and some of, some of it can't be because of the high-speed requirements. the low latency requirements. And if like telecom goes down or the internet service provider goes down for whatever reason, squirrels or a failure, DNS or something, then we now have no longer access or control over that system that's in the cloud. So where we're starting to see a lot of migration is having some things in the cloud as a backup or a tertiary control system, and then still having this hybrid, like, on-prem solution. So we're seeing a lot of places go that way, manufacturing especially, going into a hybrid approach to migrating OT to cloud.

Okay, so speaking of OT and cloud, we've got this nice, you know, MITRE ATT&CK matrix for cloud that everybody more or less gets. How is it different in OT? Like, how's the attacker goals different in an OT world?

Well, the attacker goals are going to be the same. 99% of, and this is just, you know, I haven't actually calculated it, but I would presume based on our experience and the other experiences out there from other responders, you know, attackers are going to go after what's widely deployed and that's Windows and Linux. Servers and workstations. They're going to target those, and they may target, you know, firewalls or web applications, but ultimately the goal is to get to the crown jewels, which may be a domain controller or some other workstation. And once you get access to, say, an engineering workstation in an OT environment, that's a Windows machine probably, and then it's game over at that point. You don't even have to write OT-specific malware. You can just Do what you want. But there are some threat actors writing OT-specific malware, but it's not, not very common. That's why we say 99% of it is going to be in the Windows. So regular, your MITRE ATT&CK that covers those bases, covers the enterprise MITRE framework, ATT&CK framework, will cover all of those things. The cloud will cover all of those things. They may have Their CNC in a cloud environment. Now, the MITRE ATT&CK for ICS, those are the PLCs, the embedded devices that are not Windows, not Linux. So our approach is to take whatever a threat actor may be doing, and in the analysis after an attack or after an incident, we go back and say, hey, what components of what pieces of MITRE frameworks did they go through? Did they use part of the cloud? Okay, what are those TTPs? What are their TTPs in the enterprise? That's where it's most likely going to be most of the threat attack vectors and their tactics. And then the last percent, the last 1% is going to be in the MITRE ATT&CK for ICS. You know, in cases that we've seen like Stuxnet, Triton, you know, the Ukraine attacks and those things. So.

I mean, this is kind of the fun part because I don't want to reduce much of OT hacking to IT hacking, but there's absolutely IT hacking involved. Like, there's no real need to hide from that. Like, it's a lot of the quote unquote classic things will bite you if you're in that land, which makes me think of the skills.

Sorry, go ahead. Yeah, it's like living off the land. That's a tactic to evade detection. Just blend in as a normal user. And if you're a normal engineer user, you don't have to use malware. You just use their Windows credentials or an operator credential or something like that. We've seen that tactic being used for quite a long time now. Obviously, it's come in the news recently with some nation-state threat actors specifically using living off the land attacks, but this is even what our red team or red teams at Mandiant do. You know, why evade detection if you can use the features? You can, you don't have to write malware that evades VirusTotal or—

Right, right. So I love that, Chris. I love the idea of red teaming with us. I heard a rumor that the team was training in a very particular way. Could you tell us about that?

Yeah, so we, our OT red team, they're a really amazing group and we were hired to basically hijack a locomotive.

Really?

And yeah, yeah. That's a dream. Was it from cloud?

Was it running? It's a physical locomotive. It's not a cloud locomotive.

No, we actually— they went on site and they had the locomotive in a test yard, so it's not on a live railroad track. It was in a test environment, and then they actually had locomotives front and behind it so it couldn't run away.

That's important.

But we were able to get God mode, basically. And we could turn it on, say that it's going 100 kilometers per hour, and it wasn't going anywhere. And we were able to honk the horn.

You were able to honk the horn?

Yeah.

Oh, okay.

To me, that's cool. If you can do it, we did it. And so we were able to move it forward and backwards, you know, remotely from the laptop standing next to the train.

Oh, that's so cool.

But that takes some specialized knowledge. We have several experts on our team that know the rail industry.

Sorry, you have rail experts in the group?

Yeah, we have a rail engineer that's been doing that for 20 years before they came to Mandiant.

Wow.

So yeah, I mean, we have experts on our team that have worked in industry, uh, like myself. I worked in the power utility industry. So we have a lot of those type of experts that have deep knowledge of these systems. So, you know, we are power users that live off the land because that's what we used to do.

Power users.

And we— yeah, I mean, it's like we know the physics of these systems. And so let's say you get a script kiddie that's trying to knock over a water PLC that's connected to the internet because there wasn't any security when that system was installed and they didn't really know better at the time. So that's a different capability.

Yeah.

Going from pushing buttons to actually Evading detection and making things move when they're not supposed to move uncontrolled, that's a whole different— that's how we approach our red teams for OT. We also know what's safe. Safety, we all stack hands on the rules of engagement. We're not going to willy-nilly just go hack a train that we haven't been hired to do so in a safe way.

Right, right, right, right.

I would hope not.

So I want to bring us back to cloud a little bit because in theory, this is still the Cloud Security Podcast by Google. What's the role of cloud in this modernization journey? Like, is there a way to get more security by going through cloud rather than like less security? Because it kind of sounds scary that the trains are connected to the internet.

Yeah. So, you know, it can be any type of OT that leverages cloud a lot in manufacturing, you know, building automation, there's so many others, some upstream oil and gas, there's some electric, like, generation as a service companies out there that are running their wind farms or solar farms from the cloud, which is great. What's cool is cloud, like what we do at Google, is was built with security in mind from day one. Whereas control systems, like when I was in electrical engineering in college, we weren't taught about cybersecurity at all.

Right.

You know, and the login was like password, admin password or something like that. Like the capability was not there in those control systems because they weren't connected to anything else and they didn't need to have security. Other than physical security. But now, since Stuxnet, since all of these other known incidents, that whole paradigm has changed. So used to, we would say, as a rule of thumb, ICS or OT security lagged IT security by 15 years. You know, so that gap has closed significantly.

You mean to tell me that the things that can explode or drive 100 kilometers an hour on the tracks are behind the rest of security? You would think we would want those things ahead of the rest of security.

Well, again, those are the wow, hey, wow factor, a little bit of hype behind that, of course. Because don't forget, engineers have been making things explode on their own for years and trying to avoid that. So like, you know, there are pop-off valves, like your water tank in your house has emergency systems that can, that have to fail after, even if the electronic system fails.

Sure. Even if somebody puts my water heater on full blast, there's still a safety valve.

Absolutely.

Okay. That makes me feel a little better.

A little.

Yeah. So, yeah. And like squirrels are the number one threat to the electric grid in the United States anyway. You know that they're, they're causing power outages all the time.

I heard a funny story once from an electrical grid guy where they had released a new generation of transformer. And the new transformer, instead of having the status light on the bottom, had the status light on the side. And because it was on the side, Billy Joe, half a mile away, at night, would see that LED and think, I bet I can hit that transformer, and shoot it. And they had all these outages of these transformers Because the light was on the side instead of on the bottom. And so there are really interesting OT threats out there that you might not think of if you're just thinking of them as a cyber system.

Yeah, that we, you know, we do have the rednecks, and I'm a redneck from Louisiana, you know.

I'm from a very rural part of New Jersey. It's in my blood too.

Nothing, nothing, you know, hey, let's go do this. Oh, we probably shouldn't have shot that. And those are the ones we're not worried about so much as the, the, you know, extreme, uh, politically left or extreme politically right that are, that are trying to create chaos. Um, and so those we have to worry about. Uh, but for the electric, uh, system, you know, that is something since, you know, over 20 years we've been worried about cybersecurity. And that really plugs into cloud now because cloud is helping accelerate the grid, like all our data centers and the others, is creating more electricity demand. So we have to be partners with electric utilities, especially in cybersecurity. So recently we just participated in GridX 8, you know, as And even in us as Google, you know, like, that's important to do.

Mm-hmm.

Because if a power system goes down, well, what does that mean for data? And do we have to, you know, make sure we plan for that ourselves? Of course. So we're partners in this, in some of these critical things like the power grid, water, transportation. So the cloud is going to be part of how we accelerate You know, more secure systems. And as, you know, some things are moved to cloud, they can restore faster. They can be more threat agnostic where, hey, if you do have backhoe attenuation and, you know, somebody digs into the fiber, well, we—

Sure.

We're still able to operate because we have something on-prem as well. So that hybrid approach Or control systems is gonna be even better, I think, in the future.

Backhoe attenuation, that's a funny term, Mark.

Yeah, you know, for some reason it rang a bell. So maybe I did have, you know, beers with OT security people at some point because it just, it just like, it reminded me of something I've heard before. So probably my paranoid fantasy anyway, but who knows. I wanna touch on one more question before we start thinking about closing. It's kind of the visibility And challenges for detection and for sort of operational data versus security data. Occasionally, I see people show up from a security lens and say, oh yeah, we'll just connect this OT system to a SIEM and we'll detect the threats. And I'm like, yeah, no, because even though the data may come, you would not know what it all means in the context of the actual system running out there. So, just maybe a broad question about the What's interesting about detection and response for OT?

Yeah, that's important. You know, engineers and operations folks have been doing, you know, detection and monitoring ever since control systems were invented. Like Charles Babbage invented the very first locomotive recording trailer to a recording car to record what happened on the rail. And that was in like the 1800s. And so, you know, if we move to say the 1960s, we still had the beginnings of some digitization of, you know, SCADA systems and control systems. And so we had to worry about tuning those systems back then. If you remember like The Simpsons, And Homer was at home and he had to just push a button to acknowledge an alarm on his machine, his computer that he could take home. He just kept doing that over and over and then got one of those little birdie that you put liquid in and it just acknowledges it, you know. So like we in the engineering field have dealt with tuning of these You know, control systems for quite a long time, and that whole thing is an advantage because when we get these OT monitorings like sim, there's OT-specific sims now. There's OT-specific control system like IDSs. You know, and like we we have it in our sim that we can ingest a lot of those logs right into Google Security Operations. But Others as well. We can get that visibility, all the Windows logs, all the syslog, all of the OT-specific logs. We have to take that engineering tuning, like we have to be able to tune it the same way and not just be a bunch of noise, not just be something that is finding a needle in the haystack. Let's really tune it to the physics of that process tied to an engineering outcome or an impact. So when this alarm goes off, if there's an alert that's with that cybersecurity alert, what does that mean? What's the context around it? And I think we're getting better. We're getting better at doing this. But we still have a long ways to go in some areas. But I think cloud is going to help digest all those logs and Boil it down to the things that matter. You know, Google's really good at dealing with billions of things and sorting them. So, yes.

That is the mission. So, uh, I have to ask our traditional closing questions, uh, because already the first episode of the year is just about behind us. Uh, one, do you have a recommended piece of advice for helping people, you know, reason about securing their OT better? And 2, do you have recommended reading? Because this is obviously like I don't know, locomotives that drive off the tracks and things that explode. There's got to be good stuff to read here.

Yeah. So the, you know, to get into OT security, there's a lot of good resources out there. There's conferences that happen just about every month around the world these days. When I started, it was only just a handful of people, maybe 100 or so known on LinkedIn, you know, if you did a, you know, back in 2013, 2012, you know, there's not that many people talking about OT security. Back then it was SCADA security, but now there's college courses, there's free courses on YouTube that's out there, really good available like recordings on from previous conferences like DEF CON, Black Hat, uh, S4. S4 is like the World Cup of OT security conferences. So Dale Pearson has a lot of that content out there. And, uh, like there's the standards, like, uh, get familiar with the ISA IEC 62443 standard, get familiar with the NIST SP 800-82 standard, and that'll kind of get you on your way. Um, there are certifications out there, but just getting that experience, even if it's like taking an Arduino and learning Modbus and turning on the LED or making your Christmas lights turn on and off with the contacts, like that's learning that. Like, so if you come from a security background, that's a good way to learn about OT, some of the basics there. Start small. If you work at a company that has OT, go take a box of donuts to the engineers and the operators, and they're probably going to share some knowledge with you. Be humble, ask good questions, and network. And as far as resources, like I said, we have a lot of good resources on our Mandiant blog posts and the Google Cloud Security blog posts as well. We keep updating some of those. And we have new ones coming out. Office of the CISO has some good ones out there about OT manufacturing and hybrid cloud. We're going to be doing some more soon on other verticals. You know, there's a lot of other good resources out there as well, like I mentioned, the NIST standard there.

Well, Chris, this was a super fascinating, wide-ranging conversation. What a fun way to start our year, and thank you so much for joining us today.

Yeah, thanks for having me, and glad to work with you.