Cybersecurity Insights - Growing Cyber Risks of Firmware with Scott Scheferman

firmwares, the little talked about component of every digital device and computer system. It is, it possesses tremendous control for every device, but it's rarely being talked about in terms of how cyber attackers can take advantage of it. And I think it's time that the technology world recognize the potential risks to the digital community and that we better understand the growing cybersecurity risks that could undermine most of the current security controls that are out there and give unwanted control to the attackers or render computers permanently useless. So today I'm talking with Scott Shefferman, uh, and currently he's the principal cybersecurity strategist for Eclipsium. And he benefits from over 20 years of experience in the security industries with big name companies like Booz Allen Hamilton, Northrop Grumman, Mandiant, FireEye, SentinelOne, and Silence. So welcome, Scott.

Thank you for having me, Matt. That's a great lead-in. You kind of covered all the bases already on firmware.

Well, we did all those things. Let's talk about some of those things.

Yeah.

You know, and I recently spoke with, you know, with Yuriy Bulgin, CEO of Eclipse, I'm on a virtual panel and we had a great discussion and firmware was one of those topics. So I'm always interested in giving a little bit of love and talk time to the topics like firmware security or hardware security that just doesn't get that sexy time in the news, right? People just don't talk about it. And in most cases they don't understand it. So, you know, I think it's a great discussion to have and we're probably only gonna scratch the surface on the challenges that the digital world faces on this. But how about, you know, we jump in a little bit and for the audience, 'cause again, a lot of people don't even understand what we're talking about when we say firmware. So from, and jump in here at any time, but you know, for the audience, firmware is that layer that resides on top of the hardware in between operating systems or VMs. And really it's that very specific code to be able to leverage the hardware. Hardware has to be able to talk to the software, whether it's operating systems or whatever, but it is that layer below the operating system and above the hardware, that interpreter, if you will. And when we look at that entire stack of a computer system where you've got, you know, data and applications and operating systems and some VMs mixed in there and maybe a VMM, and then you've got firmware and hardware, where it sits, it can look up the stack. So it can see what's going on and it has some level of control to the operating systems and the VMs and whatnot. And bottom line is, if you compromise that layer, you can basically render everything else, you could destroy it, you could hide from it. And that's one of the reasons the attackers eventually wanna go down there. Right now they don't have to. There's so many other gaping holes to go after, right? They don't have to target the firmware, but if they did, they could hide because the software, the security software, most of the things that we think of can't look down, can't look beneath into the firmware. So help me out here. Why is that firmware beyond kind of what I just laid out? Why is it so critical to cybersecurity And my follow-on question is, why has the world been ignoring it up to this point?

Well, so the kind of the short version of the why, like why has the world been ignoring it or why does the world not understand it? Part of that is because when, you know, folks like you and I came up, we've been doing this long enough, couple decades, whatever, some people longer, you know, firmware is this kind of concept of a burned-in thing on a chip. It's immutable, it's there, it does its thing. And like you kind of articulated the, The applications in the OS itself don't normally talk to it other than that specific function to do the function with the hardware that the software needs the hardware to do. So that's, that's what firmware is in everybody's minds. But the reality is like, we're in the year 2022, firmware has evolved just like everything else, but the industry hasn't caught on to that yet. And the mindset of what firmware actually is, is very different from the reality. The reality is that the firmware is, well, start off, it's, you know, one of our researchers, Miki, says, you know, firmware's nothing more than software, just stored differently. So it's not on your hardware.

In an EEPROM or EEPROM or something like that. It's just code.

So start with the fact that yes, it's code. And what do we know about AppSec? What do we know about DevSecOps? What do we know about supply chain? What do we know about complexity of code, input validation, Default credentials, go down the list of all the SANS Top 20 over the last 30 years and there you have the same problem space that you have in firmware. But before we go into that problem space, like, understand that firmware today, like, if you look at the BIOS, today pronounced the UEFI, the UEFI, right? It's extensible by definition. It's massive, it's a playground of code. It's bigger than the Linux kernel itself. You can run video games in it. It has its own IP stack. It's got, you know, storage on it, way more storage than you think it has. You don't have to go find a register to find a place to put something in. It's got storage, it's like, and it's all interconnected and it's not just by itself as the BIOS. It's also connected to the BMC, connected to, you know, memory and chip problems that we have with like Spectre and Meltdown. All these things kind of conflate underneath the operating system, like you just articulated earlier. And so you have this playground of complex supply chain challenge, we'll talk about that, code that is highly vulnerable, that ironically is very accessible from the operating system even though we don't think of it as being so. The attackers certainly know it is. Microsoft knows it is, right? So the vendor, the OEMs know this, the platform folks know this. The attackers know this. The only people that don't are like the rest of us.

And the complexity gets worse too, right? I mean, you had mentioned BIOS and UEFI, but a PC, just a simple PC, will have more than one BIOS, right? Well, more than one piece of firmware. Your video card will have its own firmware, right? Anywhere, any specific add-on hardware device may bring, and likely does, its own firmware. So it's not just one firmware we have to worry about. It could be many and it is many. And then there could be different combinations. Well, so it becomes even more of a problem, more of a challenge, right?

If you think of it like a server, a rack, right? Let's say it's 10 blades in a server rack in a small room. And you think like, I've got 10 operating systems there. Maybe they're all Linux, maybe some Linux, Windows, Mac, whatever, right? Underneath those primary operating systems we do worry a lot about in security are like 10 other operating systems, 'cause firmware is more than just, like we said, burned-in static code that does predictable linear things. It's, it's updatable, for example. It's, it's complex, it holds secrets, it has all the, it holds certificates. All the certificate trust problems we have with like the SolarWinds breach and Mimecast and all the stuff that happened with, you know, No Bellium, and that now the Iranian hackers leveraging the same TTPs out in the wild, all that stuff plays out also at the firmware level during the boot process, right? So, and here's the weird thing about it is, if we worry about one thing in cybersecurity, if you were taking it from like a risk impact perspective, like what can hurt me the most? Or as an attacker, what can give me the most privilege, the most persistence, or the most amount of destruction or disruption possible. It's firmware. Yet, we don't prioritize that on the risk register as an enterprise or as a CISO.

We're not even thinking about it right now. I mean, we're not. We're worried about the software vulnerabilities and the operating systems and the applications and all those things. We haven't looked that deep. And there's actually good reason though, right? I mean, we have to prioritize our resources and most of the attacks are higher in the stack. Sure, getting down to the firmware and hardware gives you absolute control, right? But it's also a lot more difficult than compromising a piece of software or something else. So, you know, there's a story arc here, right? I mean, eventually the attackers go down that stack because they want more control. They want to get underneath the security, you know, and be more stealthy. So it's a race to the bottom, but it's also more tough. So, more difficulty but more power, right?

Perfectly said. The reason why I sit here at Eclipsium after working in the endpoint space and doing incident response, managing a global team of IR for several years at Silence Consulting, right? The reason I sit here now is because I've done the endpoint thing. I've done the EDR, the XDR, the IR, the forensics, all of these things, you know, even going back to FireEye Mandiant days, like, We worried with HX and those, those kind of things, we worried very much about understanding the endpoint as the nexus of the security problem itself. And it truly is, but these days the, the endpoint is not defined by the OS and the applications or the containers on top of that. It's, it's, it's really defined by the entire device as a holistic problem space. And yes, the network is always interesting. We should talk about firmware on things like VPN appliances too, because those are getting smashed in the wild, it's the same problems on those devices, right? You open the hood on there, they look just like a server blade 'cause they are. Yeah. Right? So it's, the one thing to comment on based on what you just said though is what I've been finding in my own journey is that it's much more easy to attack the firmware than at least I myself in my journey with my experience ever imagined. And I think it's easier now than it was 10 years ago 10 years ago, it's easier now than it was 20 years ago, it's certainly easier than it was even 4 years ago. Our team, you know, Mickey and Jesse are two best, you know, researchers at Eclipse and that break all these big stories and do all this research, they just did a little thing called Everybody Gets a Bootkit, right? And the TL;DR on that is any Windows machine from like the last 10 years has had the ability to be bootkitted easily and bypass everything, all your Secured Core, all of your Windows Defender, all of the configurations that the Microsoft— and not just Microsoft, but the industry would tell you should prevent these types of attacks. And that vulnerability, as a collective, as a whole, has been there for the last decade. And not a single vendor has attacked that, even though the attackers have been attacking that. All these rootkits we're finding now, the UEFI FinSpy, eSpectre, There's another one people are looking at just this week. All those that are hitting have been in the wild for 6, 7, 8 years, and we haven't seen it. So, it's not really the difficulty of the attacker so much as it is our own inability to see it from a, you know, observation, you know, visibility is everything kind of perspective in cybersecurity, see it, then also do anything about it because after you've compromised the firmware, you can lie. So like, you know, Cisco ASA gets popped, you can do a vulnerability scan, it's going to tell you it's the right up-to-date non-vulnerable version, but it's going to be maliciously configured and running, uh, not counterfeit, but malicious portions of code. And you're not going to be the wiser ever. Not as a vendor, not as the platform security team, not as NetSecOps, not as the CISO, not as anyone. Unless you actually can interrogate that device, pull off the firmware, statically analyze it, right? Heuristically analyze it and find bad sectors, find backdoors, malicious code, those kind of things, right? So we actually have to start inspecting firmware just like we inspect the endpoint.

Yeah, it's, you know, when you, again, it's all that stack and inherently you have to trust what's beneath you and the firmware has to trust the hardware, right? And the operating system has to trust the firmware. Otherwise it can lie. If you compromise it, it is so easy to remain stealthy. And that tends to be one of the big objectives, especially for the more sophisticated attackers. They want to become and maintain stealth so they don't get evicted because as soon as they're detected, they're going to get evicted. They're going to lose access. They're going to lose control. So yeah, they want to do bad things, but they want to do bad things for a long time. It's not come in anymore, grab some data and run away. No, the industry has moved from this, you know, we're just going to compromise the confidentiality, is I want to mess with the integrity. The integrity of your transactions and you, you know, gaining access to things and the other systems and people that trust you and your device. I do that, I own the world now and I own you and I own everything that you're going to do moving forward. So firmware is a great way to do that. But again, it doesn't have all the research and effort that we've seen in the past as compared to application or OS vulnerabilities and exploitation. But I think that's changing because you've got some big players, right? And I'm thinking nation states who can put billions of dollars in do into research, into developing tools that can find vulnerabilities in firmware and hardware. We've got academia now making a big focus towards hardware and firmware vulnerability testing, assurance, attestation, things of that sort. And these are all tools. Any tool that you can use to find a weakness, the good guys can use that to patch it, but the bad guys are going to use it to exploit it. So as more money pours in for the advancement of tools, whether they're, you know, start out as benign or start out as malicious, it doesn't matter. Tools are in the wild, they're going to be used, and the bad guys are really good at leveraging that. So with the billions of dollars that we're seeing being poured in, and we're also seeing organized cybercriminals as well now that are gaining hundreds of millions of dollars, they've now got the purse strings to start investing. And even if they don't want to invest, they can start purchasing vulnerabilities and exploits in this level. So even those good-natured researchers, you know, willing to make a buck, they're selling it and they may not know who they're selling it to. To or how it's eventually going to be used. Either way, it becomes public knowledge at some point down the road. Information demands to be free and it's going to be used to the detriment of people. So I think that it's just going to get worse and worse and worse and probably an exponential curve downwards as more tools, technology, investment, and everything open up to the world and the bad guys realize, hey, I know how to use these tools. Sure, I could hack the OS, but if I spend the same amount of effort and I hack the firmware, I'm 10 times more stealthy, 20 times more capable of destroying the system and getting past all the detection capabilities. I don't know, what do you think? Am I out of my mind here?

Yeah, I mean, that was like the best bump set spike probably I've ever had in my career. So I knew we'd get along. I know Yuri and Jennifer both said we would. So this is not surprising, but also, invigorating. So, yeah, so on the bad— on the APT side and the criminal side, if you just, like, take two quick examples to really nail home your point, the one I use for the APT side, let's just say the Russian GRU, APT28, right? Mm-hmm. Behind NotPetya, okay? I did incident response for WannaCry and NotPetya. I did it at a Global 5 organization for WannaCry, and then NotPetya was another big challenge, another large organization. And in that process, right? And I know you've lived through those and have colleagues as well that have. One of the things we remember from NotPetya is it was destructive in that it destroyed the master boot record, which made restoring the devices hard. You had to get another hard drive to stealth.

If you could crater the MBR, guess what?

Okay, so now let's walk the dog.

Let's have fun here, right? It's a desktop visit, right? I mean, come on. In all reality, you gotta put hands on the device. I'm gonna go full FUD, okay?

And full disclosure, I am so sick of the industry telling me that what I'm talking about is FUD or that the way I sound is FUD. In my mind, because I was on the other side of the aisle for 16 years doing DoD supports, protecting satellites, submarines, and million device networks, I get it, I understand vendor FUD, but this is not FUD. What this is is what I call anticipation for us to actually think about what we've just observed and imagine and anticipate what very well could manifest and get started now on mitigating that risk because those risks are real and they always catch us by surprise. So I'm sick of that. So this is a real FUD story, right? A real story. Imagine WannaCry replaying with the same actor, with the same destructive goals, right? Not ransomware income profit, but destruction. Hitting again the same way it did with some other Windows worming zero-day, maybe it's a CIA tool, NSA, who cares? There's always gonna be a way to worm around and do bad things, even if it's called ball strike, I don't care. But imagine NotPetya hitting again and the only thing you changed is instead of targeting the master boot record, you target the UEFI. Now, first of all, attribution-wise, which we don't need to speak too much about, but the Russian GRU, the same folks behind Drovorub, a really deep-level rootkit, the same folks behind Lojax, right, the ability to implant at the UEFI level. The same folks, to your point, that use off-the-shelf tools like Read Write Everything to be able to implant the firmware when it's a signed open source tool set that's been out there for 4 years, right. All this comes together even on the attribution side, it comes together on the motive side, and it comes together on the technology side when if all you changed from NotPetya was one thing, you end up bricking devices and operational devices operational environments indefinitely.

Mm-hmm. There is no— It's a permanent break. It's permanent denial of service.

You burn the device. Yes, right? So that's APT28. That's a story about the Russian GRU that could manifest very easily and has high likelihood, and no matter what lens you look through it, right? And then look at criminal actors. Well, so recently, like some friends of ours at Advanced Intelligence, Vitaly, Kremens, et cetera, They just did a piece on how the, you know, the Emotet, Trickbot, Ryuk/Konti triad is back in play again. Since 2019, they went to the razz, everybody outsourced. Now we're back in play with this trifecta triad, if you will, of power. Emotet's back in play, and Trickbot, and when I came to Eclipsium, my first order of business as the strategist and, you know, heading up research, et cetera, was to say, I believe Given what you guys are telling me here at Eclipsium, what I'm hearing from you, Yuriy, and all the researchers, that criminal organizations close to Russia would have just the same capability very easily. But I did all the research I could and I couldn't find an example. So I went on a quest for about 2.5 months. Long story short, we finally found through advanced intelligence the ability to find new TrickBot modules. One of them was PermaDLL. We've since re-termed that, or re-coined the term of TrickBoot instead of TrickBot, TrickBoot. Bland words. And its ability to actually look at the UEFI and if they wanted to very easily with one command implant the UEFI or brick the device, to our point earlier. And so you ask yourself from a criminal perspective, would I want to implant at the UEFI? Well, yes, you want persistence, you wanna be able to come back again, you wanna be able to resell that access to a third party later on and profit twice, or you wanna perform a function for North Korea with whom you've worked already before via anchor campaigns. And maybe you do want to destroy an environment that you find yourself in opportunistically, and you can do that, as well. Or you can say, "I will brick your operational environment if you don't pay me a ransom." Right. Is that leverage? Because digital extortion's all about leverage. It is. Right?

So, not that I want to give ideas to bad people, but the reality is they're going to have them anyway. And, you know, you talked about FUD. You know, it goes back to the, you know, one of the axioms cybersecurity. Security isn't relevant until it fails. So, FUD is making stuff up to try and typically get somebody to buy something. But when, you know, as strategists, our job is to figure out, all right, what will happen, right? And when we look at the motivations and let's take, you know, the ransomware folks. All right. Well, I love the ransomware people because it's just straight up cybercriminal, you know, financial gain. There's a small element of nation states, but we're going to ignore those for the time being. I'm talking about the financial criminal, cybercriminal. They're pretty easy to predict. We know their motivations, we know their objectives, we know what they go after, things of that sort. And if we take the normal ransomware right now and we decide to embed a capability to hook into the firmware, now what you've got is you've got a ransomware person or group that can infect a company and say, "All right, I've got 100 machines or whatever and I'm gonna," I'm going to destroy that data if you don't give me XYZ. You can take it up a notch to say, I'm going to do a permanent denial of service. I'm going to brick the system permanently, right? I'm going to fry your, you know, your EEPROMs. I'm going to burn them in, whatever. You're going to have to throw away those machines. Good luck in getting, with all the supply chain issues, getting new servers. Good luck, right? You're down for a lot longer than just rebuilding the operating system. No, I'm destroying the hardware now. Or even better, right, is saying, and this is my evil Matthew hat, do that, but you know what? Put a hook in the firmware because a lot of firmwares also have the capability for their whole network stack, right? They've got a network stack embedded in them and some other things and they can make other calls and do all sorts of wonderful things. Do a ransom on somebody, give them the keys. Once they've paid, let them decrypt their data. And then 6 months down the line, right, the systems reactivate, reconnect, pull down your latest version of your ransomware because the old versions, people will have decrypted it, done all sorts of other stuff, pull your latest versions of it, right, and start all over. Yeah. And no matter how much they search the operating system, and even if they were to rip out the hard drives, which is best practice, you get infected with ransomware today, you should be ripping those hard drives and grinding them. Buy new hard drives, re-land the OS, everything else. Doesn't matter. This is below that. You can burn all your hard drives. It doesn't matter. It's all going to start happening again. And then let's make a marketplace, right? If you can do that, why not make a marketplace on the dark web and say, hey, I can get into any one of these companies because I'm not activating all the systems, right? So I've got sleeper agents everywhere. That I can activate or will call back every once— I mean, there's huge market, there's huge amount of potential money because there's huge amount of damage that they can just turn the knob on and say, "You have to pay." So what do we know about attackers?

All the time we say, as red teamers, we say attackers will take the shortest path, the easiest path.

Sometimes we call it the path of least resistance. Right? Yeah. All these things. But they're beautiful, right? They're lazy like me. They want the easy money. They want it fast. They want the easy button. Everybody does.

And it's, I think it's way easier than people realize because you can send somebody a spear phishing email with a Word doc. They can click the Word doc and be implant, get a bootkit at the UEFI level. People don't think that's possible. I guarantee you 70% of the CISOs listening to this do not think, believe, or know, one of the three, that you can literally click a traditional spear phishing attachment and get implanted in the blink of an eye. It takes less than a second. You don't even see it. And you're implanted at the— you have a bootkit that Defender, EDR, XDR, none of your security controls will see or tell you about ever. And like you said, you had a really good point. That's a fine point, but we should elevate it really quick, which is, once you have persistence at the UEFI, you can go up, as well. So now, you're going up to the OS. You're disabling things like Credential Guard, you're making Defender think it's working fine, you're disabling EDR/XDR. And remember, you still have, like you said, too, your own security stack during boot time. You have storage, as well. You put it all together, and it's like, this is why, like, the Bloomberg articles came out and everybody's like, "The grain of rice on the Supermicro board," and everybody's like, "It's all FUD." Well, they came out with a second story trying to redeem themselves as, you know, application, and they listed like 3 other scenarios. Now, I'm not gonna tell you which one, but I know that I know that you and I both mutually know somebody that lived through at least one of those 3 scenarios. And you have to under— like, not you, the audience needs to understand that these scenarios are, they're not just possible, they're not very complex, they're not as sophisticated. I hate that word in cybersecurity. It makes it all these things. All these assholes, pardon my French, all these, you can edit it out later, they make all these guys sound like they're smart. They're not. They are in fact lazy. They do the darn thing. They reuse open-source signed tools. They bypass Defender because you can't, you can always escalate privilege on any other day as well. And with Windows, you have everything you need on any given Sunday to do the damn thing. Yep.

It's not hard. And fellow CISOs, this should not be too difficult to grasp. Look in the past few years of chaining vulnerabilities, right? That's what all the attackers are doing. And it's really just building blocks. I'm gonna grab this building block and this building block and I'm gonna chain it together. The building blocks are available. Firmware attacks are becoming more building blocks. So adding that to the malware, the attack, the chain, it's really not that difficult. And the attackers are gonna get there and it's not gonna be that difficult because again, It's gonna be packaged. It's easy to just implement it in. And we're gonna see more given, again, all the research, all the development, all the optimization and everything else that gets waterfalled down, cascaded down. It's gonna be as simple as, you know, going out to a website and I wanna create some malware and clicking a box. Yeah, add in a firmware hack, add in this, add in firmware persistence. You're so right. And, you know, compile. There we go. I got my malware. Right? We've seen that with other malware. This is going to be no different. This is just another, you know, string on, you know, on the bowl of spaghetti. So.

I'm going to, I'm going to say, we talked about the UEFI and the BIOS and kind of like your traditional IT compute, right? But there's this whole area of two other areas we should talk about really quickly. I'll do one really quickly with a short story and then we'll pivot to like the VPN appliances that we should talk about too.

All right.

Because they have just as many firmware problems. But the, um, but short story, so a set of red teamers were tasked with attacking a data center. Typical task, right? They scan the IP range that they're in scope and they find a SQL bug, not surprising. They get a shell from the SQL bug, this has been going on for decades, no problem, right? Decade. Once you have a shell, what do you do? Well, some people will go, "Okay, I'm gonna live off the land, I'm gonna use, I'm gonna hit lsass, I'm gonna do UAC bypass, I'm gonna, I'm gonna do Download Cobalt Strike, I'm going to live off the land and try to evade EDR, XDR, and all these things that are— Right, evade detection. Right. But these— this is a true story, and this is like 8 months ago. These attackers said, "No, we're not going to do that. We're going to get at the shell. We're going to scan around what we see on the network of that interface of the box we just popped in the data center," right? And we found— they found some BMCs, baseboard management controllers, writing IPMI protocol. Oh yeah. You're laughing because you're correct. Well, some of the most vulnerable, crappily coded things in the world that you can send 29 alpha characters to and they'll spit their admin password right back at you in any given day. These things are horrible. These talks about DEF CON have been going on for like, you know, 12, 15 years. So, they found those, they popped those, and then from there they were able to scan even more. They ended up popping like 12 or 15 BMCs throughout the data center. From there, long story short, they got 20,000 devices and all the backup instances as well for the entire data center without ever worrying about EDR, XDR, all the operating system controls, Windows, Linux, Mac defenses, the network— forget all that. They dove down because they're going to be more stealthy. Now, they handed in their report and the data center got an F or whatever. And they said, you know, fix your IPMI, segment it, and do all the things you should do. This is ridiculous. But what if they were attackers? They could have cataclysmic— they would have owned it. Data center forever.

Zero restore capability because they got the backup because they had the backups. Yeah.

So that's a story like NVIDIA has these AI boxes like the DGX-100s are like $250 grand a pop, right? They're built for doing AI, all the math. So research universities have national labs, have them. Well, a set of researchers, I think the Odyssey team, forget their name, They were doing a normal pen test. They found one of these things and they found a BMC port exposed to the internet on one. And across the internet, they found like 9 CVEs, 3 of them I think were critical. And NVIDIA had to patch those. Now NVIDIA responded with, "Oh my God, this is great. Thank you for the coordinated disclosure." But they had to go to another vendor, I think Quanta or AMI, one of those two, and get Quanta upstream supply chain. So we're bringing in the supply chain to the conversation. —to fix the vulnerability in their hardware. And it took them a year to do it.

Because most development houses aren't that savvy. I mean, and you know, we've been talking about data centers, big companies, chips, PCs, servers. What about smart vehicles, right? What about all these other IoT devices? And their development teams know nothing about firmware security. They're just taking base firmware and going, ah, okay. "this is the standard, we'll just throw it in," right? And they're putting no controls and no security around it. They're doing a 5-point checklist and then they're done. They'll never touch it again. You know, we've got other problems as well. It's not just those data centers out there. It's gonna be everything that touches our lives. And I don't want my smart vehicle getting a firmware hack. I don't wanna be permanently stranded somewhere or have it take control of the device and accelerate accelerate and turn into traffic, right? It brings ransomware to an entire new level. But again, we've got problems all over the space. So let's— we've been talking about all the bad things and we've talked about how the attackers are probably going to go there eventually. We don't know the timing yet, but what are we seeing right now? What kind of things do we see in the industry that are starting to address some of those. I know on my side, you know, when I'm talking with developers, whether it be for products or services or devices, it's one of the topics I bring up, right? You know, are you patching? And they're, oh yeah, yeah, yeah, we're patching. What about for open source? Oh yeah, we check open source. What about your firmware? What? What? Okay, you're going to add that to the list, right? Okay, let's add it to the list. That's now part of your scope. That you're going to keep and maintain. You know, so I see it in some of the development houses. They are taking, at least starting to, taking that into account. And I'm seeing on the academia side, they're putting more research into tools to help identify vulnerabilities. But we know those same tools can be used to, again, by the bad guys to figure out targeting. What are you seeing in the industry?

So, you know, if you take like a higher step level step back, you have organizations like NIST and CISA now. And by the way, hats off to CISA. We've all got friends there. They're doing so much better and so well compared to years of DHS. I'll just say it, I'll be the guy, whatever, we don't have to go into it, but like lately CISA has been killing it. And, you know, last year at like RSA conference, right, they gave two talks and they're both on the same topic, which they call, they've now labeled BTOS, stands for Below the Operating System, just like your introduction. And they're largely focusing on the UEFI BIOS discussion that you and I talked about in their examples, but they are expanding it. Then you also have the executive order that we all heard so much about lately, and one of the things that tried to do is define what is critical software, what, you know, and if you look at the 5 criteria that they put out for litmus for what is critical software, well, firmware matches all 5 of those literally almost by definition, almost so much so it's laughable., right? So, so that's in play and will eventually affect firmware when it gets to things like SBOM and what we affectionately call here at Eclipse, the FBOM, firmware bill of materials, right? So, so there's that in play. And then like NIST has their Center of Excellence, their CCOE initiatives, and we're part of that Eclipse, are, as long with like HP, RSA is in there, Dell, and this consort— this particular Center of Excellence is centered around firmware supply chain, and one of the things that like Eclipsean will be doing is validating that the thing that you ordered and took delivery of as a customer is the thing, right?

So that you can trust— It validated it hasn't been tampered and all sorts of things, yes.

Cryptographically, you know, ensuring that, and that's using basic crypto hashes and, you know, certificates.

The tools are out there to do it, but people aren't doing it. It's not easy, it's not convenient, it's not trustworthy, and who the heck is who's going to do it, right?

And the other challenge is, just like any environment or like a new PC you put into an operational environment, the second a device goes in an operational environment, it's exposed to everything from firmware updates to insiders to USB ports, PCI. There's 1,000 ways to attack a device at the firmware level, right? Well, maybe there's a dozen, but it's a lot. And so when you think about it, these devices have that whole, you know, take provision, provision it, all the way to sunset. And even after sunset, there's firmware risks, just like there are hard drive and, you know, data risks. So one of those things the CCOE is trying to figure out is, look, you know, I might validate that the thing is what it is, but after I put it in my environment, how do I monitor the integrity of this continuously so that I know when you have a bad day? Because you're not going to have, like, an antivirus signature at the firmware level very easily. You might have a campaign that goes for 4 years before you even have an IOC. But yet you still need to understand whether or not you initiated an integrity change at the firmware level or something or someone else did. Right. Right? And then you have to be able to statically add or heuristically analyze that as well to say, "Okay, I actually do have a problem. There's a backdoor here. There's an implant," right? Or, "This is a counterfeit piece of code that's come out of Taiwan," right? So you have to be able to manage your devices kind of holistically as they're operationally in play. Yeah.

Okay, so, and we're coming to the end of our time here. I wanna leave the audience with some recommendations. What should they be thinking about? How should their mindset be shifting? And I'll start off. Let's go back and forth and give a few. So I think, you know, if you're the CISO, you've got a number of capabilities to predict, hopefully prevent, detect, respond. And most of those are likely around data breaches. They're around ransomware. They're around other things. I think a good CISO moving forward needs to start expanding that scope to understand what happens. How does my, you know, prevention, detection, response, how does all that change if I start to include firmware? Because your detection is different, right? The preventative controls are completely different. And how you can respond is fundamentally different if that BIOS or UEFI or whatever, or one of the BIOSes, right? One of the pieces of firmware in a system get compromised. What are you gonna do? And if you don't have a plan, we know your response and we know the impact is gonna be greater. So my first piece of advice would be, if you're that CISO, expand out your scope, Beyond just the operating system and apps and data, start thinking about firmware because eventually you're going to have to deal with that. Good. Yeah.

Okay. So we didn't really get to cover like the VPN devices we alluded to, but— All right.

Well, let's talk. What would you recommend around those completely secure VPN security devices that I'm sure cannot be hacked? Right.

Ironic. So it's ironic. So you bring up the best part of the whole story because people, it's almost like it's so obvious you forget the irony. This is your primary security control in a remote workforce post-COVID world. It's the number one thing that enables your business, your revenue, even your safety and uptime in operational environments, your VPN device. And it's externally facing the internet, as they do, as they need to.

It wouldn't be worth much if they didn't connect to the internet. Right.

And so here you have this device and it's running it's running firmware too, and it's running a low-level device operating system too. And those systems and that software is, like we said earlier, coming largely off the shelf or via third-party development houses, which are being attacked, which have all sorts of problems that we don't have scope to go into now, right? But that trend that we've seen the last 18 months. So, you put all that together and you get this device that has remote code execution vulnerabilities facing the internet. Now, here's what's lost upon most organizations. They think when they have an incident, when the attacker got in through the initial vector of the VPN, that the way they got in was through stolen or purchased credentials on the dark web. And they're not wrong. But if their playbook doesn't tell them to go back far enough, what they fail to realize is that the reason those credentials are for sale is because 2 months or 6 months or a year prior, that device was actually exploited. And then, the attackers from there are pulling creds off the device or even creating accounts in the device. Right. Not all attackers have the same motives. Many of them have a primary motive of selling and reselling access. Mm-hmm. And so they don't do anything further once they get the VPN. They're not going to the AD like you might think they are. They're not dropping payloads or ransomware drop or any of that. They're just popping the device and churning out accounts. So you have a problem that you don't even realize as a victim. It's a really big—

And you think changing the password will fix it, and it really won't. Excuse me.

I apologize. I got a tickle in my throat. So like the— I won't say who— like one of the organizations that we read about in the news that had all their plants go down attributed that to like Qbot and then like one of the ransomware groups I won't name either because it'll give away who the victim is. But if you, if you actually had access to deep dark web intelligence and you did forensics intelligence in that arena, you end up discovering that their Citrix ADC device, those creds were for sale from a vendor that's known for popping and exploiting those devices prior. That story never made mainstream media. It never made the lessons learned. It never made, to our point now, like what are the best practices you can do? It's because people don't bring down their VPN devices 'cause they're critical to uptime. Right. It's because people don't— people also trust the operating systems to be what they say they are when they interrogate them with simple scanners like Retina or whatever, Nessus, right? So, those scanners lie to you. So, you actually have to do better work. You actually have to do zero trust. You have to verify then trust those devices. And you cannot rely on the platform vendor that makes the product to do that function. You cannot rely on that. That's why everything— That's a disturbing statement right there.

That's a disturbing statement. You know, you can't trust the security vendor whose reputation should be on the line here, you know, for their, their VPN product. That's, that's tough for me to choke down because we have to trust somebody. I'm just hoping that those security vendors elevate rise to the occasion to where you can trust them. And if they're not, I hope they're going to go out of business, honestly. You know, they're a detriment.

And I promised I'm not going to shill Eclipsium, but one of the things we were forced to build from our customers is the ability to discover devices around endpoints, including VPN appliances, and in some cases even pull that firmware off and analyze it. And inspect it. So you have a mechanism and a tool to be able to do that verify but trust. But that's, the industry hasn't had that for 20 years. This is a new thing, to your point.

Okay, so that leads me to my next one. It kind of dovetails in. So for the CISOs and everybody out there that, you know, have their budgets and are buying tools or services or whatever, my last recommendation would be make security, right, the firmware, because you're already looking at the apps and the certifications and everything else, but embed, you know, firmware security in that evaluation. It should be part of the success criteria, that purchase criteria for products and services and vendors. And again, there's not a single vendor out there that can honestly say, hey, our firmware is bulletproof. There's no such thing, right? There's no such thing as a perfect app or operating system or anything else, even hardware, right? But, you know, look at the vendors and do they talk about it? Do they have a plan? Do they update, right? Or do they just release their product and they've never updated their firmware? That's a red flag. You know, is their firmware update process not validated? That's a red flag. Do they not even talk about it when you're talking to that company? Firmware? Oh, we don't worry about that. Right. Red flag. So make it part of your purchase criteria and vetting process because it's going to be more and more important every single day. And it should be. And if people are using that as purchase criteria, That then financially incentivizes those companies, those technology companies to take it seriously, invest the time, effort, and research to make sure that their products are hardened and that they can sustain that over time, that they can patch it and they can interrogate it and they can validate it and they give more— all those kinds of things. So if we use our wallets as voting, It will make a difference and turn the industry. So that's my last one that I've got from my colleagues. What have you got? Last recommendation for listeners.

Well, on a small level, we have a blog coming out shortly. Your listeners will be the first to know this. We're putting out a tool that will help people for free understand a problem with MikroTik routers that TrickBot and Meris and DDoS networks use. And know whether or not those millions of devices and hundreds of thousands that are already exploited are part of your enterprise and your work-from-home environment. So that's one little micro nugget. It's going to be a MicroKICK checker tool. And that'll be out when we release the blog. And you guys are, I guess, the first to hear it. But beyond that, there are open-source tools that you can use, but they're not enterprise-ready or enterprise-grade. So, you know, I don't know how else to say it without just calling it like it is. And I shoot pretty straight. Like, if you want to actually solve this at the enterprise problem, for me, medium, large size, you need a tool like Eclipsium. There aren't many tools like Eclipsium at all and that's because it solves a really weird, very complex, very hard problem. You know, we have to store millions of images of firmware to understand good from bad just like you have to do with viruses and malware. And we have to do that at the firmware level. We have to analyze things and we have to be able to scan and understand all the devices in— at your home or in your operational environments or in your enterprise. And it's a very hard problem to solve, but we're actively solving it and we have a really good solution to do that. So just get a demo. We even have a thing on, what I call a thing on a stick, which is the ability to actually plug it into the laptop that you're looking at right now in front of you and say, do I have a backdoor? Have I been compromised? Do I have a vulnerability known to be exploited in the wild by actors like TrickBot or Iranian actors or APT28 like we talked about? Apt41, Chinese actors. We know whether or not your laptop has those problems or not just with a quick thumb drive in and out type of solution. So we're happy to let you use that, as well.

Outstanding. Free tools, something to play with. I love it. I love it. All right. Hey, thank you so much, Scott. This has been a great conversation. I look forward to many more in the future. We can talk about hardware. We can talk about VMs. We can talk about just some of the, the bad actors that you named out there and what they're doing and where they're going next. Yes. So until next time, thank you everybody for listening, and we'll catch you next time.

Thank you, Matt. I appreciate it. Thanks for watching.

If you like these cybersecurity videos, please click the like button, leave a comment, and subscribe to the Cybersecurity Insights channel. I put out a video about every week on various cybersecurity topics, risks, ideas, events, and best practices. Let's continue to communicate and collaborate together. That's how we make cybersecurity strong in protecting the global digital ecosystem.