The Cybersecurity Vault - Best Practices in Securing 3rd Party Supply-Chain with Cassie Crossley

Welcome to the Cybersecurity Vault. I'm your host, Matthew Rosenquist, CISO, industry cybersecurity strategist and advisor. Today, we're going to talk about the challenges and best practices for securing third parties and supply chain partners. And I'm going to be talking with Cassie Crossley. And Cassie is the VP of Supply Chain Security at Schneider Electric. And the author of a book, Software Supply Chain Security. Welcome, Cassie. Thank you for coming.

Thank you for having me, Matthew. I'm very excited to be here.

Okay. So let's dive into this, right? We talk about third parties. We talk about supply chain risk. It's truly an ugly topic because, you know, a lot of CISOs look at it as this isn't my world, right? These aren't my servers, systems, people, policies. This is the third-party vendor that I'm using or trusting or whatnot. And there's a ton of potential risks there. So, what are those kind of risks that those third-party bring that, you know, if it isn't managed properly, can go south? What kind of risks are we talking about here?

Sure. Well, there are so many to even start with. But first, let me say, no matter what you do and manage, something is gonna happen. So first off, it's just going to happen. Well, you're full of good news. I know, I know. Well, and I get asked, you know, what can we do? You know, cyber resilience and just resilience in general is so important these days. If you don't have a backup plan and business continuity plan for these vendors of what's happening out there, there today, you know, it's going— it will happen. It's happened to everybody. I don't know a single CISO that hasn't been impacted by a supply chain attack with one of their suppliers. And these are large corporations that wear large application security, cybersecurity, and so on. You know, everything's all in place. But you just can't predict because they all have suppliers and there's other suppliers and there's so much— I mean, CISOs know better than anybody else. There's so many ways for threats, for attack that, you know, just a simple phishing email can compromise a supplier. So sorry to be the bearer of bad news on that, but it's going to happen. So be prepared.

And we're all chained together. That's why I call it a supply chain. So you're only as strong as the weakest link.

Right.

So, yeah. Okay. Okay. So what kind of bad things can happen?

Yeah, well, I— but I do want to say that that doesn't excuse us for ignoring and not doing proper due diligence. So what can happen? I mean, we, we see suppliers every day. We are buying software. I like to point out, like on your iPhone or if you've got one of those 50 preinstalled applications, So just think of that. The operating system and 49 other applications had thousands and millions of developers and thousands and hundreds of thousands of people putting together that one device, right? Uh, so can something happen? Yeah. When you talk about the amount of work that goes into any of these, whether it be a software application, a SaaS, product, anything, it's all going to happen. But that doesn't mean that you shouldn't be asking your suppliers and really pushing them to improve their security every day, their application security and their cybersecurity. So one quick tip is if you are asking them about their cybersecurity program and you're not asking them about their application security and product security program, you're missing a whole area. We have lots of people who are doing ISO 27001 certifications and SOC 2 certifications, and they never ask, are you developing it securely? Are you designing it securely? How is it all being put together? And you need to push that because if we don't start pushing suppliers to do that, what's the incentive? There's not a regulation. I mean, even There's attestation forms and things like that if you're gonna sell to the government. But from a supplier to supplier, how are you, how are they going to improve, right? Every day somebody's writing code, whether it be an AI system now or something else, and, and it's all being brought together. And if you're not asking them if they're doing it right, then even if they don't have a great answer, then next time, and they should work on that improvement. So that's my biggest suggestion in regards to supply chain security. If we don't keep pushing those suppliers and vendors to really up their game, it's just gonna get worse and worse.

We have to align the incentives so that there is an emphasis on it. You bring up such a good point because, you know, you can ask, "Okay, how's your infrastructure?" But what about the actual code, the program that I'm installing on my systems?

Right?

You need to have confidence that it was crafted and programmed and vetted and that it's being maintained. It may have not had any known vulnerabilities the day it was released 3 years ago, but if they don't have an active program looking for vulnerabilities of the 7 to 10 that get discovered every single day, right? And then managing those and updating it and patching that, you could, you know, be dealing with Swiss cheese and attackers coming in and out as they please. So I think it's a really important issue. Don't just look at the infrastructure of that third party. Look at their products and whether they're managing those products securely and with forethought or not.

Yes. Right.

So you had mentioned one thing, right? Look at that. What are some of the other things that you can do to address the risks in these partners, or essentially their partners or suppliers or business partners. And, you know, they may not want necessarily to open the kimono and show all of their practices. So how should companies be negotiating or approaching that internally or even via contracts, right?

Sure, sure. And you brought up the best point is that in— I have a whole chapter dedicated towards supplier and supplier management in my book. And one of the points that I really address is you may not be able to get assessments done. We all get questionnaires. So for a supplier, you know, you get a lot of questionnaires and those that I do get when they ask about the products, you know, several years ago, even if I had answered a question and it had a partial or a no, we don't do this, no one ever came back to me. So if you're going to ask a question, be reading the answers and following up. I mean, what good is it if you don't do it that way? But then also bringing that into the contracts. So I work both on this, you know, with our suppliers but also our customers. And I'm right in the middle saying, you know, to my suppliers, I have to meet these obligations because in my customer contracts it requires this. So customer contracts, like, just, just say, for example, incident notification or something like that, 24 hours or whatever. So if I have a supplier that refuses to do something in 24 hours, you know, I can't meet those obligations. So as when I'm going through supplier contracts all the time, Those are actual conversations. And so what I suggest and recommend is for your critical suppliers, do something we call it here called a CISO Connect, but a C-level connect where I, for those critical suppliers, know the cyber leaders at that company. If something happens, I want to know who to call, you know, where they're located, have had that relationship established. And that's really important. When you are— let's say they may not even be your biggest spend supplier. So for example, when I'm including products and different maybe software libraries in something we sell, I may not have a large spend with that software publisher. So let's say an operating system, real-time operating system. But yet if they're in 30 of my products, I want that relationship. I want to know how they manage their application security environments and who they are. And so, you know, You know, let's say it's Tom and I call, you know, I wanna have that relationship with Tom. So if we go to security conferences or I reach out to him that he knows who I am and can respond and he worked with us together and discussed those. When you're working through those supplier contracts, and I'm not talking the humongous 100-page ones, I'm really talking about those cyber addendums, which have those commitments. You can always tell the cyber capabilities of a group by what they redline. If they're redlining something, then they probably don't have a process for that. And that indicates a maturity that you need to have that discussion. And, and it's when I'm working with those suppliers, a lot of it is tell me what your program is. And I can't do that if I'm having legal talk to legal. Right. They're not, they're not understanding, you know, what's this disclosure policy? So that's my, my significant recommendation is really establishing that with your critical suppliers. And what that does also is if there's a breach, you see them in the news, you know, something happens, they know who to call and vice versa. My, you know, our customers have access to our regional CISOs, our global CISO, to me. You know, if something happens, you know, we saw a customer that had a breach, we called them, say, is there anything we can do to help? And we go there. It wasn't our product, but we wanted to be able to help and provide assistance because if some of the systems could come back operationally, it would help them. So there's a lot that you can do in reverse to really build those relationships.

And from a CISO's perspective, obviously we're not the first person that goes out and talks with a new potential supplier or vendor. So is the is there an internal alignment that really should happen with your legal team or negotiating teams or contract teams or frontline business groups that are bringing these vendors in to prepare them so that in those early conversations there is a mention, "Oh, by the way, there will be security scrutiny. You do need to be able to have certain controls. We've got a security expert that you are going to be talking with. And this is required. This is not, oh, after we've signed the contract, oh, now can you do security stuff?

No.

You know, setting that up early.

It is. And our— I think many companies, especially medium to large, will have a document that explains our supplier assessment process. Not in detail, not how we score people, but this is what we believe in. These are the expectations. And in And in there is our cybersecurity controls and our processes. So it is explained to suppliers upfront. We have this vetting process. I think everybody's used to now there's a cybersecurity vetting process that goes on.

Nobody likes it, but everybody expects it. Yeah, they do.

They do. And so internally, making sure that your legal procurement, that like there is not a sign-off now without making sure that there's a cybersecurity approval and it's to help alleviate business risk. This is not a cybersecurity risk. They treat it like it is, but it's a business risk. What happens to that supplier? It doesn't matter if it's something physical or something else. It could be a quality. You never know what it is, but sometimes it's a cyber risk. And that's why you need to have a seat at the table and to be familiar enough with that whole process. So being and having those injection points So with our process, we have some initial questions, you know, just to say, what is your security posture? It asks some of the general questions. Do you have IT security, this kind of that? Because if they can't even ask and answer that at the minimum, then, you know, you can't go much deeper, right? There's a big concern. And then that adjusts the weighting internally of that supplier. I mean, that will make a big decision point because if it hits that certain Here's a red alert. They can't go past that without a lot more in-depth discussion on that. And so, you know, it's a very important part of the process. And I think most people see that now, but they also treat it like, okay, this is cyber, I can't talk about it. I absolutely— every meeting that we have with customers or suppliers, I make sure the business people, procurement people, you know, legal are on there. I want them to hear. They may not understand anything that I'm talking about, but just that discussion and that relationship, they'll get more comfortable. So I have a lot of people that come back to me and they're like, okay, we're really only stuck on these couple points now because they know where, you know, is, uh, let's say, you know, this is acceptable, but this isn't. And that makes a big difference.

So is there a difference between the IT and OT worlds when it comes to managing those third-party risks? And I know you had mentioned a runtime operating system, very, very common in the OT side of the house, but what are some of the challenges? And I mean, is there internal things that you have to think about fundamentally differently?

Yeah. CISO that has— let's just say there's IoT, there could be industrial IoT, there could be other forms of IoT and OT. So let's— let me break them apart. There are— when it comes to, let's just say, any kind of firewall devices or anything that are IoT and internet-facing IoT kind of products, those from a standard patching maintenance Cadence, all of those. You should follow those very closely. Be very aware. Have those prioritized. Top of your list, anything internet-facing in your company, top of the list. And when you're talking, however, about industrial IoT or OT for operational technology, that could be— it's not only what some people think about, about SCADA systems or industrial control systems. So these are things that could be operating your manufacturing floors. They can be operating the elevators in your building and your, you know, your environment. So there are usually operational technology. It means it's some— it's some form of physical action that's associated with it. And there are, of course, IoT sensors and different things working with OT environments. All of those, um, first off, unless absolutely required or they have the proper controls around them, are usually not internet-facing. So the worst thing that you can do is put a non-internet-facing device that doesn't have the appropriate security controls out there. And we see that whether it be with Shodan scans or BitSight now has features that can identify operational technology and they find things out on the internet. And that is why from a critical infrastructure, that is very scary to know that there are products out there that don't have the defense mechanisms in place to be able to stop that, to stop attacks, to be able to handle DDoS and everything like that. So operational technology, when it's an environment like that, most of the time, the majority of the time, there are safety concerns. All right. So you generally are not going to have automatic updates. I mean, if you automatic update your IoT printer, that's fine. But what about automatic update for an elevator door, right? You know, what happens if the safety implications, the sensors that detect somebody is going through the door, something happens? So, and by the way, Schneider Electric, we do not do elevators, escalators, or anything else like that. But I'm just using it as an example for, for the audience. So operational technology, I don't know of anybody in real strict environments for operational technology. They don't do automated patching the systems. Don't have these automated patching features because the safety implications are very, very high concern. You know, they can lead to death.

So you're saying you don't want to reboot the airplane mid-flight is really what you're saying.

Exactly. Exactly. And also, people want to test those environments, right? You want to make sure that you've tested it, whatever changes. So, for example, a lot of OT systems, whether it be manufacturing— so just picture a manufacturing site and it may be running 24/7. They expect operational technology to last decades. All right. The lifespan for our firmware PKI is 30 years for certificates because that's what we expect these devices to last. So that means that back in the 1990s when people had hardcoded passwords and default things, they're still out there. And they're running every day, right? And they're using non-secure protocols. A lot of them, if they even had web servers, were HTTP, right? There's lots of products out there running every day with TLS 1.0, okay, and TLS 1.1. And a lot of products shipping today have to support those older technologies so that they can communicate. That backwards compatibility and interoperability is key for them to maintain. Otherwise, you're going to do a $2 million upgrade every time you need to do a patch and get a secure protocol in. It's not going to happen. So from an environment, what I, what I truly— it's the same kind of work for IT— is you have to have that asset inventory. All right. And I think you find that when you do an OT asset inventory, Some of those suppliers may not exist anymore. Does that mean you have to rip and replace? I don't believe so. I think you need to secure them with the proper defense mechanisms outside and look in as part of your modernization plan. But let's say you have, you know, even, even a printer that you bought 7 years ago and it's still working fine, but yet, you know, it has some features that probably shouldn't be, you know, Bluetooth or some of the different things. Turn it off, learn how to secure it, or put some, some appliance between it and other appliances so that you have those controlled tunnels. You can build secure tunnels. We all know how to do it from a technical side to make sure that those devices can communicate without having to do rip and replace. But having that OT asset inventory, you don't know otherwise, you have no idea what's out there. And so, you know, when customers ask me for software bill of materials for products and, you know, for those firmware, almost always they apologize. It's not the latest version. I understand that, you know, OT is only patched at about 3%. And, and so these older versions, it probably could have the original firmware it was ever put on there, you know, decades old or whatever. And it can only be upgraded with USB and it's not meant to be internet connected, even though it's got a SIM card. Serial cable. So that's what I recommend is really to understand your patch management plan for it, but also to understand how you are really defending the overall infrastructure because, you know, $100 or $200 deployments to replace something that would cost $250,000 to modernize. I love modernization. But let's be practical here. And that's not going to happen at wastewater treatment plants where they're, you know, living on a shoestring budget.

Yeah, everything you said just resonates with me. So when I was at Intel, you know, I was overseeing security in the factories, 26 different operating systems going back to DOS 3.0. Yeah. So there's no patches for these available. And, you know, these systems are running kilobytes of memory, not gigabytes of memory. You know, and you had vendors that no longer existed, right? We can't get into this system. You had other vendors that the moment you touch it, install, do anything, you invalidate the warranty.

Yes. Ah!

And you have other ones that simply cannot accept any type of patch or anything. And they're, you know, they were designed at the time, but that's again, 20, 30 years ago.

Yes.

So, you know, it's a nightmare scenario from somebody with an IT mindset, but—

Yes.

And I came into it with an IT mindset and it was like starting an entire new book on security because you do, you have to understand what do these things do? How are they protected?

Yeah.

Can I enclave them, right? Put something between them and where the back— who's going to come in? What is that going to look like? What kind of— And it's an entirely different— it's tough enough in your environment, but now we're talking, okay, apply that same sensibility to your third-party vendor and their chain of vendors. This is where the complexity compounds. It's not just IT, it's OT. It's not just your environment, it's your supplier's environment. It's not just your supplier's environment, it's all their suppliers' environments.

That's right.

Okay, okay. So before all my hair turns gray, let's talk about what are some of those, you know, give me your top 2 or 3 mitigation controls. And for OT, you had mentioned one, right? If you can't patch it or fix it or update it, put something between it and the threat, right? Put some type of barrier, a device, a firewall, something, right? Isolated communication But give me a couple of your best-known methods, whether it's for OT or IT, in how to kind of manage those risks for those third parties.

Right. So I think that overall, I think some of the safest things that you can do is you ask about their developer infrastructure and if you are creating products. So a lot of CISOs, you having stuff built internally or you're selling something, right? If you are a technology company, what I see a lot is that the development infrastructure— and we saw this with SolarWinds, and I had already written and, you know, the chapter not for my book, but for a previous book on software supply chain security. I'm like, you don't understand. No one watches our developer environment. We, as developers, and I'm a developer, we have complete control. We're not on a corporate endpoint. We can create our own virtual machines. We can, you know, put together Linux. If I want to take down a project and compile it, how do you know that that code that I just pulled down and compiled, because I probably didn't read every line of it, doesn't have logic bombs in there? And keyloggers and everything else. And so I'm not saying, you know, kill all developer productivity, but you have to take to the development and R&D organizations, both you, your organization and your suppliers, and say, this is a risk. You saw SolarWinds. This is not only about the build process. And that's part of what some people are thinking, oh, it's only when we build to make sure code— and in that case, what happened is somebody got in and added some code. And so it was built into a finished product. So you had no idea that it included this, this, you know, this malicious activity.

And it was inside the shrink wrap. It was embedded in the product, right?

It exactly was. And what that means is that there are lots of ways for them to get in right at this starting point. And if you can get it at the starting point, how do you know that, you know, Cassie wrote that code versus, you know, John? And are you tracking that? But also, does Cassie understand that if she brings something down onto that machine and is using it and it is, you know, it can inject something into what she's writing that she never sees, and yet she commits that code to the library. So it shouldn't stop me, but what shouldn't be happening is Cassie should say, I'd like to look at this project. Here's what it is. Somebody else looks at it too to say, oh, are you kidding? That project's only been out there 3 days. That's probably, you know, a chance for malicious. So setting in those guardrails and those really understanding, you know, where the risk could come in at is very important. And the same when you're asking your suppliers. So that's the, like I said, there are 50 applications on your iPhone. I am not expecting you to do due diligence on your iPhone. But for Apple, you know, we should be pushing back and just like, you know, everybody else. Are you certain? Do you have the provenance? Nobody has the provenance. What that means is there is not a history and record of every transaction that happened with a certain open source library there. By the way, if anybody's got a Windows machine, you have lines of my code because I helped write paint.exe that got sold to Microsoft, and back in the late '80s or, you know, mid-'80s. So there's nothing that says Cassie wrote those 7 lines of code, or, you know, 2,000, uh, and we're never going to have that. We can't go back in time. So what that means is that next step is you need to make sure that you have the right testing processes for those things that are critical, that you are evaluating it. There are services out there that can judge and evaluate not just external-facing, but also let's look at that product or let's look at that service that that group is and monitor it. And it's worth it to, to really pay experts. Most companies don't have an application security expert. Right? But there are companies out there that are really doing that due diligence. In fact, in the critical infrastructure utility industry, especially in electrical industry, there are service providers that are keeping track of suppliers like us and the products that we do. And they'll be able to tell that utility providers, you know, this is a product that has these CVEs and it, you know, it looks like you have about you know, 1,000 of them spread throughout your region, you need to be aware and provide guidance. So I think that we're going to see that change more and more where we're not just depending on BitSight Risk Recon and Security Scorecard for an IT presence, but also for application and software supply chain security. There is— there's many niche groups So if you're in a niche and you can see, but that's where some of the ISACs, the information sharing and, you know, communities are going to be geared toward where they're going to not only hear, let's share this, this TTP, but also here's some information I know about this product. We've done an analysis on it, whether it be software or, or an IoT device. And that's really what there's going to be doing. So I know people don't have an unlimited budget, so you really have to determine where those risks are, but you're not going to be able to do it all yourself. So I think you're going to have to prioritize.

And you're not advocating jumping into your suppliers' development environment and validating what they're doing. You know, if I'm hearing you right, you're saying these are the best practices in-house. Yes. And you want to hold those best practices. And when you do your inquiries with your third party and you vet them and work with them, you need to understand that they are doing the oversight, the compartmentalization. They are following these proper and respected practices around code building and attestation and everything else. Okay, okay, okay, good.

Yeah. And that means that you can put in a contract, you know, that you follow NIST SSDF, the Secure Software Development Framework, or like framework. I mean, we follow IEC 62443-4-1, which is stronger than SSDF. So there are things that you can do that you can put in that contract, but then there's services that can actually validate it.

Got it. Okay, so last question, because, well, this topic is just an ugly, gnarly mess, and why would you even want to crack the lid on this? But But from a cybersecurity leader's perspective, right, for those CISOs out there, what should they do now? What should they start doing tomorrow to get on this path to logically and in a controlled manner start to understand the risks, build a strategy, and implement the necessary controls? What's step 1 or 2 for them?

Well, step 1 or 2 is I think everybody's at a different level. And so in the book, I actually gave 78 controls to say these are in these different areas, different controls that you can do. And some of them, some of them clearly are, if you already are following this ISO standard or something like that, or already have a supplier process, you don't need to do anything. So in general, I think that we've been treating supplier management and third-party risk as a procurement topic, and we need to take it more from a— what we're more used to is a control topic. Do I have control of this situation? And that means having the appropriate documentation and records and really understanding, do you know your supplier? Landscape and are— where— how would you rate them from a security standpoint? And are, you know, are they a critical supplier in your eyes? Not, not your CFO's eyes, but in your eyes, are they a critical supplier? Because as I mentioned, it may not be about spend and really being able to identify your supplier landscape. So from a resilience standpoint, because a lot of people, if they've got a risk management plan and other business continuity plans, it's still not the same as the cybersecurity risk. And I haven't even mentioned too much about data breaches. Right. And so whereas cyber, you know, we realize, okay, that's going to be a large impact. You know, who's— a lot of small companies, they don't have, you know, data breach, you know, data officers and things like that. So you have to be everything, right? So I really Number 1 is understanding your supplier base and you identifying what's important and not important and what controls you're going to like maximize as the most utmost important. And then make sure your other teams, you know, procurement, legal, everybody else involved, including whoever decides, oh, you know, this is a cool new supplier. Well, a startup working with startups are great. But if you are giving them access to your entire business data, probably that's where you're going to list that one as a higher risk. So that's where I'm seeing most of it is we can't treat suppliers all the same. You don't ever have the capacity to do it. So just like everybody else prioritizes suppliers, you need to do the same. And what that generally means is you're looking at the services that you're, that you're getting. Like if Microsoft Teams went down in your organization, what are you gonna do? I mean, what if it's down for a whole week, right? We've seen Outlook, we've seen all those. Can you do business for it? Because they're gonna be calling you because I'm sure it's, you know, a cyber issue that they're gonna be, that they're gonna do it. And immediately the CISO is gonna be on the phone. So that's my number one tip. Item for is to understand where your suppliers are. And then secondly is, and I know this is a, you know, it's sort of more of a squishy kind of thing, is that we are focused so much on IT security that you really need to bring up the awareness of the overall rest of security inside of your organization. We focus on IT security, we focus on physical security, and also let's just say phishing and general awareness. But we're really not educating people yet on software supply chain security, what it really means. I mean, when we saw Colonial Pipeline, you know, cut off, you know, half the Eastern Seaboard, I was there on that side of the country at the time. It was a big deal because nothing was segmented, this and that. So I really recommend that awareness at the business level and doing simulation. As part of that is to really be able to say, okay, now I've got these suppliers, but what are we going to do in a crisis when this supplier goes down? And you, you know, you walk through the list, just tabletop it. It's not hard. Like, okay, what are we going to do if Teams goes down? Okay, what are we going to do if, you know, WebEx— I mean, if you're WebEx or Zoom— and what are we going to do? Let's just say Zscaler or something else, like, you know, you can't even log into your systems or Okta. I mean, we've seen Okta stuff happen, right? So all of those suppliers, and if you don't have sort of even just a baseline kind of tabletop and they're not aware of what it is and the impact to their business, you need to walk that through. And I think that overall we are the resilience leaders in the company. And as I said, something's going to happen and it's so much bigger than cyber.

Well, I love the advice because it's really about a mentality change, right? Change, you know, open your visibility, understand those risks, be able to communicate that out to the other important parties because people don't know about this stuff. And yet it's the CISO that really needs to be the expert and to own it. And to advocate and champion for it. Otherwise, it's going to be a big black eye. It's eventually going to happen. We all rely on software. So, and you had mentioned your book, right? Your book, Software Supply Chain Security. It's out.

Yes. Yes. Yes.

How can people get a copy of this here?

Yeah, it's on Amazon. It's on Barnes Noble and ebooks.com and different places. You'll have it, the link in the show notes, and you can always search my name. And another thing that's just with this book, I wrote it because I have all these conversations with people who do not know this. This is not— it's got technical topics in there because, yes, would you need to know what something is signed, you know, from a code signing standpoint? I explain it. So this is really a practical book with the controls and everything in it, and it goes through environments, it goes through people, it has a supplier section. And I wrote it because I work every day with procurement and legal and even technical people who this is their first thing. I, you know, if I ask for secure coding rules and I just get coding rules, I'm like, no, secure coding rules is different. So it's based off of years of working and teaching this with a large R&D organization being responsible for making sure that they understand product and application security and apply it. That's why it's really important. And that's why I wrote the book at that level. And it includes— it's a global book. So I mentioned all the different supply chain security regulations that might be out there, and they're changing every day. Cyber Resilience Act has— in the EU has information about supply chain, but also product security. So it's It's really hot, a hot topic at the moment and will always be, I think.

Outstanding. Yeah. Thank you, Cassie. Thank you. Really appreciate you sharing your insight and thank you all for watching and be sure to subscribe and catch all the Cybersecurity Vault episodes where we chat with industry leaders to dive into the most relevant and interesting cybersecurity challenges, perspectives, and best practices. We'll see you next time.