CrowdStrike Global Outage: Unpacking the Fallout and Future

Welcome to the Cybersecurity Vault. I'm your host, Matthew Rosenquist, CISO, industry cybersecurity strategist and advisor. And today we're going to talk about the massive global IT outage that affected over 8 million devices. And it was caused by a cybersecurity vendor, CrowdStrike. And I'm going to be talking with Ira Winkler. Ira is an award-winning CISO, top-rated keynote speaker, bestselling author, and one of the most experienced veterans in the cybersecurity industry. So it's always a pleasure. So thank you, Ira.

I'm honored. I should travel around with you giving introductions to everybody.

So, well, it's great to have you back on the show, and I really appreciate you talking about this. This is a really interesting topic. It affected so many people that made global news, right? And although this incident has not been characterized as a cyberattack, in fact, their CEO came out very early on and said, "Hey, this is not a cyberattack," but it has really brought to light some of the potential issues that cybersecurity solutions, 'cause they need a lot of access, right, that they may pose and the fragility of our global computing ecosystem. So, yeah.

Well, I mean, I'm, I don't know about you, but it bothers me more that this was not caused by a hack, malicious hack or whatever. I remember, and now I'm dating myself a little bit, but I remember back, I think roughly 2003 or so, there was a massive power outage on the Eastern United States where 50 million people lost power. And I was supposed to go into CNN because they were saying, oh, is this a case of cyberterrorism? Because obviously post-9/11 issues and everything like that. So they're sending a car out for me and everything. Then they called me up a couple hours later and say, Ira, we're not going to do the story because the FBI says it's not a cyberattack. A tree fell down. I'm like, and that's okay that 50 million people could lose power on the most— well, I'm from New York— on the most productive corner of the United States between Boston, New York, Washington, D.C., whatever else. And it's okay because it's not a cyberattack. I'm like, that's even worse. At least if somebody malicious does it, there's a reason, not because a tree fell down. And in this case, not because Somebody made a one-line error, which again is baffling that, you know, the fact it didn't take a nation state to bring it down. It just took a programmer who didn't test their code to take it down.

Well, I'm glad that it wasn't a cyber attacker because I think the outage that we had and the impact that we had, which is being tabulated in the billions of dollars, the latest number I saw was over $5 $1 billion of impact. I think it would've been far greater if it was an intentional attack. This was an accident. As you said, one line in a code, right, that got through all the checks and balances. If it were an intentional hacker, it would've probably been a lot more destructive.

I'm gonna be skeptical about that because frankly, when we have seen actual terrorist or criminal incidents, even those that intend to cause harm. We have never seen the harm that's caused. I mean, I go back to, for example, this is extreme cases, but it was Hurricane Katrina. I remember this well. It was around the same time again as 9/11. Hurricane Katrina caused more damage than Osama bin Laden in his wildest dreams. A tree falling down costs billions of dollars worth of stuff. And when we've seen actual attacks, the actual attackers have so far, I'm not gonna say never, never ever, never say never ever. But so far, the attackers have not been able to cause the damage that accidents and natural disasters ever could.

I would agree with you there, but we have seen, and you're really talking scale, right? The number of systems, how, but we have seen attackers do damaging attacks against targets. We've seen wipers really evolve since the 2022 invasion out of Ukraine. We saw attacks against oil facilities in Saudi Arabia, Saudi Aramco. That basically bricked the systems. So the scale wasn't as big because it didn't have the fingers and the reach, but of the systems that they did impact, they destroyed them. They wiped them. It was time to get in new systems, essentially. And so if we combine those two things and where an attacker does have a supply chain, and we are seeing supply chain attacks, and then they can do permanent harm, permanent bricking, and it absolutely can be done at the firmware level and even to a certain extent to the hardware level if you've got the right code. Guess what? I mean, you're not going to just have to go out and send a tech to reboot it and do a couple of things with a USB key. You're gonna have to go to the supply chain with everybody else and try and get a whole bunch of new systems. Good luck with that.

So I'm not gonna disagree with you that, again, it's not, possible because you look at, for example, SolarWinds. SolarWinds was detected as an exploitation, not because there's— everybody likes to, you know, I, again, dating myself. It used to be called information warfare. Now it's called cyber network operations, which is cyber network defense, cyber network exploitation, and cyber network attack.

It's all marketing. Yeah.

Whatever, wherever you want to throw money at, whatever the bucket is. But so now it's like, okay, what we tend to see by nation states is cyber network exploitation. You know, these are the people who truly have the ability. So for example, China has been accused of, oh, I forgot the name of it. It's not Broken Arrow. It's like there's some group out there where China has been supposedly infiltrating the power grid of the United States and other places, and they have been going ahead and preparing the battlefield.

Right. Not causing harm, but simply ingress and persistence. Yes.

Right. And the ingress and persistence, but more important, that is for preparation of the battlefield. Should they ever choose to attack, the logic is that they now have their proverbial finger on the button. And hopefully we are playing a game of cat and mouse with them to keep them off. And I would assume that we are playing, they are playing the same game with us. We're just better at it and not being detected. So the problem, and we've seen this back, I remember 2007-ish, Siobhan Gorman,, you know, wrote a story about how Russia and China have the finger on the proverbial button of the power grid way back when.

Yeah.

And so I still go back and I see this and I say, okay, it worries me because we are not even seeing an intentional attack creating massive amounts of damage. And now, is that a sign? I mean, the fact that our networks are not resilient enough for this, because I would say a good part of this whole thing is we are now seeing where weaknesses theoretically are. And if we have half a brain, which I don't necessarily give us as the collective IT, you know, companies and everybody like that have, if we have half a brain, we could look at what has happened and proactively do something about it. It's like Y2K. The only reason people did anything about Y2K was I think the US government had to pass a law. That basically said you have to have a Y2K plan that goes— that went into your like 10-K filing or something. Without that law, most people would have done nothing and just said, oh, we'll deal with it. But, you know, it just worries me that hopefully we're learning. Like, again, I was impacted by the Delta outage. So was my wife. And it was— it's bad that, you know, a major communication system that enables business that, you know, hopefully people weren't traveling to get a job, which probably was the case. I had friends that were stranded in Atlanta for several days. You know, hopefully whatever the case is would have learned, you know, honestly, to me, I look at CrowdStrike, you know, EDR endpoint, you know, endpoint issues, but clearly it's embedded in a lot of servers as well.

Yes.

But that goes to the point where, okay, so it's embedded in the servers as Well, the servers went down. So now we have servers as a potential critical point of failure. So we found this. I'm still going to go back to don't stop automatic updates because it's still— what we experienced is still infinitely better than if we weren't doing automatic updates. So I'll say that. But you look at this, and hopefully it's not just Delta, but everybody else is sitting there wondering, OK, What would happen if this happened to me? So now, I mean, I know a lot of people, hopefully, maybe it'll be a little bit like SolarWinds. I call it pulling a Tylenol, where I don't know how many people remember when somebody took poison and put it in the Tylenol capsules. And they created tamper-resistant bottles and everything. And they went to the wild extreme that I can't even get a Tylenol out of a bottle anymore, but that hopefully the airlines are going to start to look at this and anybody else involved in critical infrastructure is going to take a look at this and say, okay, what happens if my Windows servers get bricked? As an example. And I'm not saying— and not because North Korea is going to do this, not because a ransomware purveyor is going to do it, just because a tree falls down again. So hopefully, we'll be learning lessons from this along the way.

Yeah, I agree. We definitely need to learn lessons here, and I think it is a wake-up call for organizations and especially for IT departments. But, you know, piggybacking onto that, I think, you know, we kind of avoided a bullet here. You're right. It was a simple mistake. The fix wasn't super simple. In many cases, it required somebody goes, you know, hands-on on the device. Okay, but it wasn't truly malicious. It wasn't an intelligent attacker that decided, "I'm going to cause irreparable damage to every single one of these systems." And, you know, you had mentioned SolarWinds, and I found it very, very interesting. And it was a nation-state attack. And the moment that they were discovered, it became public, they were at a crossroads. They had a decision to make because they had access to 18,000 systems out there. Right. Or I'm sorry, 18,000 companies and all their systems, including critical infrastructure. And they could have, one, decided to— we're going to send a wiper package. We're going to send something that will destroy all of these systems. And they could have done that. Right. They had remote access to do that. They had permissions. They had the ability to install things without anti-malware blocking them because it was turned off. Right. Or they could go the other route and go less self-delete and try and minimize, you know, the evidence. And they chose at that time to self-delete. We saw them start self-deleting, which is great. If they would have went the other direction, it might have even constituted an act of war, right? The severe amount of impact that it would cause. But we can't ignore the fact that it could. Have been a severe impact. And if this particular case was malicious, it isn't Delta being down for a week because they don't know how to reboot their systems, you know, and, you know, get past, you know, hard drive encryption or whatever it was. This is down for months and months. And that could have been catastrophic, right? That's the difference between an oops, we made a mistake, let's go fix it together versus an intelligent adversary going, I am going to intelligently go after you. And I will compensate for what you're probably going to be able to do. Big difference.

Here's the case. And we have to start looking at, at what level are people willing to pay the price? And so is Russia or China at this point going to pay the price?

I think right now, and it may not be China, but Russia, North Korea, Iran.

I was getting there.

Don't worry.

Okay. I'm saying Russia or China. At this point, Russia or China know that, let's just say, retribution will be swift and massive. That even if we don't go to, quote unquote, missiles flying as a result of it, we have the ability to shut down what infrastructure there is on a Well, let's just say from open sources talking to people, the US has a very significant capability of preparing the battlefield the same, but we are just better at it. We're not as egregious as China, where China's just like, no, we didn't. It's like the Shaggy song, wasn't me.

Right.

Watch you in the shower, it wasn't me. Have you on camera, it wasn't me. It's like China will just go, wasn't me, for as long as they have to. Russia at this point, I don't know, they don't want to be caught because it might be an embarrassment and maybe they'll be killed.

They don't care. They don't care at this point. We've got sanctions. We're giving military aid to their enemies. But it's degraded, unfortunately.

If they choose, however, to cause us to just say, hey, press the button from our— us having the button, it will be massively worse. On the other hand, you have countries like North Korea and Iran, like you mentioned, which at some point, how much do they have left to lose?

Yeah.

And in theory, are they as capable as Russia and China? Definitely not. Have we like Iran? We could probably, you know, Iran has more of an active infrastructure to damage. North Korea, not necessarily anything. Like if they lose power in North Korea, half the country won't notice.

The half of the country doesn't have power. 80% of the country doesn't have power.

Exactly.

Okay, so one other thing though, because several years ago under Putin, when Putin was originally in power before he actually left and Medvedev, I can't remember, pronounce his name. He came into power as a puppet, but they created a, an inter— what they called an internet circuit breaker, right? Where they could disconnect the Russian internet and they went down and they codified it in their laws that they had to have this, they had to test it, which they did. They built it, they tested it, they have it. They also then started a program of self-sufficiency. Right? So whatever is in the West, okay, we've got Google in the West. Well, they've got their own version of it. We've got Facebook, they've got their own version of it. You know, we've got all the— and so we see duplicity there and these digital twins that have propped up. And just as a great example, right, this issue with CrowdStrike didn't impact any companies in Russia. Now, that wasn't because of the circuit breaker technology or anything. But they've already thought ahead a few steps, right? The chess players that they are. So there's, you know, I think there is still apprehension because they still have vulnerable systems. They still need connectivity, especially out to their allies. But I think there is less apprehension, and I think it is a disproportionate impact. And they recognize that if the global internet goes down, the US and its allies suffer far greater than what Russia will, especially given its current sanctions and its ability to isolate its internet.

So here's the thing. If you say you can isolate the Russian infrastructure, that's like saying a major corporation can isolate their own networks as well.

Mm-hmm.

It's not gonna happen. You know, frankly, if we haven't prepared the battlefield to act autonomously, I'll use that word. And I'm just saying, how would Ira hack them? I would say, hey, you have your kill switch, fine. If we don't get a proof of life, your network, your— once you isolate yourself from the rest of the world, you go down completely. That's just the world according to Ira. If I had my free rein, I would put malware resident on the network. And if the malware is not able to communicate back home for a period of time, it's like bye-bye network. Now, do other people in Cyber Command and other places think the way I do? I would say I hope so, because it's the most obvious thing to do.

So not that I want to get into technicalities, but the moment you start putting resident malware, right, to be able to remotely think, and so it becomes much more visible, your ability to remain stealthy and undetected becomes much more difficult. So yeah, in a perfect world, that is an option. But again, now you're trading some of your stealthiness and deniability and persistence for, you know, you're exposing yourself.

The Equation Group software sat there for more than a decade before people finally found it. Yeah, good point. So let's just say in order to get rid of it, you know, much like they would have to take down their entire network to get rid of, you know, because I've worked for companies that had Chinese malware on it. And, you know, we were sitting there debating, how do we get rid of it? And the way you have to get rid of it, because it's a whole self-infecting It's brilliant software. And it's, it like self-infects. If you take one system down, when you reboot, when you take the one system offline, bring up a whole new system, fresh operating system, within, you know, a couple minutes, it's going to be reinfected. Yep. And, you know, I'm like, well, what do we have to do? The guy who was like, what do you have to do? Well, we have to take your whole network offline piece by piece, like just massively wipe, you know, just take that whole factory down, clean it off, isolate it, and everything, one thing at one system at a time to do something. Yeah. I, you know, and that's assuming we found it. So anyway, I'll leave it at that.

Okay, I want to get back to CrowdStrike. Right? I mean, because this issue, it came out, and I think some really interesting things happened. Both good and bad, right? So let's briefly talk about, you know, what they did wrong. We know that there was a problem in an update. You know, it was a single line. It caused basically a memory issue and blue screen of death, you know, was the result of that. You know, what are the top things you think they did wrong in this particular situation?

Well, the one thing that's obvious, you know, you don't have to be a genius to figure it out, is they never tested or at least they didn't test this update. You know, George was very specific that it was not a content update. It was a content update. It wasn't a software update. So in other words, a supporting, you know, a file that the computer that CrowdStrike needs to run, one of those 50,000 files that you have to install with the CrowdStrike software itself. They updated one of those files without testing it, I assume. And that, once it was installed, basically bricked the system because it did. And to me, the first obvious thing to do is say, wait a second, before we deliver any software, given the fact we're tied to the operating system fundamentals, we better test not just software updates, but content, sorry, Dr. Evil content update.

The dynamic updates, yes.

Yeah. And anything that gets updated, you know, sent to production, because here's the problem. When you are an organization of any significant size, you don't have the ability, or you lose the ability to go ahead and say, I'm going to test every update before I install. It would be great if you could, but I think you were the CISO of a large organization. I was in large programs, and it's— yes, you can have your team go ahead and before an update is pushed out, they can test it in a lab. The problem is that in many cases you might test it to the most common software, but even small organizations, I mean, they might have 17 different versions of PCs as an example. With who knows how many different extrapolations of software combinations because, you know, right now this impacted the operating system, which is normal. A lot of software updates also might impact other applications that are running. You know, like for example, I've seen updates that stop database software from running as an example. Like I think it was like a Unix update once stopped Oracle from running or something to that effect. And that's bad. But in this case, you have a piece of software and it becomes incumbent upon the vendor who's delivering the software to make sure it's tested at least when you— if it was, I mean, I'll have to say it this way. If it was like 100 systems of a weird configuration, because there's countless systems that exist on outliers. Of every different hardware, every different software, then maybe. But when you have the entire 8.5 million computers, I think it was, that goes beyond missing a niche. That is something that should have been tested. And it's obvious in hindsight. And I'm pretty sure somebody is— I'm pretty sure a lot of people inside CrowdStrike are sitting there, what do you mean we don't test, we haven't tested these updates? You know, cause this is like, I would almost say it's like 101, but somebody's like, oh, it's just a content update.

Yeah.

And I'd be like, any update, cause some moron will do something. And I don't even know if they're, you know, divide, I think it's like a divide by zero error and a referential memory error.

So yeah, it's just, yeah, it's just ugly.

I mean, it's like the equivalent of divide by zero. Somebody should theoretically test to make sure this doesn't happen on at least major numbers of configurations. That's what any good software vendor should do is test to ensure it works in their environment. Yeah. I'll leave it at that because I can—

I'm with you. I think the biggest area, the biggest point of failure is the quality assurance, right? And for any software, you know, piece of software out there, and most of them nowadays do send some type of updates or telemetry, bidirectional telemetry. Telemetry. But when we're talking about EDR, and especially with security software across the different suites, not even just EDR, but they're constantly updating because we have intelligent attackers and we have to adapt to them. We have to do it very quickly. So there is typically some type of bidirectional telemetry that's going back. And when we talk about these dynamic update files, you know, back in the AV days, they were called DATs. DATs, DAT files, right? We've seen this exact error happen before. Back in 2010, McAfee sent a bad DAT file out to its customers and bricked them all, right? And so, and it was a horrible outage. I was at Intel at the time and I had to help with, you know, the IT crew figure all this stuff out and resolve the issue. And it was painful. And then 2 weeks after that, we announced we were actually buying McAfee. And I was thinking, what are we doing? Who's making this decision? His decision? Why didn't you contact me? I would have vetoed that. But okay. But we've seen this error. This is not unique. This is not the first time. And when you read the— CrowdStrike recently just came out with a PIR. It's actually a preliminary post-incident report. And they outlined a whole bunch of things. And they outlined how for that software, that content, they call it the content, right? Which is really the program. They thoroughly test that. So before you get a new engine update, they're thoroughly testing it. Awesome. But then they go on in a very convoluted way talking about what they do for those dynamic updates. And that's those data files to say, hey, what the engine should do and how it should do it. They don't. They actually rely on the quality assurance that's resident in their program that sits on the client's systems. And, you know, they describe it as being very resilient. If there's a problem in this update, it'll recover. It'll be very resilient to any kind of issue. And we now know that's not true, right? There are—

I mean, honestly, when you say that, the one thing that comes to mind is the meme of Anigo Montoya, it's like, you keep using that word.

Yeah, I don't think I know what that word means.

The word means what you think it means.

I mean, they are making a fundamental flaw, right? And that fundamental flaw is you do need to test any code that goes out to your end customer. Otherwise, you're using your customer's production environment as your test environment. And that's just bad business.

No, I mean, that's like, I don't think, frankly, if you read the terms of service, I don't think you are going to be our beta tester by default. It comes up as blatant as it should in that case.

Okay, okay. So QA definitely, you know, and that needs to get fixed. So let's turn it around because I think there's some good things here. What did they do right?

So in my opinion, and there are two things, I mean, fundamentally, I think they did right. First off, I like the fact they owned it. You know, I knew George when he was, I knew George when he was like an associate at Ernst Young, I think, or PricewaterhouseCoopers, maybe before it was PricewaterhouseCoopers. I can't remember where it was. In the Cleveland, Ohio office. And if I ever thought I'd see him on, on the Today Show, which I did see him on when I was ironically in Minneapolis and turned on the TV and there's George talking about how he caused billions of dollars worth of damage. That would have been a good one. But you look at that and you stop to think, okay, at least they came out, they owned it. It wasn't like, okay, the customers did anything wrong. It came, and why this is important is when you take away who's at fault, you take away a whole bunch of distraction to let people, because too many people, like I remember, and like the big thing is attribution. Like to me, when I was like the power outage I mentioned, does it matter why 50 million people in the US Northeast are without power? No, it doesn't. We got to get everybody up and running. It doesn't matter if it's a terrorist or whatever. In this case, we took away the distraction that would have likely come on like, gee, what was the matter? Was it Microsoft? If it's a Microsoft problem, well, gee, then it's this. And then, but at least CrowdStrike came out and said, okay, this was our fault for lack of a better term. And I'm going to assume they likely had lawyers, you know, hopefully.

The lawyers probably weren't happy at that, but yes.

I mean, because they likely had like lawyers go ahead and review this. But, you know, the reality was they came out, they took ownership, they hopefully came out and helped people fix it. I was not in an organization impacted by this, so I don't know the level of support. And that allowed for the distract— that allowed for distractions to go away so we could come up with a definitive solution for the problem. As quickly as possible. And then people could then run off and do this. And then there was a little bit of the reason we're here was I put up a post regarding they tried to at least give a $10 gift certificate to the—

Oh, we'll get to that. We'll get to that. Hold on.

We'll come back to that. But that I thought was at least it was the only time I have ever seen acknowledgement for the actual people who have had to deal with the incident firsthand. And that, at least to me, was notable.

For me, and I've got to give huge props, tremendous respect to George, right? The CEO. He did come out very quickly. And for everything that you said, yes, it takes away the distraction. But the other thing is it simply was refreshing that instead of a company being remaining silent, oh, we'll wait for the experts to point to us. And then, you know, Instead of delaying it, instead of trying to minimize it, instead of trying to point blame at somebody else, some other tech company or the customers. We've seen security companies point at their actual customers. Well, it's their fault, right? It's their fault.

Yeah.

Ah, you know? And so it is very refreshing to see actual responsible ethical leadership. Bad things happen. They do. And, you know, our technologies are very, very complex. They're going to happen. But if you don't have good leadership that can immediately step up, and he did, he stepped up and he fell on that sword. He said, mea culpa, it's us. It's our product. It's not an attack. We know what the problem is. We're now focusing on fixing it. But did it quick, was open and honest, was ethical about it. Got everybody just as you said. Okay, let's not worry about the complexities. Let's not worry about if it's an attacker. Let's not worry about if it's Microsoft. All that's aside, let's just focus on the fix. Okay, get everybody up and running. And that ethical stance, that leadership, refreshing. It's what's needed. And that's why I'm still very pro CrowdStrike.

Well, here's the one— well, two things come out that I should mention as side notes. First thing that comes out is, in one way, I was actually happy that so many people were impacted because now I at least know a lot of good companies are using it.

I'm so grabbing that soundbite. Oh, I'm so trashing you right there just with that. Yeah.

So that's, I mean, it's like the people are like, at least people are using it. That's one thing.

Yeah.

The other thing that kind of came out was there's a little bit of poetic irony, because a lot of software vendors came out and criticized Microsoft for bad practices, which I'm not going to comment on myself at the moment. But I will. I mean, we can see where that goes. But, you know, but the fundamental hypocrisy is now I think CrowdStrike was among the people on the bandwagon criticizing Microsoft. And they were, you know, this is as, I don't know if I could, I don't know if there's a scale to even measure which one, if this is equal, but it's showing that a bad practice is embedded in both organizations. And, you know, for one vendor to call it out, because I really hate vendors that call each other out. If somebody's going to say, look, here's where we're better than them, but I remember you know, different companies, like it never serves them well when one company is trying to call out another because eventually it comes back to bite them. In this case, it came back to bite CrowdStrike.

Yeah. But it wasn't George that did that though, right? It was some other—

Well, it was like, it was a conglomerate. I know, like, for example, Tenable was part of it. I know, You know, I know that DHS came back with a cybersecurity review board or something and criticized them. You know, that's— it is what it is, but they can't stop and say, okay, but all these other vendors are doing everything perfectly. No, call out one vendor for doing something. But, you know, the— what happened with CrowdStrike is almost Quality Assurance 101.

Oh, it is. It absolutely is.

And I'm not there to blame them because at the top, you assume that all these things are done, but somewhere, someplace, decisions are made to simplify or whatever. And these things happen, unfortunately. It doesn't make it right.

I'll only say one thing about the relationship between the culpability of Microsoft versus CrowdStrike. In my mind, CrowdStrike was 100% at fault for having an untested, flawed package go out to their customers and brick them. Now, where I look at Microsoft, I actually hold them accountable for an opportunity, right? The opportunity is we haven't changed the blue screen of death for 20 years, right? We haven't well isolated applications and their resources. So if one critically fails, it can bring down all the rest. And that's an architectural thing. And that's very, very complex. And you have to tie operating systems in with VMs, with firmware and hardware and everything else. But I think Microsoft is one of the few leaders in this space because of Windows that it should be driving the industry forward. And I don't think it has over the last 20 years. So do I think Microsoft is at fault for this CrowdStrike? No, absolutely not. It's 100% on CrowdStrike. It was quality control. It was their product. It was their update. I'm fine with that. But I do want to use this as an opportunity to call out the fact that Microsoft and other operating system vendors have an opportunity and have, in my mind, a responsibility to evolve their products, knowing that the landscape is getting more complex, the attackers, the, you know, the IT world and everything else. They should be evolving their product to be much more resilient, to be able to isolate and compartmentalize the different applications within their environment. So that's all I'll say about it. But I do want Microsoft to—

I'm not gonna disagree on to a large extent, it would be nice to happen. You know, I'm trying to think if I have other vendors I'm trying to think of the other operating systems that are out there short of like, you know, like defense-grade operating systems when you're talking about, you know, like A1-level secure systems.

Yeah, right. You know, RTOS systems, you know, dedicated, much smaller. You know, it's tough, but you know what? I think we need it. And I think Microsoft is in a place that can drive that. They've got the relationships across the firmware, the operating system, the hardware drivers. Everything else, if anybody's going to make inroads for the betterment of us all, it's going to have to be Microsoft.

Yeah, I mean, we need malware writers, you know, APT-level malware writers to install that level of resiliency.

Yeah. Okay, so where do they need to go forward? And their PIR talks a little bit, and I have issues with what the current preliminary PIR states is their kind of improvement plan, right? They've said that, hey, we're going to make that, that interrogation engine that sits on our customers' clients, we're going to make that better. And we're going to allow better communication of when we're sending things out and, and what's in the updates. We're going to And I'm thinking, well, wait a second. So you're still going to allow an architecture that sends untested updates out to the end client. And to me, that's the fundamental error that needs to be corrected. Before you send any updates, you should be testing it. And even if it's a high-level test, right? Or a small narrow test, but you need to— causing blue screens, you should have recognized that. Any first-day Q&A person would have recognized that.

Yeah, I mean, to me, a large extent was I almost fail to, I almost fail to understand even knowing how operating systems work, knowing how software works at the operating system level, that somehow this never came up on a test system that somebody basically just changed a .dat file, copied and pasted a a pointer in and all of a sudden it never made it through. You know, they never even had it run on the system itself that they made the update to. And that is kind of on the baffling side. But I mean, I'm sitting here thinking, okay, what could they do? I mean, they need to learn the lesson well. I think it's sort of like, okay, we have an issue of pushing things out because there's the, I'll call it the argument that could be made that, well, when we make an update to these content things, they have to go out as soon as possible to increase resiliency and security.

And closing the window of opportunity for the attacker. Yes, absolutely.

You know, you take care of like decrease the window of vulnerability. And this hasn't gone wrong in a decade. And I mean, you could say that, but clearly the fact it hasn't gone wrong in a decade at this scope of, I mean, the cost is in the billions of dollars, as you mentioned. It's like, I think, you know, CrowdStrike's market cap was like, it was a significant, the damage they caused was a significant portion of their market cap. Yeah. And I don't know even if their insurance is going to be able to cover this. And they have to realize what is the scope of damage that we can cause because they are a security company. And there is a valid contention like how long would it have taken for us to install this update on a PC and boot the PC? You know, and I don't think it's long and longer than would justify let's give it a shot. And so that's number one. But then there has to be like a full analysis of what else, where else can this be? Because much like, for example, I mentioned, you know, how I was personally impacted by Delta because of what happened there, along with friends, family, you know, Delta is a good airline. I know there are people who are good smart people, and they need to look back and say, okay, how come this happened? How come it got this bad? And what could we have done? And more important, it's an opportunity to— I don't want to call it tabletop exercise because that's more for people, you know, like how do we deal with external purposes? But they should be going through a lot of reviews of their architecture. And not just Delta, everybody else got lucky. You know, what would have happened if it was another McAfee error and it was like another airline using McAfee instead of, you know, or sorry, it's not McAfee. What is it? Trellis now. But, you know, it's like now there's no McAfee didn't put the company out of business. But every organization should sit down there and start reviewing their architectures and saying, okay, where is our resiliency at stake?

Yeah.

And what would happen if Windows systems bricked up? Because, you know, I'm a little bit more optimistic against hackers. You know, like what happens if the worst happens? What happens? And we don't have to care why it happens, maybe to an extent of extrapolating out the possible results. But What happens if we lose our backups? What happens if a data center goes down? What happens if a significant portion of Windows servers just crash for whatever reason, whether it's an attack, whether Microsoft pushed out a bad update, whether it's another CrowdStrike issue or whatever? And people need to start going ahead and doing this because I remember when I was CISO of an organization, just as an example of things, the first thing I did was saying, Okay, let me, you know, I started reviewing things. I'm like, oh, simple thing. Do we have a phone list so I can call people if the network is hacked? They're like, well, you're worried about our network being hacked? I go, yeah. I go, I'm worried what happens if there's a potential incident and I have reason to doubt the integrity of our Outlook servers, for example. I go, I want a phone number so I could at least start texting people. At worst case and start to say, hey, you know, here's what's going on. Can you come in? Because what happens if Teams go down as an example?

Yeah.

You know, like resiliency. I'm like, if Teams go down, how can I get in touch with my people? They're like, well, I go, does anybody have a phone number for the watch? You know, it's like, well, we call them up on Teams. I'm like, I go, I want this list. They're like, it's a privacy issue. I go, it's not a privacy issue.

It's not.

And I had a fight to get, you know, so these are the things like us, like, and I was a new CISO walking in and just saying, you know, like just, just doing, what do you call it? Housekeeping things.

Right. The basics.

The basics. Like what, what happens if, if, if I fail? You know, I hate to say what happens if I fail, but frankly, that's the mark of a good CISO almost.

Yeah.

Happens, are you planning for your own failure? And it's not because I'm bad at what I do. It's because something will go wrong.

Yeah. And our job is to prevent things.

You're a fool or a liar.

Yeah. I mean, our job is to prevent things where they can be prevented, but it's also to minimize the impacts when bad things do happen. So it's not, you're going to put all your eggs in and I'm just going to count that I'm going to prevent things. No, you need to have prevention and detection and response, recovery, all those kinds of aspects to really be able to deal with stuff.

Right. I keep telling people, we're not security people. The definition of security is being free from risk. Our job is management of risk. Our job is accepting the risk that we need to accept, mitigating the risk we can reasonably mitigate, transferring the risk, or whatever the case is.

Right.

But we have to plan, you know, like how and why a lot of CISOs fail is because they fail to define that they cannot provide perfect security. They have to define that their job is managing risk and dealing with the inevitable failures that are going to happen in a way that creates as little, well, or mitigate the optimum amount of risk that they can. Because you don't want to say, oh, I want to minimize it because then you're spending way wasting money, but you want to optimize it. And that's the goal of a CISO. And going back to your question, you know, we need not just CrowdStrike to go ahead and take care of what's the obvious, but take care of what's less obvious, which is, okay, now I need you to go ahead and let's go through all these other scenarios that could go wrong. And I hope Tenable, SentinelOne, and all those other people are doing the same. But likewise, all these enterprises have to go ahead and fundamentally implement these— I don't want to say what-if scenarios, but go ahead and do the brain resiliency exercises, which is more than— which is much more technically involved than doing tabletopping where it's like, oh, here's a scenario. What would you do? What would you do? You need to figure out, okay, What's going to take your network down and keep it down? And how do you make sure it— when it happens, you're mitigating it as quickly as possible?

Yeah, and I would even extend that further. I don't see this particularly as just a security issue. This is an issue for all software, right? If you have critical software and it's doing some type of bidirectional updates or telemetry, even going back and forth, that can cause a critical failure. And so I think organizations need to look at all their software and figure out where are those single points of failure, right? And what would you do if it brought down that or brought down a host of systems or blue-screened or even caused a permanent bricking to where it isn't a reboot, the thing will never reboot again in its lifetime, right? And run those exercises. Yeah.

I've seen cases just for example, where you don't even have full control over your own architectures. I've seen cases where ISP-type organizations, their load balancers failed and all of a sudden people couldn't get to a corporation and people need to go ahead. And that, yes, that's a networking, that's a cloud security effort. It's not necessarily cybersecurity. But, you know, I look at a CISO as being a good partner to the entire organization. And if a CISO has to go, look, I go, you— one of my— I have Ira-isms, and one of my Ira-isms is you can lead a horse to water, but you can't stop them from pissing in it. And a CISO has to at least go ahead and go to the CIO and say, look, we need to go ahead and examine our resiliency. Resiliency is not the job of a CISO per se. We have to go ahead and protect you from theoretical attacks, make sure confidentiality, integrity, availability— availability is, you might argue, resiliency, but it's not the type of resiliency that goes into a traditional network architecture task. So a CISO should at least raise it to the level of the CIO. And if a CIO is not already thinking about it, that CIO is negligent. Because they should constantly be concerned about what is the availability of my systems and so on. Because I know a lot of CISOs and cybersecurity teams got involved in the CrowdStrike issue, which fundamentally is not a cybersecurity issue at the end of the day.

Yeah.

I mean, it involves cybersecurity software and some CISOs somewhere are having fingers pointed at them saying, you're the one who told us to buy CrowdStrike. But the real issue is, okay, yes, I acknowledge CrowdStrike went down, but we have a whole bunch of systems that went down. What was the backup plan? Why did all these systems go down? And why is it so hard to have resiliency on our network? That should have been planned for proactively. It's unfortunate it happened through cybersecurity software. But it's still a critical issue that should— I don't want to say should have been addressed, but right now we now know it should be addressed. And companies around the world should be asking themselves, what happens if all of a sudden there's a problem with Microsoft, there's a problem with Linux, there's a problem with macOS or whatever else it might be? You know, we need to make sure that we have resiliency across our environment.

Yeah, I think you're making great points here, right? It comes down to that resiliency. And I think we agree, although it was with a security software vendor, it wasn't a security issue. It was an IT outage issue. But it is bringing the, I think, a lot of really helpful discussion so that we can learn forward to figure out, okay, how do we ensure resiliency, whether it is a cybersecurity issue or an IT issue? Okay, so final thoughts here from you, right?

Yeah.

So mistakes were made, right? It resulted in catastrophic impacts. Trust has been lost from CrowdStrike. How does CrowdStrike get some of that trust back? How do you see them doing that here in the next few weeks?

So I'm a little bit, I don't know if I'm necessarily the right type of, I'm always biased because I maybe I'm overoptimistic, but I always think a good company when they are involved in an incident comes out even stronger than where they were before. And it depends why those issues are fundamentally there. I give you that if it's an endemic problem that says, oh, there's fundamental flaws in the software, that's one thing. But with CrowdStrike, I don't see it being a fundamental flaw I mean, obviously bad implementations were made, but it's not a fundamental flaw in their software. And if I was CrowdStrike, I'd— and frankly, I'd do a lot what Tim Brown at SolarWinds did, where they went out, they constantly assured people of what the problems were, where they constantly said, open book, here's what we're doing to make sure it never happens again. And in that case, I'd rather go with an organization that royally fucked up. That understands what the fuck up is and is constantly examining for more fuck ups, as opposed to another vendor who's just going to come. And I hate vendors who play like, you know, ambulance chasers who come out and say, hey, you heard what happened there. It's like, I mean, dating myself, I was involved in the original Citibank break-in back in 1994. And what happened was Citibank acknowledge that somebody broke into our computers and stole $12 million. And I'm like, why are they making that? That's a drop in the bucket compared to what's stolen at banks. But they use that as an excuse to enforce multifactor authentication. And all these other banks were going after the large Citibank customers and saying, hey, we don't need any of that multifactor authentication. They got as much stolen as Citibank did on a regular basis. But the reality of the circumstance was, was that, hey, Citibank used it to implement better mechanisms. And if they lost customers because of the better mechanisms increased what I'll call is friction in requiring multifactor authentication, then so be it. Because CrowdStrike will probably lose a few customers. There are people who are going to have a grudge against them. I don't— I don't deny people their grudge and just say, they cost me enough money, they ruined whatever, I'm gonna leave. Fine. At the end of the day, CrowdStrike could just go ahead and continue what they've been doing to this point, which is creating trust, creating goodwill that they have done and continue to do it because I promise you, we have the memory of a goldfish in this community and everywhere. People will forget really, really quickly. Maybe not, you know, I did own, well, I do own CrowdStrike stock. I must admit that. Not a, you know, I'm not a few thousand dollars, nothing major. I'll be upfront about that. But it's, I don't have a strong bias for them or against them, but they just got to do what they've been doing because what they've been doing has been doing pretty right. So far with, aside from a major, and I'm not going to say it's a fuck-up because a fuck-up sounds like an isolated issue. This was a process that they had in place that they are hopefully going to address. Short of that, there might be other processes we haven't witnessed, but other organizations are going to have other processes likewise heinous. But, you know, they just need to do what they're going to do and they, and they're going to be spanked in a major way. I could promise that you are going to have class action attorneys crawl out of the sewer over this. Yeah, you are going to, you know, and people should take their $10 gift certificates because that's all the workers are going to get, I promise.

Well, you know, I would agree that I think they need to continue on the path because they started out with being very upfront up front, very timely and very transparent. I think they need to continue down that. That is one of the biggest and best strengths, to be able to show and have confidence in leadership. The second thing is they, they need to figure out and fix those systemic issues. And again, in their, in their preliminary PIR, what they're calling out is putting a Band-Aid. It's not actually fixing the, the core problem. So I think they either need to get some people in there to help them figure that out, or they need to wake up and go, okay, yeah, That makes sense. We will never push code that is not at least tested, right, in our environment. Okay, we get that. We'll change our architecture. But I think in doing that and keeping in contact with their customers, just as you said, to continue to reinforce and strengthen that rapport, to take in those lessons learned. And I think we're going to have other outcomes, right? I think we're going to have insurance companies that as part of that 600-questionnaire for their cyber or IT security, it's going to have a few more questions about, hey, tell us about your quality assurance processes for everything you send to your customers. I guarantee you it may take a few months, but I think those questions are going to appear.

And here's the problem. Let me just say, I understand they will make those questionnaires more difficult. The problem is they don't read the questionnaires. The problem is they don't know how to interpret the answers. You know, because I promise you, if somebody would have read what CrowdStrike's doing now, if an insurance company read that, any underwriter would have looked at and said, that's why.

Okay. Yes, sir.

Yeah.

That's the problem. Yeah.

I don't, you know, it's like the, there's a lot of things that are going to cosmetically happen.

Yes.

Cosmetic things, you're going to have, you know, there were, I don't know, was there a Senate hearing yet? I can't remember, but.

He's been called in. He's been called in to, I think it's House. So it's under Homeland Security.

Yeah, there'll be congressional hearings. Congressional hearings are for show. Nothing productive ever came out of a congressional hearing, no matter how many people want to ignore or think otherwise. You know, there'll be lawsuits. That's not going to change anything. That's just a transfer of wealth between CrowdStrike and the opposing attorneys. Everybody else will get a coupon for free AV software, you know. And it's— so there's not going to be that much. We need to make sure that CrowdStrike realizes this, much like Microsoft did in 2002, I think it was. They need to realize that it shouldn't be as bad as what Microsoft was in 2002, because CrowdStrike generally has good practices besides of what's been outlined here. But, you know, they need to go ahead and do a bit of a stand-down and say, okay, let's review every possible process we have, and where can we fundamentally improve those processes? And if they could do that, I promise you, they'll be better than just about any other cybersecurity software company out there.

And I think they have the leadership to do that. Again, you've got a good leader They're going to drive that. I'm not too worried about that, honestly. I think they will emerge out of this much stronger than not just themselves before, but as you said, the rest of the industry. So, hey, Ira, thank you so much for sharing your insights here and experiences on this very timely and somewhat painful and very visible topic to the world. People are still recovering here a little bit. So thank you very, very much. I appreciate you coming in.

It's my pleasure. You know, I love to rant off with you.

So do I. And thank you all for watching. Be sure to subscribe and catch all the Cybersecurity Vault episodes where we chat with industry leaders to dive into the most relevant and interesting cybersecurity challenges, perspectives, and best practices. We'll see you next time.