CrowdStrike Outage Explained

Let's talk about the CrowdStrike outage in plain English. A LinkedIn follower asked me to opine on the CrowdStrike outage situation and describe in plain terms how the issue we see unfolding could have happened. Okay, so let's dive into it. First off, what is being said in the news? Well, it's being described a bit technical. Right, a faulty update by CrowdStrike caused a kernel driver error that resulted in Windows blue screens of death. Yes, technically that's absolutely correct, but let's go ahead and translate that and explain it in plain English. All right, first off, let's talk about cybersecurity tools like CrowdStrike. These type of tools need deep access into the operating system to monitor other applications, data connections, users, and even various operating system telemetry. And they do this for a couple of reasons. First off, they need to detect suspicious and malicious activity. And second, they then need the power to interdict, contain, and eradicate, and in some cases, even repair, right? All those attacks and whatever methods that were detected. And that means it needs some kernel access. Now, the kernel is the deep core of an operating system. It's where all the control resides. Think of it as the powerful brain. Now, the CrowdStrike Falcon product is an endpoint agent. And don't worry about if it's an EDR or MDR or XDR. All that is just marketing gibberish trying to sell you stuff. It is an endpoint agent. And that means it's a program that resides on the PC or server or device. It runs locally to do its security thing. Okay, so let's talk about the attackers. Well, as attackers are constantly adapting and finding creative new ways to exploit systems, endpoint protections need to be updated to keep pace. The cadence of the updates can vary greatly. Back in the day when anti-malware, antivirus, uh, first came out, it only ran once a week, maybe once month and was rarely updated. But that slowly changed, and nowadays such products can receive new instructions several times a day. Some endpoint products which look for general anomalies may not need updates for months or even longer. So there is a huge disparity. But anyway, CrowdStrike did send a flawed update.. And that is where we saw the problems begin. Although that's not where the problems actually began, but more on that in a minute. Okay, so let's talk about this bad update and those BSODs, those blue screens of death. The flawed update, once uploaded and employed by the system, basically began doing some privileged things. And remember, it has deep access to the system. And these things weren't proper, which caused a critical condition that was detected by the operating system. For such critical conditions, the standard response for Windows is to halt all functions and display that dreaded blue screen of death, or the BSOD. The BSOD is a throwback to one of the early versions of Windows. And the thinking at the time was that if a program was doing something really naughty, The system should fail safe to protect the data and stop the harmful activities. That was great 20 years ago. It was all we could really do given the limitations in hardware and software. But that architecture has never really evolved the way it needed to. Remember, attackers have rapidly evolved, but the BSOD has largely remained the same. Let's talk about the impacts now. So with CrowdStrike being one of the biggest cybersecurity firms and having a large user base, when the bad update went out, the BSOD screens began lighting up like Christmas at the Griswolds. Okay, so that's a movie reference, Christmas Vacation, Chevy Chase, go look it up. But to add insult to injury, security tools often load very early in the boot cycle so they can observe what other programs are doing at startup. Which means if you get the blue screen of death, even if you reboot, the problem reappears before it's easy to do anything about it. And it becomes a nasty cycle of blue screens and reboots. It is fixable, but to fix it is not necessarily easy or straightforward. There are now instructions and tools from CrowdStrike and Microsoft which are available, but even some of those require a tech person to visit the device, put their hands on the device, and that's not exactly fast or scalable given the size of this problem. Which Microsoft just reported so far, they're seeing about 8.5 million devices and they may not be able to see all of the devices. So this is a significant problem. Okay, now let's go back and talk about where the problem actually originated. CrowdStrike is a behemoth of a security software company and it is well regarded within the community. Well, at least until last week anyways. But they do updates just like all the endpoint security products do. So this is not their first rodeo. They likely have a very mature software pipeline, which is standard fare for, well, all software companies, not just cybersecurity, which means they have developers that create the updates. Those engineers have processes and tools to check for errors. Sometimes stealthy problems will get past them, right? Ones that would not be noticed. Then the code moves from the devs to the QA team, and that's the quality assurance team. Now, the QA team has a set of test scripts. It's typically very formal, right? And they validate the functionality, the backwards compatibility, the performance, and a whole bunch of other things. Right? And those are— this team is often tasked with finding those well-hidden, those subtle issues, then report them back to the devs so that they can correct them. And then they resubmit the package back to the QA testing channel where the entire test process starts all over again. And this cycle continues until the code passes and is approved, and therefore it is allowed to move from pre-production production to production, where it's then pushed out to the entire world. And even then, in some cases, it's pushed out in stages, not to everybody at once. And again, there's some kind of canaries that they push it out to, and if everything goes good and they watch it really close, then they start pushing it out to the rest of the community. Okay, that was a lot, but here's the issue. This was not a well-hidden or minor bug that would go unnoticed. A blue screen of death is about as big as you can get. This should have been easily caught by the devs and absolutely caught by any QA team that has a pulse. I mean, really, even a new QA person on their first day would notice this. So how did this get past all those checks? Nobody at CrowdStrike is saying, which is a little odd, but this is where the problem actually began. The quality control of the code that was written, it didn't stop it from going out. And that's why we have over 8 million devices and the digital services that depend upon them that need to be fixed. And although I won't speculate right now what might have caused that bad code to actually get past all those checks and balances and get pushed to users, I might publish something later if people are interested in my thoughts on the matter. So I hope this helps explain how we got into this mess, given the current data that we possess, and why so many people and organizations are frustrated. Thanks for watching. Be sure to subscribe for more cybersecurity insights, and let me know if you have any other questions or any other topics you want me to chat about.