Failures of CrowdStrike Quality Testing

Let's talk about CrowdStrike's quality assurance failures. CrowdStrike has released preliminary details of how their bad update made it to client systems, which caused the blue screens of death. It showcases they have a complex product release architecture, which in this case failed. Improvements need to be made, and I'm concerned that their plans for incremental changes to a flawed quality assurance architecture won't result in the desired long-term outcome. But first, let me say that CrowdStrike has a good reputation and leadership as showcased by their CEO, George Kurtz, who quickly came out to take responsibility and rally his team to help their customers. Now, there have been many companies, including security companies, who were not transparent or timely when their products caused problems. In fact, it seems more common to initially deny, downplay, or blame others when something goes wrong. So what George did is truly wonderful. Him coming out and quickly letting the customers in the world know, mea culpa, it is us, we're working on it, let's get this done. Well, it's a testament to CrowdStrike, CrowdStrike's work ethics. So yes, I am going to be criticizing the systemic failures that led up to the massive global outage and the preliminary plans that CrowdStrike published to address the problems. But let's keep in mind, this company actually did very well and it's showing very ethical behavior. It's been very responsible, and the leadership has stepped forward, unlike a lot of other companies who really just try and hide and downplay it. So definitely my congratulations to George and to his team. But let's talk about this, because this has been a major outage, and I believe CrowdStrike needs to revisit those preliminary improvement plans to account for a flawed strategy that allows for dangerous code to make it to the endpoints. And that's something that should never be allowed to happen, especially by design. So the world is watching and the lessons learned, this incident will likely be used to help improve the operating practices across the industry, not just cybersecurity, but across the software industry. So it is important that we talk about it. Now, CrowdStrike has released a preliminary post-incident review, or a PIR, of how the flawed update made its way to millions of Windows systems that suffered blue screens last weekend. First, the PIR, well, it's a little confusing to read because it attempts to assure readers that they apply comprehensive rigor when it comes to testing their products. I get it, it's marketing, it's legal who helped approve and write this thing. They're trying to not get themselves into any bigger hot water issues from liability or litigation. And the marketing people are, well, they're trying to spin it, right? Ultimately, we need to tell the marketing department to stand down because their failures are already obvious as over 8 million devices went offline. So let's not try and spin it. Let's continue down the theme that George started of saying, yes, we made a mistake. Let's go figure it out and figure out how we're not gonna make it in the future, okay? So cutting through all the confusion in their PIR is important. And here's the heart of the issue. CrowdStrike claims to test its main software product as it goes through revisions. This is the core program that would be installed if you became a customer. But they also send updates called rapid response content on a regular basis. This includes template types, which are instructions on how to gather and process data, as well as template instances, which are the data instructions processed by those template types. And that specifically instruct what should be observed or processed. So think of it this way. If you buy Microsoft Office, which is the software product akin to the CrowdStrike Falcon, for example, within Office, you have individual templates that can process various data structures like Word, Excel, all of those things, right? So those are the templates that can handle certain kinds of data and they do things with it. Now, that is equivalent to the CrowdStrike template types. And lastly, you have data files like a .doc file or an XLS file, right? And those are the ones opened by Word or Excel. And in this case, that represents the template instances that are interpreted by the template types. So essentially, you need Word installed to open a doc file. Okay, now CrowdStrike says they rigorously test all sensor content, including template types, before it's deployed to customers. But the sensor content is not part of what is dynamically updated. So this would be, you know, the core program, Microsoft Office, Microsoft Word. These are tested really well, but not the dynamic content, which would be those data files. So here is where we must read through the lines in the PIR, as CrowdStrike does not want to make their failure super clear. But what they don't test rigorously before sending out, according to the PIR, are the template instructions, or the template instances, which represent the low-level data that instructs what the sensor should be doing. Interestingly, they do say they are validated, but we already know that does not actually include thorough testing, as is evident from all the massive outages. So in this game of CrowdStrike Clue, right, the culprit was the content validator as part of the content configuration system, which failed to adequately test the flawed template instance that killed the Windows clients on the internet. So for all of you Clue board game fans, that's how it kind of translates. But it appears that CrowdStrike was okay with the lax testing because they believed their content interpreter, which resides on the client system as part of their software, would, and I quote, gracefully handle exceptions from potentially problematic content. They were wrong. So CrowdStrike has implemented an update architecture that only rigorously tests some of the updates sent to clients. The template instances are not thoroughly tested before landing on systems, and they instead rely on endpoint functions to handle any residual problems. This is a serious design failure for their product quality assurance. CrowdStrike should be properly testing every piece of code that is sent to client machines. Just to show how egregious this is, CrowdStrike actually allows customers an option of selecting which versions of the sensor engine, think of that as the big program, right? The Microsoft Office program. In this case, the Falcon engine will be updated on their systems. And many customers do this because they wanna be able to test those big engine updates, those big software updates. And they can keep with the current or the latest release, right? The end release, Or they can decide to delay and keep older n-1 or n-2 versions in place, right? And again, that enables them the time to do internal testing and validation on their platforms before committing to the move to the new update. But that option that's inherent to the system, it doesn't apply to the poorly tested template instances, which will go out to all clients. And victims have been attesting that regardless if they have indicated to be on an older version, N-1 or N-2, right? The sensor engine, you know, the older sensor engines, the customers were still impacted because they were updated with the new instructions, the template instructions or template instances. So it didn't matter that they had chose to be on an older version, which kind of says, well, wait, we're choosing this to make sure that we can vet software, we can vet updates, and yet it wasn't part of the deal. Okay, so at the end of this PIR, the CrowdStrike has identified several areas of improvement, and CrowdStrike has committed to more types of validation testing with an emphasis on the content validator and content interpreter. And those are the pieces that reside on the customer's system. They also want to detect failures on customer systems faster so they can respond in a timely manner once they've, you know, sent out a bad update. Now let me be clear, this is absolutely the wrong strategy, okay? Bad code should never find its way to the clients in the first place. It's essentially saying, I wanna use my customer's production environment to test and see if I've got bad updates and hopefully it will detect it and self-resolve. CrowdStrike should instead commit to conducting much more rigorous testing in their in-house quality assurance pre-production environment or the content configuration system which resides in their cloud before it reaches any customer client systems. They should never send potentially bad code to clients. Now, the CrowdStrike CEO George, George Kurtz, has now been summoned to Congress to appear before the US House Committee on Homeland Security. And I hope by the time he arrives to answer for this outage, he will have improved the plan moving forward and will clearly commit that all updates, including those template instances, will be thoroughly tested before ever being distributed to customers. Endpoint devices. Quality assurance is about keeping bad things from production environments. And given the deep access that especially cybersecurity tools possess and the widespread need to have security in our critical digital systems, we must really set a precedent in how these updates are properly prepared tested and deployed. In the future, when we look back on all of this, the CrowdStrike incident of 2024 will be seen as a major failure. There's no way around that. 8.5 million systems, you know, several billion dollars worth of losses and impacts. The most important question, however, will be, Did we use this situation as a catalyst for learning and adapting to make our digital world more secure, private, safe, and reliable? It's my hope that we will get stronger because of this issue. If you like the content here, please, uh, hit the like button and subscribe, and I'll talk to you next time.