Unveiling Cyber Resilience: A Deep Dive into Attack Surface Management #MSP at Roarcon

Alright. Hey everyone, what's going on? You got Wes Spencer and Alex Farling. What's going on, brother?

Not much, man. We're chilling here in Frisco, Texas, and I do mean chilling because we're indoors in a dome, and it's kind of nice in here. It's hot outside.

This is a practice stadium for the sportsball team Cowboys. Sportsball.

Dallas Cowboys.

This rivals like a college football stadium. This is where they practice. This is not just like game field.

I always heard that the star wasn't just a logo, but it was a rating. It's certainly not a rating for this building. Like, this is a 5-star building that we are hanging out in, having a workout.

Is that like a euphemism for they don't win very much or something?

Got it. Got it.

Well, I can tell you this place is stellar. And shout out, huge shout out to Lion Guard for throwing this event.

Yeah, yeah. They kind of took a shot on having their first event, bringing a bunch of MSPs to Frisco. Was supposed to be in a much smaller facility, and man, they crushed it. Yeah. And, uh, and here we are in a much bigger facility. They had to move across the street and expand and blow it up, and we have a whole football field behind us.

And it's creative, right? This idea of like, hey, why don't we try it in an NFL practice facility? Let's do a tour on the next day. I think people— it's—

we—

everyone talks about a— there's too many events, too much stuff going on. I don't think that's the problem. I think the problem is, what's the— why should I go? Why should I take time out of my day to travel to an event? What am I going to get out of it? What kind of experiences am I going to get? And there's a lot of ways to answer that question, but I think LionGuard's approach is twofold. One is let's have a great, really neat venue in an NFL practice stadium. And then secondly, a focus on attack surface management, I think is a pretty cool idea.

Yeah, I do too. It's a topic we don't talk a lot about in the MSP space, and they've already started the pre-day, so I'm sure we'll talk to some folks and hear how that went. But hopefully, you know, they've listened to the community and they're building on education and all the pre-days and then the stuff that's to come. I know the talk that you and I are doing is 100% education-focused, and hopefully that's the direction this whole thing goes.

So let's talk about attack surface management for a minute because we've seen it come full circle with the CrowdStrike attack that— or sorry, not attack, the CrowdStrike incident that happened that caused, I mean, flights to go down, hospitals to be out, banks to be unable to transact, on and on and on. We just, we have an attack surface problem for sure in this industry, and I don't know that we'll ever perfectly solve it. But boy, it rears its ugly warts when we have major systemic events like this happen, don't you think?

I do. And, you know, I think what's of note to me here as the non-super technical security guy is that our practices have changed over the years. You know, you remember there was a day when a Windows update would come out and you'd push it out to 5 machines and test it, see what happened, and a week later push it to all the rest of your machines. In the event of cybersecurity breaches and in the world that we live in today, The methodologies have changed. The best practices have changed to patch early, patch often, get every patch out there. And this is one of those cases where if this had been pushed to 10 machines, we'd have found it. We'd have known. And it is interesting to me that I don't think that's even a capability in a lot of software platforms today. Does that stand out to you as an oversight? Have we gotten too aggressive in our urge to patch early, patch often?

It's a large degree shocking that this happened, right? When you think about how we deploy software, we think about how we do testing regimens, generally there's a rule of thumb that there's a proportional level of damage that is detectable by a certain number of endpoints that you test upon. In other words, if I have something that I throw out to a test group of machines or whatever, and I come back with no failures, I say, okay, we're probably ready to push this out everywhere. Now, no doubt you're gonna have 10, 15, 20 machines or customers isolated that have a weird one-off thing, It's not a global outage, right? Whereas typically when you have something where every single company has an outage that's running a particular piece of software, even a simple test should show it. There's a proportionality that comes with this. And so no doubt we have a lot going on politically. We have a lot going on in this country, but I'm sure CrowdStrike will be called into Congress. They will be forced to testify and we'll get more information that we just don't have. But I think that's a great question of like, how could this have happened with normal testing regimen? If every single, almost every single person running CrowdStrike was gonna have a hard outage, surely this should have been discovered in testing before it was pushed from some kind of QA environment into production.

Yeah. But I think it's also in my mind telling that my limited experience with tools like CrowdStrike is that I don't have the ability to go into the tool and say, this device lands in my beta testing group and this device holds off for 3 days or 6 days or whatever my, whatever my delay is. Like most of those tools now don't even have a ton of that baked in, and if they do, they're not talking about it. I feel like this is something that as MSPs, I'd love to hear from folks how you handle it. I'd love to hear from you, like what you think best practices ought to be. Like what, CrowdStrike aside, this could happen with any vendor. What should MSPs be doing with their patching methodologies, with their deployment of tools? What can they learn from this?

Well, part of this is pick your poison. Right? Okay, let's play this out. Before this had happened, let's say something else. Let's change the scenario from it just for sake of argument. Let's say a global attack of some kind happens, some vulnerability, and now we're going to go criticize the EDRs or any endpoint solution because they didn't push out their updates and protections fast enough. You see how quickly we can flip this coin and all of a sudden say, well, we need to be able to push out in real time as as possible to be able to fix something, right?

That's how we got how we are, where we are today.

We got what we asked for.

Yeah.

Now I know that people listening would say, yeah, but timeout, Wes, that should have been tested so this didn't happen. Fully agree, right? We're going to learn some more out of this. But so I think there's— does there need— maybe the question needs to be this: does there need to be some kind of middle ground? Do we need to identify critical systems that just simply have some window of which we say even something that is critically important, we're going to give a 12-hour window so we can see what happens and other testing areas in our organization before we push it out so we don't ground flights, so we don't have hospitals that shut down and lives at risk. Maybe this whole idea of push out immediately with no knowledge, no time to assess, can be dangerous for— now I'm not talking about just every everyday average endpoint, but critical endpoints. Maybe there needs to be some ability for all critical systems to have some kind of time before that happens.

Well, because if we know we have a live, real-time, active exploit going on, and we're patching to beat an exploit. Even critical systems, we have to consider if this thing we're patching crashes every Windows machine on the planet, it's worse than a couple of systems getting beat. And we should be able to deploy compensating controls, right? We know if we know where we're getting breached, we should be able to deploy some sort of compensating control to protect us for a brief period while we test things and make sure that we're in a better place. As MSPs though, the minute we pull point our finger at CrowdStrike and say the vendor should have fixed this, but we lose our ability to control our own outcomes. I think that's what's scary to me, is that I feel like right now MSPs need some advice and some guidance on how they can control their own outcomes. I don't know that that's out there. I don't know that that's available.

Can I ask this question? Do MSPs control their own outcomes realistically?

I think we should— probably not. Um, and I think that we should work as much as we can to understand where we don't. And wrestle back some of that control. I think this is just proof of that. And it's not realistic to be able to wrestle it back everywhere. But I think to say that we are 100% at the behest of our vendors puts MSPs in a really bad spot where now you have to go look at, are your contracts up to speed and is your insurance up to speed to cover things like this that are going to happen to every MSP eventually? If you can't control the outcome, how can you put in compensating controls, mitigating controls make sure that this doesn't take your business down with it.

Yeah, and I just keep coming back to, I'm not sure that we do control our outcomes like you said. On the Cyber Leadership Summit last week, I asked Patrick Beggs, the CSO of ConnectWise, that question of like, Patrick, it seems to me in light of not, let's not pick only on CrowdStrike. Microsoft actually had a pretty significant global outage at the same time and plenty of other companies have had and will have this happen. I just think we have way too many single points of failure in this world. And I asked Patrick that question, like, am I off my rocker? Am I just a doomsdayer or whatever, or do we have way too many single points of failure? Like, when I think of single point of failure, I think everything works, everything works, one point of failure, it's not gonna happen. The real— the reality is it's like, here, single point, here, a single point, here, a single point. We think about our national economy, and even internationally speaking, everything is so wired together. We have, I think, single points of failure we've never dreamed of yet. We've never game planned for. It's ridiculously scary.

It is. It is, and I don't know, like, we can tabletop our way into figuring out what a lot of those are. We can have some good conversations. Our favorite Brian Blakely statement of, you know, hey, Mr. Customer, show me how money moves through your business. If we do that at a larger scale and talk about how— show me how money moves through the economy, we should start to pinpoint some of those, start to point some of those out. But holy crap, I mean, if you take down Windows, What we've learned is that, you know, way too many critical systems rely on just a small number of vendors.

So resilience once again becomes the name of the game. So I think as we close this video, in my mind, what a lot of this comes back to is resilience. You're gonna have single points of failure. Some you're going to identify and plan for, it's gonna happen anyway. Some that you're never going to dream of. Some that have no fault of your own and something else happens in the supply chain further up from you or further down from you causing a backflow to you. Regardless, It's going to happen. And so I think resilience is more important than ever before. It's part of our talk that we have tomorrow is we're going to talk about cyber resilience and its importance. But boy, we can't undersell the need for spending time in analysis and resilience. And the whole idea of resilience is how do we stay operational in the midst of adverse circumstances?

I think I'll just add to that, you know, all the salespeople are being trained today to avoid using FUD, right? Right? Don't sell with fear, uncertainty, and doubt. And as the cyber game has changed over the last 5 years, I've kind of started to ask, like, shouldn't you be afraid? If you're uncertain as to whether or not there's a problem looming for you, you're probably not paying attention. None of us should be in doubt. So FUD is more about not selling with fear, but selling with the reality that technology will fail you. And every organization should have some sort of a technology down continuity plan. Like, how do we continue to run the business? How do we continue to ring the cash register? You and I talked about this briefly earlier today. How do we continue to make sure that even if we can't process payments, that we can queue up payments? Even if the business can't continue to do business as usual, how do we make sure we don't lose money? And our conversation was around like a community bank, right? Do they really have a huge cost of being down? Well, they think so, but the loan officers know each loan that's coming through. They're not going to not close a loan because of this. It might close 12 hours late. The loan officer is going to get yelled at, but he's going to talk to his customer and go, hey, that piece of business still gets done. Southwest, on the other hand, or we'll pick one that was actually down, Delta, on the other hand, lost a bunch of money, like truckloads of money. And so those kind of organizations are going to have to figure out what do they do? How do they operate when they are without core technologies? Then every business regardless of size should start to ask themselves, am I really hemorrhaging cash when I'm out, when my technology is down? And if so, what are the remediations that I can put in place to make sure that we, that we continue to operate?

I think that's great. I think we're out of time, but that should be the next thing we talk about is thinking through what does recovery look like and how do we triage and plan for who gets recovered first.

And how do we do business while we're in recovery, I think, is a —important point. Sure is.

Love to know what you guys think. Leave us some comments down below, add to the discussion, give us some, some points and ideas and thoughts that you have. We'd love to hear. Thank you guys for watching.

If you're at RoarCon, the booth is right over there. Come by and see us. Right on. Thanks, brother.