Cyber Resilience: Mastering Business Continuity Planning with Todd Wilkinson

Oh, Simplifying Cyber. Sorry, I need a Post-it note for that. Thanks for tuning in to Simplifying Cyber. I'm Aaron Pritz, and standing in for Cody Rivers today, and actually standing, sitting in his office, we have Todd Wilkinson. So Todd was on season 1 of the show, I think maybe 3 years ago.

It's been a while.

Worked at Elanco Animal Health, and actually, by the time this is public, I think we will have revealed that Todd Wilkinson has joined as a Cybersecurity Consulting Director at Reveal Risk to go help others do some amazing things like he's done prior. So welcome to the show, Todd. Welcome back to the show.

Thank you. Thank you. It's good to be here in a different room.

Absolutely. Well, today we wanted to jump right into a hot topic, and it's one that if you were to ask me 20 years ago, 'cause it was still a thing then, if it would've been a hot topic, I would've laughed all of us, including myself, out of the room. So that topic is business continuity planning and BCP, disaster recovery planning and incident response, has always had a role in cybersecurity, at least adjacent. But I think what's changed or what I've started to notice that, Todd, I want to really dive in with you and your experience, is BCP is being talked, owned, and actioned very differently than what I experienced as an auditor at a Fortune 500 company where I would write BCP findings. Affiliate doesn't have BCP, and it would be handed to the IT director of that, that location, and then they would have to figure it out. And I think that never worked. It feel— it felt misplaced. And some companies felt like BCP was a second version of a disaster recovery recovery plan, but I always saw it as I think what it's starting to kind of re-evolve back into as it's actually a business continuity plan owned by the business to operate the business if IT didn't exist. So Todd, I'm going to just pause there, kind of let you kind of give some opening comments about what you're seeing here.

Yeah, no, no. And again, thanks for the introduction and happy to be here again. Yeah, a couple things are changing that just from 10 years ago, and we used to have a lot of BCP conversations. Number one, the introduction of SaaS is really a key component of driving many companies, and that includes manufacturing facilities as well. A lot of manufacturing facilities used to be— have everything on-prem. More and more are moving them to the cloud. So SaaS is changing that landscape. Certainly disaster recovery is still a key piece of that and how IT executes that. But the business driving BCP, I find, is a much more effective conversation for companies to really think through what it means, because it might be It's okay to say, hey, let the system be down for a week or two. We're okay with that. And we've got other ways we can navigate the problems that are out there.

That's when the really cool office parties happen. Like, systems are down. Let's have a good afternoon.

Yeah. Yeah.

It doesn't always work that way, but that's how I like to envision it sometimes.

Everybody go home. We're just, we're down for the afternoon. So that's one of the changes that is occurring. And two, the reliance on IT system is much, much, much bigger than it was a decade ago. So the context to how we think about it needs to start changing.

Yeah. The other thing, we do a lot of work in healthcare and pharmaceutical industry and payer and health insurance. Obviously I spent a lot of time on the corporate side in pharmaceuticals, so I have a lot of passion for this industry. But with the recent incidents, I mean, I would say in the last 5 years, but specifically with Change Health, I think that really rocked the healthcare industry in that so many companies like were completely not able to operate because they couldn't do payroll. And I think I heard the stats that like a third of all US healthcare companies relied on that one company and the one system that they operate to pay between physicians and distribute products from pharmacies and things like that. If you think about that, and Todd, your point of like a business relying on a system, obviously moving money and controlled by IT and overnight jobs and all that stuff, that is critical. And if that's not there, I don't think a lot of entities were even thinking about like a payment system or, or whatnot. So what, What have you seen in that space? And I guess within kind of having been in life sciences and human health, what, what do you see from that perspective as like kind of the, what's changing about this?

Yeah. Well, let me introduce a personal story just in the Change Healthcare.

Love those.

This, this was, this all happened within a 48-hour window for me. I'd kind of heard about Change Healthcare, what was happening with that, but I hadn't, quite honestly, I hadn't really been paying attention to the details of the impact. I'm standing in line in my pharmacy. And there were 3 little old ladies in front of me and I watched them one by one walk up, try to get their medication. They were told, "Hey, this is really expensive. Your insurance isn't available right now. You either have to pay cash," and it was a very large number for all of them, "or you just have to not get this right now." And I watched all 3 of them essentially turn around and say, "Well, I guess I won't get it right now," and disappear. And that was impactful, just kind of watching that. And I'll come back to that point. And then my eye doctor, I was in there, I just happened to ask the question, is this impacting you? And he's like, yep, 70% of my revenue has been gone for the last 6 weeks, a full 70%. That's a hugely impactful number for a small business to just have disappeared. So those are the two things I witnessed just on the ground. I think to your question, one of the things that I think is key to a BCP plan that can help with some of these scenarios, because what is different, a SaaS system is going to go down. It's done. It's down. Your IT teams might be able to help get that back online. But a key piece of that is just knowing, hey, when this is down, what am I going to tell people? What is my actual instructions of what they're supposed to do? Should I tell them to wait? Should I tell them to call a different number? Having thought out that portion of the conversation can really help quell a lot of concerns or just confusion in the marketplace. And then also your own teams internally can know this is what we're telling people. This is what I expect to do. And having thought that comms plan through ahead of time, I think can be huge. And if nothing else, and just in case of change healthcare, hey, I might need to just not go to the pharmacy for a few days or wait a week and try again later. Just something along that lines. But that's not normal, I think, historically as part of BCP plans, because they're so focused on internal communications. Who do I call? Where's my information stored at? How long is it going to take to recover? And so there's that connection to the public, I think. That's impactful.

Yep. So on the, you're kind of referring to things that could be in the plan. What's your perspective on separating business continuity plan from disaster recovery? And then I kind of also want to maybe edge in the question of the shift of ownership from IT. So kind of what have you seen, what are you seeing right now, and where do you see plans and how they're structured and owned going forward?

Yeah, well, and I've sat through a number and it's sometimes DR and BCP plans get thrown together. And I think functionally, that DR plan is how do I rebuild a system? What's that tactical plan to bring things back online and what that actual functional plan is? Like, my stuff's gone, I know how to rebuild it. And where is it at? That, I believe, really should stay within the realm of IT or information security, depending who runs that. I think that's a clear line of ownership. The BCP plan is more function. How do I just keep things running in the meantime? Back to that communication strategy. Do I need a phone? Do I go to paper processes? That's the con— how do I keep people safe and healthy? Almost a personnel plan and a commercial plan versus the behind-the-scenes nuts and bolts with what you have in a DR, how to actually physically restore a system or bring operations back up and running.

Yeah, great point. Todd, you've spent a lot of time in, you know, life sciences manufacturing. I think for me, business continuity planning is an easier concept for manufacturing because availability and uptime is everything. So they naturally get like, we need a plan for when we don't, we can't do that and we still need to get product out. How is that different in some of the other business functions that maybe have less of that key driver? How have you seen change or how do you get them as motivated as maybe some of the organic interest from manufacturing?

Yeah, yeah. Manufacturing kind of has an easier story and a harder story at the same time there, right? You get your commercial, let's take an example of a product I'm selling to customers. That one's a clear line in the sand. I think you can make a decision of, Hey, I can have a fast-acting BCP plan to keep the business running, but it's going to take this amount of dollars to invest. And it becomes an easier conversation if you know I've got to invest this much money or this many people resources to have a fast-acting BCP plan. Or it might be okay, and this is okay in some cases, hey, we're going to be down for a week or two. My building is gone, it caught fire, the roof has fallen, and whatever it might be, the data center's gone down, or my key SaaS provider has just gone out of business and I don't know when they're coming online. Online. It might be okay to say, hey, we know what's going to happen. We know how to communicate to these people. Let's just go ahead and tell people this service is out for a week. A lot of people might be okay with that if you tell them it's probably a week versus I'm going to have completely redundant systems across the board, practice to the nth degree. On the flip side of that, it might be, no, we really can't be down. I need to have a backup plan, and I might need to be prepared to bring in a second SaaS provider or somebody alternative, I can kind of get things going, at least in a basic sense. It's that conversation up front of going, when it really gets down to it, how long is okay? Right.

Yeah. The kind of the business impact analysis or assessment to kind of understand priority and all that classic concept, but really making sure that that pulls through and how you're implementing the plans. You mentioned software as a service or SaaS. How does kind of data and the user interface pushing out to SaaS and less being on-prem and the introduction of artificial intelligence kind of recomplicating that already complicated situation. How does, how does the tech landscape shift affect BCP and how you might approach it?

I think there's probably kind of two angles to that if you throw AI into it a little bit. One, just, just the generic SaaS provider. I'm not sure if I say more and more are behaving this way, but certainly we've seen SaaS providers, hey, we're down for the day or we're down for an hour or we're just down. I don't I don't know when we're going to be back up. It might be 5 minutes. It might be 3 days or longer. I think Change Healthcare was a good example of something that was way longer than anybody anticipated that had huge ramifications across the country. But you're going to see, I think on that side of the house, it's just how long is long. On the AI side of the house, what I am seeing is the rate of pace of change that they're delivering. They're throwing new models out, they're putting updated models out there. And what you find is I may not shut down my business, but all of a sudden those AI models, you don't necessarily have the time to do full regression testing, especially if you're consuming a service. And all of a sudden your answers may be different than what they were last week. And you may have to temporarily shut yourself down going, hey, I need to take this portion of the service offline. Or it may have done it for us because the interface changed. So anticipating, I think, more often but maybe shorter disruptions in business is probably the right way to think about that, just given how quickly things are changing. And again, back to, hey, we need to tell people we're down for a little bit. And if you can communicate to them quickly and effectively, what I found, most people are like, yeah, okay, I can live with that for a while. I'll come back later. It's the, I don't know that it's down or nobody's telling me it's down. I'm just going to get frustrated and keep trying.

Great point. Let's talk about the different levels of business continuity planning, and I'll go back to another audit story. Again, I did a lot of vendor audits when I was in corporate audit. Kind of interesting because you got to go outside of the walls of your own company and, and look at others. But when you think about business continuity planning and when you ask a company or a department, do you have a BCP? A lot of times it's yes, and here it is. And sometimes it's like, here's our master plan, and then here's functional area level plans. Give me your thoughts on, I think there's the enterprise BCP and then functional area business continuity plans. In your experience, Todd, have you seen that full ecosystem of top level and then local level, or do most stop at the top level and then the local areas really never institute their actionable plans that they can use in a crisis?

Yeah, well, I guess the short answer is I've seen 'em all across the gamut at multiple different companies. Usually those high level ones are just that they got a basic comms plan who's in charge of putting together the comp plans and that's the end of it. And then you get to those functional areas or facility plans. But my favorite one that is in there, it says, here's who you call and then also call IT to bring everything back up. And I think that's the one that's challenging because if we're really calling for a BCP plan and we're gonna have to execute, a building's caught on fire, the network's out, tornadoes rip through your area, there's a flood, IT's busy.

Assume they're down. Yeah.

Yeah. Assume they're gone for a while. So figure out something else. That's sometimes a helpful question to ask when you're trying to figure out those plans.

Yeah. No, I recently, I've helped some organizations build them actually in healthcare as well. And one interesting thing is people always assume tribal knowledge of like, we would know what, it's kind of like that false confidence. Like we would know what to do if it was down. We've got a lot of expertise. We'd figure it out. But then for the organizations that have gotten into planning, you start to whiteboard their business process. And a couple observations I've had is sometimes they can't draw it on a board. And if you can't draw it on a board in a non-crisis, you're not going to be able to articulate it or draw it on whatever you have in the crisis to even make sure you've got the right people on the right bases. And then I think the other thing is kind of when we get into testing the BCP, that's a lot of times like people assume, well, like, oh, we'll use that spreadsheet. Well, SharePoint's down. Did you have a backup somewhere? So I think really thinking like Thinking through your process, defining who's supposed to do what, when, without IT, and then testing it to make sure that all of those assumptions that you thought would work until you didn't realize the actual impact would destroy your plan. Those are the kind of the big ahas that I think we see a lot of companies having as they progress on this.

Yeah. Well, I would say the other one, you may have somebody that could whiteboard that, but when a disaster hits, the person that had the plan is also on vacation.

Yep, absolutely.

We hear those stories that the one person that had it, he's just unavailable.

Yep.

That's common as well.

If you didn't write it down, it doesn't exist.

If you didn't write it down. So I am a fan. Not everybody needs to be fully trained up, but at least a couple people that have an awareness of where things are at, where to get them. Do I have a backup copy of your information as well? Because to your point, if it's a big disaster, you might have trouble getting to where the information is at, or it's in the vault and it's in the bottom of rubble, or it floated away, whatever it might be. Weather patterns are changing. We have a lot of facilities, hey, the roofs are collapsing in, or there's, you know, those fires. I remember years and years ago, we had a key network connection that was down, and I'm on the phone trying to get the telco provider to bring this thing back online and how to test it. And like, Todd, really, we have a fire here. We can't deal with you right now. I'm like, no, I understand you're busy. And this went on for a good back and forth for, I think, 90 seconds. I was not paying attention. Like, Todd, you don't understand the actual Telco provider is burning. I can send you a picture, but it's not coming back up on any, anytime soon. Okay. I need to have a different plan. This one is clearly not working. Yeah.

Awesome. Um, let's talk about the question of maybe for the organizations that don't have robust or any business continuity plans, where do you start? What kind of opposition are, are, are kind of new leaders that are going down this path up against? How would you coach a newbie, if you will, to have some success given we've all kind of had our fits and starts with these types of initiatives?

So I think take disaster recovery out of the conversation for a second and separate those two. I'll go back to where I started in the beginning. I think if you're going to guide somebody, there's that context of assume things are broken for a week. I think that's a good lens to look at it. What are you going to tell the public? What are you going to tell your employees? What are you going to tell the customers that use your services? And thinking through it from that lens. And if you can get comfortable with starting there, going, hey, am I okay telling people that this is going to be down for a week? Okay, I can get there. Or maybe it's what, what's that next step? Who are they going to call? Who are they going to ask for help? Or what should they expect afterwards? I think that's the context and the lens that people can think through in a more practical sense versus I need to come up with a BCP plan that can handle a tornado. And a fire or a system outage, because now you're trying to think of all the different scenarios, but start with the real connections that matter. You've got to look somebody in the eye at the end of the day, or when I'm sitting in the pharmacy, what is the little old lady trying to get her medicine? What should she plan for? I think that's a much more personal way to think about it rather than some vague system is down or some company that we don't know about yet is having problems.

Yep. And then assuming a leader or a company gets that first kind of high-level step done, how do you push past that kind of enterprise level and get into the getting the commitment of local senior leaders that maybe own a function to really think about that organization? And my guess is it's not have IT do it for them.

Yeah.

But how do you— how have you done that when you were driving some of these efforts?

I had a peer and we kind of tag-teamed this conversation and the one going, hey, Don't call IT, we're busy. I'm not even sure I can participate in your BCP. Include that as part of your thought process. Really help to put that ownership back on the other side. It's not to avoid the work, but it's to help them think through, well, this is really my story. This is my challenge to think through if some service or some technical function isn't going to be able to help us, at least not for a few days. Not like they won't come help. Again, that makes it personal, and I think it makes it a little bit more approachable.

BCP, make it your story. That's a good tagline because I think that you're really speaking to ownership and the fact that they're going to be the one holding the bag when they don't have IT, their boss or their customers are asking them for answers. It really— it's an organizational culture and ownership issue before it is anything to do with IT and who's, who's coordinating what.

Yeah. And the manufacturing side of the house, they in some ways have an easier story. They can spend a lot of money and time on redundancy and backup plans. But there's also that pressure to not overproduce, but it might be far cheaper to go, I'm, I'm going to build an extra week of buffer into my inventory. That becomes part of the, part of the financial plan rather than I'm going to have a backup plan or a backup facility and, and, or just hope that things don't go down and we can bring things up within 24 hours. That's where you start tying those. My BCP story becomes part of the financial story and I've done the cost benefit analysis. Analysis.

All right, well, I'm going to start a new part of the show that's called Unexpected Unprepared Question. And your question, Todd, and, you know, producer doesn't even know we're doing this. Todd, you are a renowned storyteller. You really know how to tell a story. So what is your best BCP or disaster story that you've experienced? You don't have to name names. You don't have to name companies. But do you have any exciting, like, squirrel exploded from a data center power storyline or anything juicy like that?

Well, yeah, I have a couple. I'll share this one. So years and years ago, building a data center, this is several careers ago, but I had spent, I think, 9 months with the telco providers trying to get redundant lines into this facility, making sure that those lines didn't run down the same railroad tracks on the same side, down the same road, truly redundant lines going in this facility. And it took a lot of work just to get them to admit where these physical connections were at. So finally, we're at the end of this project. The lines come in, they're coming from different regions, but I take a day off. I think I was on vacation. I come back on provisioning day and they had pinned up both of the lines on the same telephone line or the telephone pole coming into the building off the road. And I'm standing out there, you can't have these here. You've got to move this. The construction folks are mad at me. Everybody's upset. "You got to move it. We're not done. It's got to move. It's got to come over there." And I walk away. An hour later, an hour later, a big semi-truck hauling a load gets into an accident and runs over that single telephone pole. And I don't know if fate was there that day. I'm not really sure, but I still remember that as going, "Yep, that's why we talked about that." Yeah, I told you to stop. Yeah, I didn't pay him. I didn't pay him.

Yeah. That's awesome. Well, it's, it's good when an actionable story comes that quickly. If you've given, given direction and it's like, no one believes that this is something that we need to know, but you know, like, uh, the fact that I've been hit by two buses, one on an audit trip with a pedestrian bus and the second time on my honeymoon rear-ended in a taxi, no damage on both. But as an auditor, you know, you'd say always document everything. You never know when you're going to get hit by a bus. And take it from me, I've been hit by two.

So you shouldn't be laughing at that. I don't know.

I can laugh now. It was scary at the time, but, you know, awesome. Todd, anything else you'd like to leave as parting comments to the audience on BCP?

Well, at some point, practice this one way or the other, whether it's a tabletop exercise or get people offsite, give it some dedicated time occasionally. It doesn't need to happen all the time, But it is something that's worthwhile to practice, whether it's tabletop exercise or actually try to activate these things. And it does take time out of your day of other more critical projects, but it's a good learning exercise. You always find things, always find things that you missed as part of that plan, guaranteed. And for your teams and your staff, it's actually a good training exercise as well because they're going to learn things that they haven't done before. They're going to get more comfortable making changes in your environment. So that's So that's my parting message. At some point, practice this.

Well, and Todd, thanks for coming on the show. And actually, thanks for coming to Reveal Risk. And for any of our audience that has a BCP journey in front of them, Todd is a great person to reach out to. He will not only bring stories to the project team, but storytelling to the business continuity exercises themselves. So thanks again, Todd, and appreciate everyone for joining in.

Yeah, thank you.