CISO Tested, Board Approved: Cloud Resilience with General Mills CISO Noah Korba

Hi there.

Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as well, for today anyway, are myself, Tim Peacock, the Group PM on Google SecOps responsible for agents and graphs, and of course, Alisha Cade, new addition to the show, Senior Director in Google Cloud's Office of the CISO. You can find this podcast wherever you get your podcasts, as well as at our website, youtube.com/@CloudSecPodcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of all our Cloud Security Podcast listeners on LinkedIn, the Google Cloud Security Community site, and of course, on the YouTube episodes themselves. Alisha, we have a fun guest today. We got to talk about our favorite cereals. Is that right?

I guess so, as long as they are gluten-free. But, Tim, thank you so much for having me on the podcast.

No, thank you so much for joining me with this special guest who you had much more background with than I did. And so, listeners, this is a fun episode where we're talking about— well, what do we talk about, Elisha? Is it, like, resilience? Is it cloud? What did we do? I feel like this is a very back-to-our-roots episode, actually.

I think it's resilience in the cloud and resilience of the overall business. This isn't tech only. This is peeling out the bins, the cereal, all across the environment and just making sure it's working when it needs to work. And I mean, I used to be a CISO in financial services, and I know resilience in critical sectors is on the top of everyone's mind, but it is really on the top of every business's mind. And it was fascinating talking to NOAH about how they approach it, because on one hand, Mm-hmm. They can be, I think, a bit more creative than the regulated sectors. But on the other hand, of course, they need to sustain their operations.

That's right. And if they don't sustain their operations, listeners, I won't have my Cinnamon Toast Crunch, which is a big issue. And so without any further ado, let's go find out how they make sure my Cinnamon Toast Crunch is always on the shelves. And with that, listeners, I'm delighted to introduce today's guest, Today, we are finally joined by Noah Korba, VP of Digital Core, Cybersecurity, and Enterprise Architecture, clearly a man who gets a lot of sleep at night, at General Mills. Noah, thank you for joining us today. I wanna start off by talking about the, what did you call it, the collab recovery that you do with GCP? We were talking about this just before we started recording, listeners, and the contrast, I think, is great. So Noah, tell us all about it, 'cause it's fascinating.

Yeah, so we call it Mills Collaborative Recovery because at General Mills, we are very much a company of brands. So everything we also do tends to get a brand associated with it. And we started doing this about 3 years ago when we were doing the normal tabletop exercises like everybody does. You know, we'd get people in a room for a few hours. talk about what we would do. And, you know, we started talking about what would happen if we actually did it. So we didn't just sit around a table and talk about it, but we actually, as our engineers would say, we're going to click and clack on the keyboards and go through the process of recovering our infrastructure. And we started doing this with just our— and continue to do most of this with just our GCP infrastructure. I need to clarify, when I say just our GCP infrastructure, what I actually mean is about 90% of all of our workloads are up in GCP.

No way.

And so when we, when, when we say we're doing a recovery of what we have in GCP, that's basically a full recovery.

Just 90% of our estate. Yeah.

Yeah, you know, so modest. Yeah, very small, very simple. And so this is something that we had— we really see, and I'm— I am not just saying this because of present company, but this is something we really see as something we could not have done prior to being in the cloud. You know, we were joking before we came on here, like, if we had tried to do this when we were, you know, 100% on-prem, We would be talking about wheeling in large server racks to build enough temporary capacity to test out our backup and recovery capabilities. But in GCP, it's actually very straightforward. You know, we partner with you guys on this. We've done this for, like I said, the last 3 exercises that we've done. We get temporary capacity expansion on what we have. We treat it as something that is new. It is separate. I would say the biggest the biggest concern that we have is that we'll accidentally recover something in a way that will impact production. So like our number one rule is don't disrupt production.

Of course.

And so we have lots of controls in place to keep that from happening. But we then take our teams through a cyber exercise about, you know, build up the scenarios to why we, why we find ourselves in this position, the decisions that we made and so forth. And then we say, Now go recover. And they spend the better part of 2 weeks working on that recovery in a phased approach, just like we would if it was real. We prioritize the core infrastructure first, then we build out kind of our, our core systems that are required to keep our business running. And then we, if there's time, we might do a couple other kind of less critical workloads on top of that. And then we do a debrief and we talk about what worked well, what didn't work well, what was surprising. And then we have a whole year to fix those things before we try to do it again.

So on the topic of surprises, what has been surprising? What are some like great surprise stories?

Yeah, I think, you know, we, we started our cloud journey mid-2020. That's when we first started looking at—

Sorry, you started cloud in 2020 and now you're 90% in cloud?

Yeah, yeah, we had and we have been for quite a while, actually. We did not get stuck in the middle like I think a lot of companies do, primarily because of our leadership that we just said, this is what we're going to do. And we see that there's a lot of benefits to doing this and we're just going to go. And we rallied the whole digital and technology organization behind it and we did it. And so, you know, we started in earnest, I think in about 2021, we had a lot of planning and things that were going on. And one of the things that we— one of our core principles in how we built our cloud was everything is infrastructure as code. And that's a great core principle. It allows for very fast recovery, makes maintenance in our cloud workloads that much easier. And what I think one of our first surprises were, again, when we did this in year 1, was some of the ways that we built that infrastructure as code didn't work very well when we wanted to go and actually recover it and have it rebuild what we thought was there. And so there were some things that were hardcoded in there that shouldn't have been. There were some assumptions that we made at the time. I mean, this was— it's no fault of anyone. This is just as you're going into something brand new like this for the first time, you're going to kind of make some mistakes.

Of course.

But the great thing about this exercise is we got to, we got to show shine a light on where those things were and gave ourselves a really long runway to go fix them. Like I said, we do this once a year so that we weren't under this pressure that we might have found ourselves otherwise if we were not doing this on our own accord.

Right.

But instead, we could give those teams the time to do what they needed to do to make those better. And by year 2, it worked way, way better. It was much more smoother.

And you gave them a whole year to figure out where it slots into their schedule, and you didn't disrupt things and burn people out about it. That's great.

Yep. You got it.

So, it all sounds pretty smooth, right? But, um, I remember, if I like, it's almost like a trip back to the future, um, you know, when, uh, cyber resiliency became a hot topic. When was it? In 2017, I think.

Probably.

I remember, yep, cyber was straight away the word that signified that this is CISO's problem, but actually you had to take the whole whole business on that journey with you. And there were always, you know, some kind of resistors, whether it was for cloud resistors or resiliency resistors. Did you have those challenges in your organization when you were embarking on this journey? And what was the kind of superpower that you needed to deploy to overcome that?

Yeah, I think 2017 is a great year to pick of all of the years to pick because of 2 pretty major events that happened that summer, which we know WannaCry and NotPetya. The reason why that's important, especially for a manufacturer like us, is for a long time we looked at cybersecurity like most companies did, which was this is a, this is a data protection issue, right? We saw companies get compromised year over year. A lot of it was like credit card data or PHI or other things like that. And so putting on my Noah hat from 2017, like those were the things that everybody was talking about. It was like, well, are you encrypting your data at rest? Are you doing it in transit? Are you protecting it? Is it segmented? It was all this very data-centered approach. And then when you get to not WannaCry, but especially NotPetya, where it was really just about disruption, that's when at least I think all of us in manufacturing and certainly any in critical infrastructure kind of looked at it and said, actually, this— the game is not just about data. It is about, are our systems available? Because if systems are not available, we are in trouble.

Hold on, Noah. Are you telling me that the CIA triad is about more than just C? It's actually about A.

It's bananas when you think about it. I know.

Those other 2 letters are about something? I'm shocked.

Yep. And so, you know, and I, I think, you know, the other thing that happened in 2017 was ransomware went from a nuisance to an organized effort, right? You know, it was not that ransomware didn't exist prior to 2017. It just was focused on like your grandparents' computer. And like, you know, Yep. I don't know if you guys remember that. I mean, I still do IT support for my family.

Same.

But it was when it moved into that like coordinated, massive disruptive attack, which continues today, is when a number of companies, including ourselves, looked at it and said, we have to look at this differently. And that's about the time we had already kind of been in our journey to start building out our cybersecurity program. This is kind of when we put it into overdrive. And the key things for us— this is going to be a little less technical, but a little more business-focused. But I would say one of the key things for us was getting buy-in from our our senior leadership team.

Absolutely.

Because they could clear roadblocks that we couldn't clear on our own. They were huge supporters of what we were trying to do because they could see the impact that this could have on us if we didn't do this right. And that leadership continues to be in full support of what we do. And they're a team that I continue to report up to on progress and, you know, changes and ways that we're pivoting our program and so forth. Because they're very well aware of how to manage this risk. And like I said, they're very vested in this to make sure we're doing it as best as we can.

So on the topic of ransomware and availability, one of the favorite things I've learned on the show, and I've learned a lot from 280 episodes of this, is from Phil. Phil Venables brought up this minimal viable business concept. And so when you were working with those business partners, how did you, you know, work through that process of figuring out what are the absolute must-do things? And I assume there's some process by which you arrived at how the Cinnamon Toast Crunch will always be on the shelf for me, right? Like, that's the most important thing is my Cinnamon Toast Crunch, right?

Yeah, yeah.

How did, how did that actually work?

We're here for you.

Yep.

How did you actually work through that and what, what did that look like?

Yeah, yeah. Well, we refer to it as our wave documentation or our wave approach. So what we did is we segmented off systems based off of their criticality to making our business operate. For a company like General Mills, and again, for most manufacturers, we would look at our supply chain as being really critical in terms of our operations. Right? If we can't make products, we're not going to be a manufacturer. Like, you have to manufacture to be a manufacturer. As it turns out. Right? And so we look at it from kind of that lens, and we work with our supply chain leaders, we work with some of our other business leaders to make sure that we prioritize those systems in terms of importance and to understand the different components it takes in order to go from raw materials going into a plant to a box of Cinnamon Toast Crunch on a truck heading towards a warehouse and making sure that we have prioritized those systems to get those things back up and running first.

Cool. But, you know, I won't say who once said it to me and when they said it. It was before 2017. They said, if we are doing backups, then we must be fine. But of course, you noted that resilience is absolutely beyond testing backups, right?

That's right.

But it's actually much more complicated. So, what are the key elements of stitching it all together quickly, the components? You mentioned supply chain, right? You mentioned other elements, the business support as well. But what actually has to happen as part of the recovery process that you cannot test. And there is, of course, a lot of human factor as well. So what's that data recovery from immutable storage and other processes that really matter?

Yeah, yeah. Well, I think, you know, CIA applies to your backup infrastructure just as it applies to your larger cybersecurity program, right? Like, you want to make sure that your backup infrastructure is resilient to attack. And you also want to make sure adversaries couldn't get into your backups and change them in a way that renders them useless. And then finally, you want to make sure that your backups are available. Like, you want to make sure you want to be able to get to them. You know, backups that exist in a place that are unreachable aren't really backups at that point. They're just kind of like things to look at, I guess, and observe. I would say the other piece of that puzzle though is, and this is kind of how we look at it, is as we divide up those wave systems, you have to know the data that is required for each of those systems in order to bring them back from the dead, right?

Mm-hmm.

And if you don't, if you've missed pieces of that, and it's not, I mean, this goes beyond like if you were, if you were just looking at your ERP, for instance, and you said, well, all of that data sits in this database and we back this database up, you know, to our immutable backup provider, everything is great. You might forget the fact that that ERP is probably authenticating against something that's not the ERP.

Yeah.

You might forget that that ERP sits in a network that was specifically configured and isolated for its own purpose and can't operate you just plunking it back in on a wide open network and plunking some data in it, hoping that it's going to work. And all of the connectors and integrations that exist between that ERP and all of the systems that are downstream that it supplies information to, or systems upstream that are flowing information back to it. So you really need to have an architecture and a data flow to be certain that you've captured all of the pieces in order to bring that thing back. And that's where one of these exercises has really come in handy is our goal is to bring our ERP back online and it's online to the point that it's usable. And so, you know, first times you do this again, like you realize we missed a few things, right? You correct those mistakes, you make sure that they're flowing into backup and that you have the information there that you can pull back. So when you get to that point of trying to restore your environment, you can have a higher degree of confidence that you're going to be able to restore it. And you're not going to have to rebuild something from scratch because you forgot about it and now it's gone.

So, when you're doing these exercises and you're setting expectations with senior leadership, how do you get them to wrap their heads around this? Because, you know, we had Arvind Bansal, who's the CISO at a CNS wholesale grocery distributor, and, you know, he made the point that grocery stores have days of product on the shelf.

Yeah.

Not weeks. And so when you're talking about recovery time and get the plants up and running and get my Cinnamon Toast Crunch there, how do you set the right expectations with senior leadership? And then how do you conversely— and there's probably 2 questions, we'll ask them at the same time— how do you set the right expectations with your team that people will be hungry if they don't get this on the right timelines? So how do you, how do you timeline that correctly?

Yeah, so I think that's where prioritizing our manufacturing systems Early in the queue comes in handy. I think getting our senior leadership on board, one, with the ex— well, getting them on board with the exercise actually was very easy because we kind of laid it out to them as, how can you— how can we have confidence we would know what to do unless we actually try to do it? You know?

Yeah.

And try to do it in a scenario where we don't have all of those pressures on top of us of when are these coming back online, the trucks are empty, you know, the warehouses is draining, you know, like all of these, all these pieces that would start to happen in a major disruptive scenario. This gives us those data points to say, well, you know, we, we've done this a few times now. We feel confident in setting these as the timelines that we're going to shoot for. And then that can be used by our business to help them plan better. It can be used in our own planning exercises. And I think that the huge benefit on all of this is that it's driving awareness of what actually happens in these types of disruptions. We bring our senior leadership team along. We tell them about what happens. We tell them about the timelines because setting some of those expectations upfront is really important for them to know. Like, this is not your home computer that you could take to the Geek Squad and bring it back, um, you know, a few hours later and it's going to be fine. This is a very complicated infrastructure to restore, and we're not talking about a few hours here.

On that front, uh, I know we've been talking about this 90% GCP thing, but I'm, uh, genuinely curious, and this just occurred to me, do you also do this for like, uh, OT in the plants? Like, surely if a grain mill goes down, it takes days or weeks to restart that. So what does that look like?

Yeah, first time we did that was 2 exercises ago. The last exercise we did was a few months ago. And that was, again, that was really helpful for our plants to see what happens in these types of exercises, but then also really helpful for our team members to understand those nuances between IT and OT. You know, there's some OT systems that you can't program unless you have a physical key and you turn it to program mode, and then you can upload the code, and then you turn it back to run mode and it runs. And this is all by design, right? This is—

Sure, sure.

To make sure that the line is safe for the operators, to make sure that the food is coming out in the way that we expect it to. And, you know, when, when you put things in that perspective, it's not we're just going to have a download from a backup flow down here and then everything's going to come back online.

You're not just going to SCP to the grain elevator, right?

Yeah, it's like you're going to actually have to have somebody there to aid in that recovery and who understands how these systems go back together and who can test these things out. And so it's, it's an added layer of complexity, um, on, on top of all of this that also then helps as we talk about new defenses that we want to introduce. This allows us to kind of pin it back to things like this and say, you know, we know that this is the scenario. If we find ourselves in recovery, we're going to maybe prioritize a few extra defenses in this space, knowing that, to try to reduce the likelihood that in a full restoration, we would also have to go after these devices, right?

Mm-hmm.

Maybe there's things that we can do to further isolate them that would make them less of a target, less of a victim, and therefore, we can spend our time focused on systems where it is much simpler to bring them back.

Can I ask you a question that's out there for debate amongst the CISOs? To exercise with the board or not to exercise with the board? And we have seen that in our Cyber Savvy Boardroom sister podcast discussions and also through board surveys. And I think there are divided opinions. Where do you stand on that?

I think it is important to make sure your board knows what you will tell them and their involvement. But I don't— my opinion is to build out enough time to take them through an entire exercise. I'm not there in terms of it being a significant return on their time. But I do think we spend a lot of time talking about these types of exercises with them to help them understand some of the things that would happen without turning this into a dedicated, you know, half-day exercise with them because they have a lot of other important things to focus on, is my opinion, I will say.

Fair enough.

Fair enough. All right, Noah, I hate to do this, but I have to ask our traditional closing questions.

Sure.

One, do you have recommended reading for our audience? It could be anything except Anton's blog. And I guess since Alicia is our guest host today, Alicia, do you have a blog?

I don't, but I have Cyber Savvy Boredom podcast.

Okay, you can recommend that. I have, yes. We'll allow you to recommend Alicia's podcast. And then 2, do you have a practical tip to help people maybe get started with doing something like your collaborative recovery?

Well, I'm gonna give— I'm gonna tell you my favorite book, which has nothing to do with computers.

That's great.

But it is called Alas Babylon. It is by an author named Pat Frank. It is like 1960s, like Atomic Age, post-apocalyptic kind of like story about a guy who's trying to survive after, you know, the, the big, the big event that everybody thought was happening, um, back in the '60s. Um, it is— I would say it's dated material, so like your mileage might vary in some of the language that's used. Um, but I, I, I am someone who loves problem solving, and this book, a lot of it is about problem solving. And so— Oh, that's fun. I absolutely love it. It's the same reason why I love The Martian and Project Hail Mary and others like that. You know, it's like, I like the, I like the engineering mindset of like taking on a problem, breaking it into pieces, and then trying to solve it.

As soon as you said, uh, problem solving, I was like, oh, I bet he loves The Martian and Project Hail Mary.

Oh yeah, yeah, absolutely love them.

Um, if you like those, you might also— and this is not one of his better books, but if you like that particular strain— have you read Seveneves by Neal Stephenson?

No, I have not.

Okay. So he's the guy who did Snow Crash and Diamond Age, which I think are his best works.

And Cryptonomicon.

And Cryptonomicon, which is an awesome book, listeners, if you enjoy this stuff. But Seveneves has a lot of problem-solving sort of stuff baked into it. So that's kind of—

Awesome. I will put that on my list.

Yeah.

Awesome. I'll put it on my list.

Amazing. Well, Noah, thank you so much for joining us today.

Yeah, this is great. I appreciate the time, everyone.

And I have to say, cheerio.

The puns are my job.

There's always got to be one, huh?

Security.