EP204 Beyond PCAST: Phil Venables on the Future of Resilience and Leading Indicators

Security warning.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us for the final episode of our 2024 season. Your hosts here are myself, as ever, Tim Peacock, the Senior PM for Risk Analytics in Google SecOps, and Anton Chuvakin, a reformed analyst, senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on our community page. Anton, we have an awesome guest today. Who do we have?

We have Phil Venables, of course.

No way.

Again?

Yes, again. We did give him a badge after he's been a guest for 7 times, it's 7 episodes. We did not give him anything on the 8th or 9th. Okay, this is the 9th.

We're glad to have him.

But he's gonna get a new badge on the 10th.

Okay, that's fair.

Which is next year, presumably.

That's fair, presumably.

Very fun episode, Resilience.

Resilience.

And it's so easy to devolve into abstract rambling when this topic comes up. I've seen a lot of talks and read a lot of stuff on this, and it's sometimes very abstract. Phil makes it very practical. You can use it. Make your life better with it. Like, it's super good.

He defined for me what was a pretty mind-melting concept, that minimum viable product delivery outcome. That, that to me was very interesting. I hadn't thought about a world where like a company is really trying to work backwards into that decision tree.

Yep.

But that's so cool.

And you may think that it's sort of natural for a company building some digital platform, but if a company sells ice cream or does something very practical, very analog, very not even 1970s, very 1870s, then there is still likely minimal viable product delivery of that sort.

So this is— And there will always be a— what do we call it— a cyber-physical nexus?

Cyber-physical modularity. But that we have—

Well, that's kind of the opposite. I mean, everything comes together now. There's a convergence.

Yes.

Yes.

That one, and I remember— not to do too much Anton with a big ego kind of guy, but I remember a blog post in 2010.

Wait, which Anton with a big ego?

I don't know.

This Anton.

I have no idea. So it was a blog in 2010 when I had this picture very crudely drawn with a crayon, probably in Paint, which showed that the digital world and analog world would start kind of collapsing together more. And it's interesting that we had to wait that many years, and now it's sort of accepted normal reality that almost no matter what you touch, there is a cyber component behind it. So it is scary.

It almost feels retro to say that.

It almost feels retro. That's exactly right. Yes.

Yeah, which I guess is when you know it's really true. So that would've been Microsoft Paint, of course. That's your—

That would've been Microsoft Paint. Yes. Yeah.

So, okay. Well, listeners, with that, let's turn things over to today's episode. Phil, super excited to have you on the show again. Thank you for being our most frequent flyer. Listeners, today's episode is pretty funny because we're talking about resilience and we suffered total resilience failure caused by Anton's choice of operating system. And so maybe with that context and particularly that choice of operating system being a relevant factor, let's talk about resilience in the industry. So first, What is resilience? Why are we all talking about it aside from Anton's bad choices in life? And 3, is it ransomware's fault?

Well, and also I should just point out, you know, we're not talking about a Google operating system. No, no, no, for sure we are not. So resilience, I mean, I think, you know, again, a lot of the listeners will probably pick this up, especially a lot of the CISOs that may be listening, is that, you know, we all know as security professionals that we can't assume preventative controls always work, and we therefore rely on detection and response to augment and complement preventative controls. But even then, ultimately, we all know that we can never guarantee that 100%— those combinations will be 100% effective to stop bad things happening. And that could be a security-driven outage, a natural disaster, a software error, hardware error. There's a whole set of things that go wrong. And so ultimately successful organizations plan for the bad thing to happen as well as mitigating the blast radius of what the bad thing is. They plan for the resilience of how to recover or fight through that thing happening to sustain their business. And I think again, when you look across all of the outages, disasters, breaches, ransomware events, The companies that do well either don't have an incident, but the companies that also do well—

You can succeed by getting lucky.

Yeah. The companies that also do well have an incident, and then they just either fight through it, sustain business, or they recover very quickly and are very engaged with their customers to support them through the process. And they end up looking actually, you know, people regard them better after the end of that. Companies that haven't planned for resilience and don't have any resilience, have serious problems, and are out for weeks and months.

So, the ones that have had an incident and recovered quickly, those are the ones where the tide went out and you did learn that they were in fact wearing pants.

That's right.

Yeah.

As a funny aside, we just aired an episode with somebody who had to transform his entire IT under the conditions of a breach. And one question I asked that CISO of LastPass, I mean, it's a public episode, was that, would you prefer to not have the breach for your transformation? And of course, he would never say, nah, take the breach. But there was a point about transformation projects that may have taken more if there were no pressure. So resilience kind of plays together with improving things for business in the long term. So it's kind of an interesting one that's stuck in my head now.

Well, and actually as well, a big part of resilience is not just learning from your own mistakes, but learning from events that go on in other organizations. And again, this is something we do a lot here at Google in our incident response and management process is looking at things that happen elsewhere. and the near misses that might happen elsewhere as well of ourselves and looking at what we can learn from that and just having that as a fast feedback loop constantly.

Yeah. It's one thing to learn from your own mistakes. It's better to learn from other people's.

That's right. It's a cliché, but it's also kind of the truth.

It's also true.

Yes.

It's also true.

Yeah. So to bring this more to something specific, you were involved in this work on a paper with a name. I probably wouldn't read the whole thing. Strategy for Cyber-Physical Resilience: Fortifying Our Critical Infrastructure for a Digital World, or a PCAST report.

So— Was that the original title of the paper or translated into Russian? And back to English.

No, that one is separate.

It's not even Gemini-created issue.

It is the humans. Yes. And in this case, you can't blame the bots. Yes. And this report sort of set some lofty goals and also had practical recommendations we'll get to in a moment. So how did this come to be? And it sounds like resilience is something you should have thought about before. Why is this coming to that level? Like that high?

Yeah. So this report— so this was part of a report published as part of a thing called PCAST, the President's Council of Advisors on Science and Technology. And so I co-chaired this report with Eric Horvitz, who's the chief scientific officer at Microsoft. And the genesis of the report on basically strategies for managing cyber-physical resilience was really 2 things. One, recognizing that our world of cyber and physical has come together. So every time there's a cyber event, there's physical consequences to it. You know, there's supply chain problems or power outages or medical system impacts. There's always an increasingly real-world consequence to cyber events. And also when you think about kind of physical infrastructure, whether it's operational technology, industrial control systems, all of that world was a world never designed for security. It was designed to just do its own thing inside some system. And what did we do as a world? We connected all of that and wired it up to the planet, and therefore we have lots of now cyber-physical incidents.

What could go wrong?

Exactly. And then the second, the second part of the phrase was resilience, which we just talked about, which is that what can we do and what recommendations and strategies can we take to not just deal with a new world of cyber-physical integration, but also a world that works on the assumption that despite our best efforts, we're going to try and prevent all attacks, but we're never going to be successful. Therefore, you also have to plan for resilience. And so it was really kind of taking a radically different perspective than things that have come before to say, look, we have to recognize cyber-physical will come together and we can never guarantee that security will be 100%. So we have to make sure we have resilience and the capability to fight through attacks and sustain services despite the presence of Disasters, attacks, outages, errors.

And the report, one thing that struck me when I was reading for many months, kind of every day, long story why, this report, I realized that their take on infrastructure, critical infrastructure, is so broad because literally almost everybody involved in agriculture for the purpose of this report is in the critical infrastructure business. And so you don't think about cyber agricultural attacks, but unfortunately, yeah, we do.

Well, that's a fascinating point because literally you go across those critical infrastructure sectors and you think about kind of agriculture is look at the amount of automation in agricultural supply chains, in the automation of how just like farm machinery is remotely controlled and diagnostics come back. And it's just literally everywhere. When you open your aperture of the problem to being cyber-physical, it's like everything.

That's very sci-fi almost. Like I'm getting Snowpiercer vibes.

Yeah, you get, yeah.

I'm getting Aeromancer vibes. And I'm getting vibes of those new weeding machines The lasers mounted to take out the—

anyway. Now I'm even more paranoid than I was pretty paranoid before.

We're getting back to the dark bomb or the dirty bomb episode. So one of the interesting things in the report is the importance of leading indicators. What does that mean and how do organizations shift to think about that?

Yeah. So sadly today, a lot of measurements of success in cybersecurity are often lagging indicators, breaches, vulnerabilities, malware events, ransomware events, all of these things. And they're all obviously necessary. to track, and you want to track those and try and reduce those. But it's important and even more important to focus on what are the leading indicators. So what are the things you can do and you can measure that if you do those well, you can't but help reduce what might be measured as a lagging indicator. And the genesis of this thought was from kind of thinking about how industrial manufacturing is done, to give an example. So if you're running a big high-tech production line, you measure the quality of what comes off that production line in terms of quality and what needs to be rejected and other things. But what manufacturers really do is they monitor all the leading indicators, the quality of the components that go on at the production line, whether there's been preventative maintenance, whether you've designed for manufacturability. And if they get all of that right, they can't help but have great product that comes off the production line. And so the genesis of this idea was the same thing. And so when you think about what leading indicators are in cyber, They're often actually not that much to do purely with cybersecurity. So, it could be, for example, what percentage of your software can you reproducibly build and deploy? And we all know that if you get that right, you can patch, update, improve software security because you're never fearful about pushing or rolling back changes. You can just drive that. But the great thing about a lot of these leading indicators is you don't just get the benefit of the security benefit, You get other benefits if you have all your software capable of being reproducibly and reliably built and deployed. You have lots of business agility, commercial benefits, efficiency, developer productivity. Other examples of leading indicators could be what percentage of your environment can you rebuild and cold restart. And to be able to do that, you've got to have a good inventory, you've got to have infrastructure reproducibility, you've got to have software reproduce, you've got all these things. And the great thing about that is if you can do all of that, Again, you've probably done an inventory and realized you've got too much stuff that's not being utilized and you've saved some money. You've got infrastructure reproducibility, you've got the same agility as you would have with software reproducibility. But the side effect of all of that in terms of the lagging indicators, if you have a ransomware event, you're not fearful of that because you know you can just restart your company and you're good. Yeah, sure. And therefore, you don't have problems with your lagging indicators because you've paid attention to the leading indicators and all the leading indicators have massive adjacent business benefits.

By the way, which brought me for a brief moment kind of to the depressing side of this, namely that for a while we had the lagging indicators, the ones you mentioned, but we also had the inputs, namely like controls deployed, firewalls configured. So you had a choice of lagging indicators or measuring inputs, but it felt like there was never a choice of measure leading. So it sounds like this report does break new ground in terms of steering you away from inputs and also away from lagging, but to leading.

Yeah, no, exactly. And it drives towards the correct organizational and technology design that should be in place that designs for defensibility. So again, in that manufacturing analogy, you talk about design for manufacturability. In technology, we should be designing for defensibility.

So one of the specific indicators that we covered, that was covered in the report, mentions this mysterious thing called cyber-physical modularity. And I remember spending some time scratching my head thinking, what? They mean what? And it sounds mysterious probably to many listeners as well. So, can you explain it? Sorry, I'm not showing up. Explain it, Phil.

You know, it's basically, you know, fancy government report language for segmentation. But it's a little bit more than that. I mean, it's effectively being able to say, like, so for example, you could say, I'm going to segment my entire environment and you're segmenting networks, business applications, business units, and then you have different types of firewalls, network or application level between them, and then you proceed to punch a bunch of holes in those firewalls. So you've got segments, but you've actually got no modularity because there's still dependencies across them. So what this cyber-physical modularity gets at is between your physical systems, your cyber systems, between different cyber and physical systems, are they capable of operating on their own in modular ways? at least for a period of time. So a good example, again, let's imagine that you're running a power plant and you've got a bunch of industrial control systems that regulate how that works. And those industrial control systems are connected to some cyber systems, which have— and the whole thing may have some vulnerabilities and some resiliency risk potential. What you want to be able to do is if you— your modularity should be you can cut the cyber from the physical.

Mm-hmm.

manage the physical locally with some like dials or switches or like LCD panels or whatever. Screwdriver, of course. And at least keep it running for hours, days while you recover the cyber part of it or deal with an issue in the cyber part of it. Sadly, today there's not enough cyber-physical modularity because there's plenty of examples that we found in doing this report and looking across industries where if you did have a cyber failing, the physical systems just grind to a halt, even though they shouldn't have to. You know, one of the other things that we found, another bit in the report, you know, we call it internet denial testing, which is related to cyber-physical modularity, which is, can you run at least for a period of time without being connected to the internet? And you would think like—

Most kids these days can do that.

Well, exactly. And you would think like, well, of course we should be able to do that. You mean, don't we? And we found a couple of cases in some kind of hospital systems where there were some medical devices that if you took the internet away, they would start operating differently or they would stop simply because there was some dependency built into, if you like, kind of heartbeat-backed—

Right. License check.

Yeah. License check.

No medicine for you.

Diagnostic streaming. And again, there's no reason for that. It might be that things should be connected at least, you know, once every 7 days, but not continuously. And that injects a bit of resilience. So, and you imagine that across every type of critical infrastructure, the ability to operate in a modular way, at least in some degraded state, to keep a minimum viable service running, that just effectively decouples all of your recovery, that gives you more resilience.

So this makes me imagine that every hospital should like Yank the internet cord out once in a while and see what happens.

In fact, you know, we know a lot of organizations now across many sectors are now doing, for certain critical areas, are now doing kind of internet denial testing, saying, like, what does our world look like for running without the internet for a certain area? And sometimes it's fine. Sometimes they're all surprised that things break and go, why is that going into the network? It's dead penalty.

What other kinds of testing should they do? The report talks about stress testing. Are there other kinds of stress tests we should talk about?

So I think the stress testing is you go into this example of, and this, some of your listeners may kind of know the language of operational resilience where we talk about coming up with more severe but plausible scenarios to test. So in everybody's disaster recovery and business continuity playbooks, everybody has kind of lists of things that they want to test. And then there are a bunch of things that you think, well, we probably can't recover from that, so we're not going to test it.

Oh, okay.

And the whole point of stress testing is to introduce some more aggressive resiliency tests using scenarios that you kind of know you're not necessarily going to be able to fully survive. In the example we just talked about, internet or broader communications deniability. The logic of going down this path is then to say, look, we're doing this test not to show we can pass it, but to find where the breakpoints are and to see if there's anything else we can do or any unexpected stresses. So it's a bit like the language comes from financial risk stress testing. So as you probably know, all major banks have to do these stress tests where they're given a scenario by the banking regulators to say, You know, imagine the stock market goes down 40% and interest rates go up this much and this happens and this happens. What does that look like to your ability to support customers, be liquid, and not go bankrupt? The same can apply in cyber. We could say those are scenarios that probably would not happen and not happen in combination. But you find a lot of use because you find on that path to the stress, where the potential fractures are that you want to fix that will help you in your everyday resilience. And so the notion of having stress scenarios and stress tests, I think a lot of organizations are finding quite useful to really push the boundaries of their resiliency, not because they have to be resilient to those extreme stresses, but because they get benefit along the way of finding where the stress points are.

So that to me is probably worth the price of a ticket on its own, because When they hear a scenario, they think, nah, this scenario won't happen. Why test? But it's what you learn on the path, not that specific thing. To me, that probably pays for the ticket itself.

It's like the Patton quote, plans are useless, but planning is useful.

But planning is, yes. The incident, yeah, IR planning. So I want to switch to something. So we wrote our own paper kind of to try to make some of this more practical. So I'm not trying to ask you to summarize the paper in 10 seconds, but if you look at the report and think about like, If somebody is reading this and he's a CISO of an organization likely from one of those sectors, so what are the takeaways from the report that people can actually use to improve security, resilience, combination thereof? Sort of like, what are Phil's favorite takeaways?

Yeah, I mean, the big takeaway, and this is one of the big recommendations that we had for kind of government to work with infrastructure, critical infrastructure sectors, was to identify for every sector, for every business, for every critical public service, again, healthcare, banking, telecoms, energy, the provision of basic utilities like water and other things, is to identify what the minimum viable delivery objective is and sort of basically say, we all know we would like all of our services to be running at 100% all the time. But if something terrible happens, like a massive disaster, a massive cyberattack, some unimagined kind of broad-scale attack, what do we expect of our critical infrastructure providers to be able to provide almost no matter what? So you might, for example, with water, you might be able to say, you know what, I, you know, we can go without drinking water for 72 hours because we've all stocked up on cases of water. But after 72 hours, we start to have a problem. So, you know, utilities should plan to get back within 72 hours.

Yeah.

Another example, I think we covered this in the report, was Financial services, let's say the banking system is under massive attack. The minimum viable delivery objective for the banking system is let us make payments so we can buy things. Let people cash the equivalent of Social Security checks so that they have money. Let companies make payroll and roll over reissuance of corporate cash in the bond markets so we don't have an economic collapse. You've got to do that, but you don't necessarily have to get like financing new mortgages or doing IPOs or mergers and acquisitions.

So, some kind of minimal viable business kind of like thinking.

Yeah, exactly. Minimum viable delivery objectives. So, like, what is the core thing that you need? And the problem is with many organizations' resiliency and business continuity plans, they effectively approach it as, what do we do today and what do I have to do to meet a recovery time objective for each of those things? And people should still definitely do that. But what they should also do is say, what is the 5%, 10%, 15%, or 20% of my business that is just absolutely societally critical that I have to make sure it runs essentially no matter what? And then back to the stress test point, you're subject to stress test the hell out of those things to find where the limits are. And then your board or whoever can then say, are we comfortable from a risk appetite perspective about where the stressed limit on the ability to run those, that minimum viable set of services is? And again, this is a different way of thinking about it than the more traditional kind of resilience business continuity type approach.

It reminded me of the point that you made about like, oh, this system may be too scary to test, probably would crash, let's not test it. If this system is part of your core, then changing it becomes kind of a hard must. You then, because to me, a lot of this, when I was reading the report, I felt some of this is like kind of hard to do. And then I talked to people who do this here at Google and they said, yeah, you're right. It is hard to do.

Well, you know, we were also, you know, we spoke to a number of people that work on things like chaos engineering and, you know, this kind of particularly like for security testing as well as resiliency testing. And again, to your point about what we do at Google, which is kind of randomly taking down things to prove that those teams are resilient. And the fact that they know that's happening makes sure that they are resilient. And so, there's a lot of that type of work.

I like the shout-out for Kelly in there.

Yeah.

So, what's your advice for people getting started with this? How should people start working their way through the PCAF framework? Yeah.

So, I think it's, you know, find and think about minimum viable delivery objectives for what they're business or services. The second is probably just pick 2 or 3 of the leading indicators to go after. I always recommend people go after the software reproducibility one because that just has so many business benefits as well as security benefits. Similarly, the infrastructure reproducibility one for the same reasons. I also like going after the cold restart problem because, again, we've done this.

That's scary.

Yeah, well, it's funny, we've done this with a few of our customers where we say to them, like, we know you think your backups are fantastic and they probably are and they test really well, but do us a favor, try and recover this business from a set of blank cloud instances or bare metal machines. And it turns out there's a common set of patterns why some organizations can't do that. You know, sometimes they have backups, but they don't have the index to the backups in the backups. So they—

Or the backup calls the AD server and AD server is ransomware.

Yeah, exactly. Or there's no They don't have software in their backups, and then they don't have the ability to reproducibly build the software on a recovered system. Or the other one, there's a— you reminded me on AD, there's circular dependencies. I've seen one organization where they were wiped and they, you know, to bring back DNS, they needed to authenticate that server to AD, but AD wasn't available.

Oh no.

But they couldn't bring back AD without DNS, and they had no protocol to kind of separately restart both without the dependency so that they could then go. So, there's a lot of things.

Teeth grinding.

I'm like, I'm shaking here.

Tim and me are both like, ah.

So, again, you find these patterns when organizations test backups by, when they do a backup, bringing a file back and going, file is there, that's great, is very different from the backup and recovery that you need in an event where your infrastructure has been wiped.

One other thing I wanna pick at here on the note of, you know, bare metal versus cloud, which is more resilient? Not to ask a leading question.

Yeah. Well, so this is, you know, to set you guys up and Anton in particular as the author of the paper, which we'll obviously send out in show notes links, is, you know, we did map—

See, listeners, he's been here so many times, he knows about show notes.

Yeah, exactly. So, the thing that we had, the paper that Anton and others put together was to basically say, let's look through the PCAST recommendations, and say, you know, clearly, we're going to say, and how easy is all this to do in the cloud? It's actually true. I mean, I know we're kind of talking our own game here, but the fact that in cloud, and this comes back to some of the things we've talked about on megatrends, the fact that we have more inherent software and infrastructure reproducibility in cloud, it's easier to demonstrate the modularity through kind of isolation between products and zones and regions. It's easier to get the telemetry to show that's happening. It's easier to rebuild systems and it's easier to even in a, in an extreme resilience way where you're wanting to operate in a multi-cloud environment with open standards and some of the other things, it's, you know, you may have to constrain yourself to not use a particular proprietary feature of a cloud to be able to operate in a multi-cloud environment. But you can then do that, which is very different from the resiliency properties that you would traditionally get in an on-premise environment.

And at the same time, I think during the paper development, it came up that Cloud isn't like a magical easy button for all elements. It is if you cover it all. I'm thinking of some kind of an integral, I guess. In this case, it's easier, but it doesn't mean that it's an easy button for everything.

Are there places where it's negative?

Yeah, well, make a feel question. Make it a feel question.

Okay, sorry. I pointed at Anton. We're all in the roomless.

Actually, wait, wait, wait.

First time we're all in the roomless.

There is one. There is one. What's that? Internet is gone. Gone. It's like gone for good.

Then.

You would have a negative in the cloud.

If internet's gone for good, I want to progress.

Well, so it depends. I mean, again, these become— this is why stress testing is it, you know. So if you had the stress scenario that internet is gone, I don't think it's gone forever, but, you know, let's say internet— a stress scenario is internet's gone for like several days and your only means of connecting to the cloud is internet, then that's something that would probably be highlighted in a stress test and you— they may want to have as we support private connectivity to us. And in fact, you know, I've worked in organizations where we did exactly that. We had kind of direct private connectivity as well as internet connectivity, and they were all backups for each other.

Okay, at the last moment, I think I did put it in the paper. So it was actually good timing. So I think it's time to close.

Do you get to ask the traditional closing questions?

I think I'm gonna ask—

I mean, do it.

I would skip the reading question because I want the PICAS papers to be the reading tip, but like maybe one tip I think Phil gave a lot of like short pithy points to use, but if there's one thing to remember from all this—

A single thing.

A single thing.

A single thing is just thinking about what your minimum viable delivery objectives are for your business. And that is something that you— the great thing about doing that is that's something that you can only do with your business leaders, your board, because very, very few businesses naturally think about what their minimum viable delivery objectives are. And a bit to your point about kind of planning versus plans, the very act of having that conversation raises the prospect to the executive team and the board that maybe they aren't capable of dealing with some more extreme stressed events that they might actually have to deal with in the future. And just the conversation of focusing on minimum viable delivery objectives focuses people's mind in a very positive way.

That makes sense. And it's also useful and immediately usable. So, I love it.

Oh, so applicable.

That's why I've got a lot of 59-time guests on the podcast.

Well, I look forward to number 10, and maybe we'll do that one in person, and hopefully with a more resilient operating system in Anton's hands next time.

Okay.

Got it. Well, great to be here.

Thank you very much. Thank you. And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode. Security.