EP132 Chaos Engineering for Security: How to Improve Software Resilience with Kelly Shortridge

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. This episode was live streamed on LinkedIn, and so if you want to catch our other LinkedIn live streams, please follow the show there and stay on the lookout for future live streams. You can also catch up on all of our past live streams by checking out our YouTube channel. Anton, this is one of the episodes I was most looking forward to. And one of the episodes I am now most proud of. And I think it also had, to give it one final superlative, the widest-ranging recommendations of books.

Yes, correct. And I think that when we posted this on LinkedIn, somebody said, this is going to be epic.

They were right.

And I felt like epic is an understatement. I felt like it's going to be incredibly unabashedly epic. And I think it was. We had very high expectations, and I think she exceeded them.

She did.

Actually.

She did. And maybe, listeners, with that, without any further ado, let's welcome today's guest. All right, listeners, we have had an awesome experience in debugging a complex-to-debug system this morning, which proved itself to not be especially resilient. It was difficult to recover from failure. And so, maybe with that just perfect introduction for today's guest, let me introduce who's joining us today. We are delighted to have Kelly Shortridge, Senior Principal Engineer in the Office of the CTO at Fastly, join us today. And Kelly, technical foibles of our podcasting app aside, what is security chaos engineering?

Yes. So, security chaos engineering, when you hear that, think software resilience. If I had my druthers, I would go back and have a time machine and change the title of the book, but here we are. So, security chaos engineering is really this underpinning of resilience. It's the ability to recover from failure gracefully and adapt to evolving conditions. What we see across all sorts of complex systems, domains, which, if we wanna nerd out about each, we totally can, and the book definitely does. But what we see in these domains is that adaptation is really the key to being resilient to all sorts of failures. And what's beautiful about using software resilience, I think, is that you can draw on all these different lessons from these domains. And a lot of these domains have thought for decades about what resilience means, and they've tried different strategies. We can just steal and borrow those, right, which is really powerful. So, I think it's a very elegant way to transform away from, you know, the Department of Know Stuff, though I think think, I hope we all know, is very outmoded, towards something better and more modern.

I have a very non-serious question. So if you're time machining to change the title, would you also change the picture? Because this bird is just giving me this vibe that I'm not quite sure what to think of it. Is power, or is it like flight, or, or anything? So would you change it to some more funny animal? I don't know, platypus or something?

I think it's the crested caracara. There's actually a lot of drama behind what the bird is. It's gone back and forth on which bird it actually is. That, again, that could be a topic for another time. We don't actually get to choose the bird. No O'Reilly author gets to choose the bird on their book.

Really?

As far as I understand, there's kind of like this oracle existing within O'Reilly that, you know, thinks about the vibes, divines it out of the ether, and then bequeaths it to the authors. So, if I could choose one, I would definitely choose a snow leopard because they're having to adapt a lot because of climate change and other human activities. Also, they're adorable. And throughout the book, you'll see Savannah Glieschka, my illustrator, illustrated Chaos Kitty, because I love alliteration, and it's a fun way to kind of verve up your cybersecurity transformation journey. So, I would definitely choose a snow leopard, but I don't get to choose.

Ah, interesting.

I had no idea it was a magical, oracular process. The concept of what you described there, though, I'm excited already because this is starting to feel like what we had last week with our SRE Part 2 episode, where we got to learn from other disciplines. And the idea of adapting to changing conditions and overcoming failure sounds a lot like me skiing. So I'm excited about that as well.

No, it's super applicable to everyday life. Like, I tried to weave in a few almost poetic philosophical things. So people are like, you know what? It doesn't matter if I try an experiment and I fail in my own life. Like, I shouldn't think badly of myself. It means that I'm learning, and I learn from this failure. It's not a reason to feel bad. So I'm hoping it's a very uplifting message too.

That's a good message.

Yes, and I think that the resilience part is also— it's also chaos feels like it's a means to an end here. Resilience is kind of a goal. So, it's almost like maybe you shouldn't take this time machine, right? Because ultimately, you can call it software resilience, or you can call it chaos engineering because one is a means, another one is a goal, right?

Agree with that. Yeah.

I wanna go to one big, I don't know, for me, it was kind of a big draw of the book. So, I was reading chapter 5, and it was about observing. Of course, I'd love to observe logs and all this stuff. But one thing that mystified me in this is that when you describe how to deal with alerts, there's a lot of stuff kind of what not to do, and what would kill you, and what would get you. But when you try, how do you practically succeed with alerts? And I think, is there any chaos engineering-inspired advice for security alerting, for SOC, for detection? That's what I'm curious about.

Oh, definitely. I think everybody knows about alert fatigue. I think what we want, though, again, thinking about the scientific method, which is something that I talk about throughout the book, and really is the basis for what a chaos experiment even is, the idea is that whatever evidence we're collecting, and I very much view signals or alerts as part of evidence about our systems, we want to make sure it's actually informing whether we reject or accept a hypothesis, right?

Mm-hmm.

If it's just, like, cool information, should we actually be collecting it? So, I very much view something like chaos experiments as extremely complementary to refining alerts. So, as an example, Aaron Reinhardt, who cultivated all the amazing case studies we have in chapter 9, he actually pioneered security chaos experimentation at UnitedHealth Group. And what he found when they did a very kind of basic experiment, which was introducing a misconfigured port, they expected that the firewall would alert on it at the very least and hopefully block it. It turns out the firewall only did so 60% of the time. So, it's like Anchorman 2, 60% of the time, it works every time, right? But I think that's really interesting because also their commodity cloud configuration management tool alerted on the change. So, it shows you that actually maybe there's a different sort of alert than you considered before that might be more valuable. It's the same thing I've seen with some kind of, like, container security use cases or cloud security use cases, billing alerts can be great signal for security use cases as well. So, I think these kind of experiments, even if you don't conduct the actual experiment, you're just using something like decision trees to walk through your assumptions, it's really getting critical about, like, what assumption is this validating? And if it's not really directly validating an assumption, maybe we don't need to be collecting this data. Though I also understand it's human nature to wanna, you know, hoard data like a dragon, so.

So, you can do tabletop chaos engineering. Wait, wait, wait. So, I'm just gonna derail this whole story to go and say, you brought up the point about not doing it, doing it, kind of like pretend doing it? So, is it like tabletop chaos engineering, then you explain how it might happen and others respond?

Sort of.

Would it work? Is it a good idea, a bad idea, iffy?

Yeah, I talk about the E&E approach in the book, which we can go into if we want. This is more, it's a concept I've drawn from behavioral science, specifically behavioral game theory, which is called belief prompting. So, the idea is that most people aren't very good, what's called, like, K-level or N-order thinkers. If you think about chess, they're not necessarily great at modeling out, like, oh, if I do this, the opponent will do this, and then I'll do this, and et cetera. Belief prompting helps you do that. And decision trees are a great device that basically prompt you to think about, well, if the attacker does this, I can respond this way. But if I respond this way, the attacker might do this, this, and this instead to get to their goal. And you really think about what is the easiest thing for the attacker to do, not the super cool esoteric, you know, nation-state thing. It's just what will they do that's efficient? And that Honestly, if you know or you aren't quite sure that you can actually detect something, that's the place to start. Don't conduct an experiment when you already know you're probably going to fail in the experiment.

Mm-hmm.

You want to conduct experiments for things like a firewall alerting on a misconfigured port that you're really confident that it'll do. And that's when you should really make the investment to validate those assumptions. But if you aren't sure, you need to just get all of those assumptions out there, because once you have a decision tree, each kind of like interaction between the attacker and defender, whatever mitigations you have in place, that's a hypothesis right there that Then you can conduct an experiment to validate or deny.

Yep.

It's almost like don't pen test before you secure, to use an analogy, right? Like, I always joke when people say, I'm going to do a pen test, and I'm like, I can tell you right away your network is— your security is kind of broken, so why do you need to pay for it? Like, fix it first and then test, right? It's the same kind of metaphor, I guess. Okay, sorry, I don't mean to interrupt you.

But I also really like the advice that you can't do this and then not pay attention to the results. That's kind of like the story we got about the CISO who got threat intelligence reports so he could read them on the can. You know, it's not useful unless you make— serious story we had on the podcast. So, I wanna get back to the cloud security angle, though, because you mentioned that, and this is, in theory, a cloud security podcast. What's the intersection between chaos engineering and cloud security? Is this like peanut butter and chocolate, or peanut butter and pickles?

Ooh, I think it's a bit tricky because I very much view, again, software resilience, security chaos engineering, whatever you call it, as more of an organizational transformation. It's a set of principles and practices. I don't think it prescribes any particular tool or architecture. Certainly, throughout, I emphasize the benefits things like infrastructure as code, I think there's a lot of kind of baked-in design that the cloud providers have heavily invested in. I promise Google is not paying me to say this. I do, though, think that there are just some concerns that you don't have to worry about, because obviously, it's a comparative advantage thing, where if the cloud provider invests in it, they receive what's called economies of scale. They get more from their investment than you can do individually in, like, for your own servers or data centers. So, I do think, again, when you're thinking about how do you enable that nimbleness and adaptation that's really important for resilience. When we think about adopting, like, cloud computing, that can be really helpful if you think about something like serverless, which is important to my employer because we offer a serverless platform. I know you all do as well.

Awesome area.

Yeah, it's really cool because all you have to worry about is, like, the logic that you put into your code. You don't have to worry about provisioning anything. It's got, like, at that point, the attacker would have to break out of 2 sandboxes to compromise it. It's just no attacker's going to do that, right? So, there are a lot of things that you gain by default and by design that are really important. So, I do think there's a good, like, symbiosis there. I think peanut butter and chocolate work really well together, in my opinion. So, I think it's more like that. But again, sometimes a cloud-based system isn't gonna be right for whatever your context is, especially maybe, like, a super-duper critical, you know, it's gonna cause, like, a nuclear meltdown thing. Maybe you don't want that, I don't know. So, I think a lot of the book, I tried to not be afraid to say it depends, but I do think overall, security leaders have been overly cautious about innovations basically at every turn. I was there when they were like, no, we can't go to the the cloud. It's too insecure. I can do better securing, you know, my own mail server, whatever. It's like, no, you can't, for one.

No, you can't. You just can't.

Right? But there are opportunities for security there too. And it's the same thing I now see with, like, oh, there's, like, shadow infrastructure as code. It's like, you completely overlooked the laundry list of benefits if security used infrastructure as code. So, maybe we need, again, we shouldn't just think about threats, but we should think about opportunities as well.

And I actually really liked your point about it depends on the episode you did with Corey Quinn. I listened to that this morning. I think that nuance and bringing in the, well, what's really the threat model for this thing? We talked about this recently with the Panther episode. Really relevant for security teams.

Absolutely.

And I think the excessive fear is— it's almost like the whole exchange, I can run my own email server, is at this point kind of a meme. Because the episode we aired during the week we are recording this, we asked— that was like the question sort of formulated to a CISO. What do you tell to— I'm not going to add any curse words, What do you tell to people, fine, I'll say people, who insist that they can run Exchange more secure than Microsoft? And like, I wouldn't ask this question myself because it would be like, I would just spew curses. I wouldn't say anything polite. But the CISO who we had as a guest kind of had a more nuanced answer. But ultimately, the answer is, don't do it. You're wrong. That's the politest version of that.

Yeah, there are a few behavioral biases that actually come into play. There's something called the IKEA effect, for one, which is when we build something or create something with our own hands, then we value it more. Because obviously, if there's like a crappy piece of IKEA furniture, our version of it isn't any more valuable than someone else's version, but we feel like it is because we own it. And it's the same thing, actually. There's that kind of ownership bias where I think it was coffee mugs, when they gave someone a coffee mug and was like, well, this is yours now, and they wanted to sell it, the price was much higher than they would be willing to pay for the same mug. They didn't even create it.

Yep.

So, I think there's that kind of— I think in general, this is getting a little philosophical, which I— it's my weakness. I always do it. But I think there's just a very primal thing we where we want to feel in control of our environment. We want to feel like we can influence the world around us. It's part of our agency that's helped us succeed. And I feel like this manifests in security in this way where we want to feel like we have control over things, even if it's resulting in worse outcomes. So, I think it's a very pernicious quirk of how we think.

Actually, I really like that.

And I think that's stuck with us. Yeah, I love it. I love it. It's stuck with us for so long.

Yeah. I don't think that's too philosophical at all. And back to what you were saying about philosophical advice in the book, I always joke on the show that CISOs often don't need more tech, they need, like, a family counselor.

Oh my God, yes. Therapy.

Right? Right?

Please get therapy. Software engineers too, by the way, but that's a separate conversation.

Yes. Software engineers definitely need therapy, but I think CISOs especially need, like, a family counselor to sit down with them and other department heads and work their shit out.

Oh, yeah. Absolutely. Yep.

All right, Anton, get us back on track, please.

If somebody, if a CISO, a security leader, a security engineer, architect read the book, and he wants to do it, he wants to now do it, implement or start using chaos engineering for software resilience, for other areas of security. What's your best advice? I don't know, however you wanna structure it, for how to start?

Yeah, it's a bit meta because one of the opportunities I describe in the book is iterative progress or an iterative approach to development. And I think that's exactly what you want for this too. Trying to do a big bang, adopt everything, it's not gonna work. I highly recommend, in chapter 2, I introduced the E&E approach, which is for evaluation and experimentation.

Mm-hmm.

So, evaluation, the first thing you want, you wanna make sure you have like flow diagrams in place, layer that over your architecture diagrams, just understand how things interact. You can even get outputs from things like integration tests. Side note, I've been very depressed to learn that many CISOs don't know what integration tests are. That's a digression though. So, once you have kind of those sense of flows and architecture diagrams, you can start building some of those decision trees where you just document your assumptions around, yeah, I think we can detect if there's like a misconfigured port change, or just how you think attackers will move throughout the system. Again, which you can layer and map to those flow diagrams. Until you get to that point, you really don't have an understanding of which assumptions you're confident in or not.

Hmm.

And so, it's very difficult to start your experimentation until you have that sense. Also, all of those artifacts are gonna be useful beyond just chaos experimentation. It's just useful for resilience to understand how the components interact, because that's really where things go wrong, is in the interactions between things, rarely just— it's rarely isolated to a component itself. That's why cascading failure is so scary.

Wait, wait, wait. So sorry for interrupting for a second, but the whole point about it being meta is stuck in my head too, because it's almost like the very principle of this is that it almost doesn't matter where you start. What matters is you adapt and you iterate and you kind of like—

Precisely.

It almost, it's not the starting point. It's the logic you follow afterwards, which is kind of a little bit mind exploding, extending for me, because I'm like, it's always like, I used to torture people to like, what's your top 3 steps? What's your starting points? What's your starter guide? And in this case, it's like, Anton, just stop. It doesn't matter where you start, it's the sequence of steps and the logic you follow afterwards, right? This is profound.

Anton, you're not gonna believe I'm gonna say this, but I wanna compare it to ASO. It's very similar to the thinking you guys have in the ASO paper, that it's about iterative improvement and about making yourself get better over time, not a big bang transformation.

Yeah, I'm a big fan of things like the strangler fig pattern, and that's part of the reason why I list so many opportunities, especially chapter 4. Chapter 4, for any listeners, is a beast. It's about what you can do when you're building and delivering systems, which, as you imagine, involves a lot. But one of the concepts I introduce is the idea of the effort-investment portfolio, which is basically, to also avoid cursing, you have only so many cares or concerns you can, like, fulfill.

To give.

To give, yes. And you have to allocate those very carefully. And so, part of what I'm doing is proposing this wealth of opportunities where maybe you won't pursue all of them, but you can think about your context, how ready your organization is in some dimensions. Like, on LinkedIn today, I was just talking about, like, some organizations, it's gonna be easier for them to refact refactor into a memory-safe language for whatever their critical system is, if it's currently written in C. For other organizations, it might be, hey, we do need to adopt something like infrastructure as code to make these deployments more repeatable, to have these audit trails, to have the ability to roll back more easily. It really just depends on the organization. And I think what's powerful, like you said, it's not just, okay, do 1, 2, 3 in order. It's, well, you actually have this flexibility, and you can try things out. And maybe you try refactoring, and you're like, yeah, this is not for us, Let's try one of these other opportunities and maybe break this down a bit more. So, there really is this flexibility. And I know this is not great for me in terms of selling out, because I think I could make much more money having, like, here are the, you know, 10 steps to perfect resilience, but that's just not the reality, right? We care about having better security outcomes. And so, the reality is, again, having that adaptation, being able to be nimble. I do think, in general, making sure that, again, collaborating with your software engineering peers, having those architecture diagrams and sense of flows, That's gonna help for their performance goals and quality goals, even aside from security. So, I think that's a very kind of, like, common ground artifact that you want. In my experience working with software engineers, they actually love putting together decision trees. They really like putting their, like, nefarious hat on and being like, okay, if I really wanted to, you know, disrupt this system, what would I do? And so, I feel like those are kind of natural steps to unify, like you said, kind of like a group bonding session.

Yeah.

But it's also just essential to inform any sort of strategy you put in place, whether you experiment or not, ultimately.

I think for a lot of software engineers who've worked on a system for a long time, that conversation is like the Office Space smashing the printer moment for them. They get to let out some feelings.

It's cathartic. Yeah, totally.

Yeah, it's cathartic. It's cathartic. So, I wanna get tactical for a second and say, like, in all of the chaos engineering experimentation you've done, what's a standout favorite experience of yours? Maybe because it was interesting or because you learned something unexpected. What's your favorite example?

I feel like there are so many. I've definitely seen examples where when we've done decision trees, again, kind of the earlier stage, maybe you wanna adopt. They're just, like, really basic assumptions that we're just like, oh, yeah, the attacker could just do X. And again, I wanna keep it because a lot of it's proprietary, but the attacker could just do X. Well, we at least now know the, like, most urgent thing for us to do, right? And it's something that in hindsight is very obvious, but that's why hindsight bias exists, because it feels very obvious, but it wasn't before. I think in terms of experiments, it's really hard to beat the 60% of the time it works every time experiment that Aaron conducted. There are some others I think are really valuable, especially, I know, And we have a case study in chapter 9 from Opendoor about logging pipelines. That's absolutely essential because, as you all well know, if it's during an incident and your logs aren't working as intended, oh my God, does that hurt, right? So, you really wanna verify that they're working as you expect in a variety of conditions. I think it was, what was it, the Census Bureau? There was a report where they were sending logs to a SIEM that had been decommissioned for over a year or something like that. You don't want that.

No.

You definitely don't want that. So, chaos experiments like that can be helpful. I built a chaos experimentation lab. implementation tool on Fastly's computed edge platform that strips cookies, and I think authentication headers, and also forces cross-site origin. Again, those are really basic security assumptions, but it's really bad if they're not in place. So, I think that's one thing I would like the listeners to know is you don't have to start with, like, crazy experiments. It really is start with, like, the basics and make sure that your assumptions are valid about them.

Then follow the process to kind of, like, iterate and learn. Because I think, to me, this is the part of advice that comes out from the book quite strongly. also from this discussion. It's not about a quest for a perfect starting point, the simple, perfect, lots of Venn diagrams, what to do. It's like, start somewhere and then rapidly iterate. So, to me, this is kind of hard for a lot of security leaders, I bet. I think that this quest— like, I'm sensing the neural pathways in my brain, kind of like signals going the other way. So, it's kind of interesting that I don't have it that often, but I do this time. And by the way, this brings us to the next point. A lot of this is very modern in the sense that You mentioned containers, microservices, and of course, you work for a very modern company. You deal with modern environments. And we see, well, we in this team, not just Tim and me, see people who are kind of stuck in the '90s a little bit in terms of security. And lately, I've seen a couple of people that are stuck kind of in the '80s, in the pre-login centralization, maybe. So, they're not quite in the '90s yet. But is chaos engineering and this drive for software resilience is maybe a kind of of a forcing function to drag people out? Because it's almost like if you talk about microservices, but I have Windows 2003 and Solaris. I don't know. I'm being extra sad.

Like your laptop, Anton, right?

My laptop is more modern Windows, so stop joking about this. It's not a Chromebook. Fine. I'll give you that. Yet. Okay. But can we use the art and craft of software engineering and software resilience to drag people out of the past into the present? Is it too philosophical?

I think in my bio, I state that, yeah, part of my mission and my career has been dragging security out of the dark ages. So, I definitely align with that sort of thinking.

The '90s being the dark ages, I'm assuming that's what you mean.

Well, it seems like we're still in them. I don't know. I'm not sure what era we're in. I know the joke is that ops is 10 years behind dev, and I would say security is 10 years behind ops, if I had to estimate. I do think I wanna make sure that modern isn't just equated to things like microservices for the audience, because sometimes microservices can be enormously I actually think the organizational benefits of being able to make changes more independently are huge. That's something a lot of leaders, even outside of technical leadership, want. With that said, if you want to sustain software resilience, you don't have to do microservices if you don't want to. A big part of it, though, is, again, that adaptation. It's being able to make sure that you can move quickly. That is very much not like a waterfall, big bang release, like, huge, hulking mainframe monolith sort of approach. that's much more difficult to pull off. But the reality is, and actually in the book I cite, I haven't talked with them, but Discover Financial Services has a great talk at AWS re:Invent, I think from 2021, about breaking apart their monolith for payments processing. They knew that they were gonna keep their monolith for sure for some of the really core stuff, especially with things like SWIFT, it's super complicated. But they saw huge performance gains, and there are also security gains there when they were able to at least move out some of the functionality and migrate some of those services into, I think it was AWS, no, No offense, I know this is a Google Cloud podcast.

Oh, it's all good. It's all cloud.

I'm sure it would be the same for Google Cloud.

Yeah, it's fine. Yeah.

So, getting those kinds of benefits, it's understanding, again, that what's your unique context? What makes sense to you? And they actually found that when they tried to migrate more all at once, and they tried to not just move to cloud, but go microservices all at one time, it didn't work. They had to get more iterative to be able to make it successful. And I think that's a really great example of, again, like, if you're a Fortune 500, you're not gonna be like the scrappy startup that starts out like super zero trust, everything is cloud, SaaS and microservices, that's not probably gonna work for you. But maybe it does make sense, both for performance and security resilience reasons, start teasing out, you know, maybe your billing service migrates to the cloud. Maybe you now get closer, you know, you deploy something as a serverless function, like A/B testing can be a great use case for serverless. There's a lot of flexibility that you gain when you just, you open your mind to the idea that, okay, our goal is to, like, be able to make changes quickly and adapt. What's the best way to get there? Forcing something that shouldn't, you know, again, thinking about, like, a heavy, like, Swift or something in COBOL, trying to move that into a serverless function, you're not gonna be adaptive because you're gonna just be mired in all that effort, right? It's making sure you're taking a step back and, again, thinking very much almost, again, about that portfolio of effort and what allows you to have enough resources to be able to adapt as things change. And also, we all know with Hacker News, there's always a special flavor of the month, especially with, like, JavaScript frameworks. So, like, Don't go all in until you're certain it's actually gonna help your use case.

But actually, wait, I think that you answered, you didn't say it, but you kind of answered it like that. Because I think it is a forcing function because if you accept that you want to be agile and adapt and rapidly move and experiment, and you look at your systems, and the systems are not like that and don't allow that, and they're building this in the '70s by, you know, a large mainframe vendor, and everything else is built over many years, you realize that if you adopt the vision, you have to toss the systems. Otherwise, the vision is not gonna fit. So, it is kind of a forcing function if you accept that you wanna drive to software resilience, is it?

Yes, I should say, I did not say that. I do think it's absolutely a forcing function to modernize. I think that's one of the big value propositions is you can actually start leveraging a lot of modern technology that you haven't been. I do wanna stress that I view systems as sociotechnical. It's both the human So, sometimes, again, transforming processes can be more impactful. Again, making things more repeatable through, like, CI/CD or infrastructure as code, that can mean more to your organization than going, like, microservices, right? Just because it means humans don't have to try to replicate something manually the same way perfectly every time, which humans aren't designed to do.

We are really bad at that.

Oh, so bad at that. It's just not— we're great at adaptation and creativity, we're not very good at Like, very manual repetitive work. So I do think it's a forcing function.

That's why, like, professional darts players are actually impressive, because they do exactly that, and we're not good at that as a species. I think one of the really interesting things that also you maybe didn't say in these words but came through is that the goal with this is to promote progress over perfection.

Yes.

We're not trying to be perfect on day 1 with this. We're trying to develop a practice that allows us to get better through that practice.

Precisely. Perfection doesn't exist. And also, perfection is, I would argue, a verb. It's very much a moving target. What looks like perfection at time 0 is gonna look very different at time 2, because there are so many variables. Just when we think about our software systems, there are just so many variables out there, right? Perfection, let's say, 3 years ago in tech is very different now with, you know, these hard macroeconomic conditions, right? Efficiency is much closer to perfection now. So, the I feel like the bar for perfection always changes. So, it's not a very good thing to chase, quite like happiness, actually, again, to get philosophical. So, for sure, just making sure that you're continuing to make progress, that you're learning, because not everything will succeed. But as long as you're understanding why it didn't succeed, and it's not just pointing fingers at people like, oh, you did this bad thing, it's actually understanding all the factors, then I think that ultimately, you will look much closer to perfection than if you tried to chase it, which is You make it sound like common sense.

It's almost like you're not saying this, and for the record, Kelly did not say that, but it's almost like if you don't do it, you're an idiot. Because what you're describing is like, that's the only— it's like you do things, you think about the outcomes, you incorporate them, you learn from feedback. It starts to sound like it's like the only normal thing to do is that approach.

Anything else would be illogical.

Anything else would be illogical. Thank you, Tim, for being so mild.

This is how I feel, but I think if I called my potential readers idiots for not doing it, I wouldn't sell many books, right? Not that I think they're idiots, but— Yeah, yeah.

But if you call it a common-sense approach, if you call it the common-sense approach to life— I think so.

I think it's— yeah, because obviously, success in life is a lot of luck. That's gonna be true in security, by the way, as well. Like, you may just get lucky. Like, what we really want, again, is to be able to maximize the opportunities that we can actually seize when luck comes in our favor. So, to me, making sure that you are able to be nimble and seize those opportunities is And again, learn when they don't pan out, like, that is ultimately the key to success. A lot of people really don't seem to get that though. There's a huge— I think it's partly loss aversion, which is that losses feel twice as terrible as gains feel good. But it's something I do in my own life. A lot of people ask me, like, how do I write a book in 9 months or do all these things? And it's because I try to just suck it up and be like, yeah, I may fail, but I'm going to learn a lot from the experience. And I've definitely had failures, right?

Yeah.

So, it does feel common sense, and it's also sometimes a bit sad when you see people that really are stuck in that fear mindset, and they don't make forward progress because of it.

Totally.

So, again, this is why I hope the book inspires you more than just the software parts too.

I feel like this book is much more than a software book all of a sudden.

That was kind of my goal. I mean, I have some very interesting elements, and I tried to weave in a lot of, frankly, some weird citations. There's one citation by Rovelli, which is basically about how As humans, we don't understand the nature of time. So, I'm trying to lead people down really interesting rabbit holes if they wanna pursue them. But I think if a book starts out with a quote from Faust, I think it's unlikely to be a typical software book, right? So, I appreciate you acknowledging that.

Yeah. I actually love Rivelli's books. I just finished one the other day, his Anaximander book. I thought it was great.

Oh, yeah. Yeah. Yeah.

Okay.

So, maybe with that, it's time for our traditional closing questions. One, do you have a tip to help people improve their software resilience? That's practicable. And then 2, aside from Rivelli and Faust, recommended reading for our listeners.

Ooh.

And your book, which, of course, we'll feature all over the place.

Of course. Of course. Everything You Need to Know. I mean, I will say, with the book, I did intend it so security audiences, a lot of people in security, I've learned, have been a little scared to admit, or a little embarrassed to admit, that they don't understand how software works, or how software delivery, like, how the sausage is made. My book is trying to teach you that. You can also give it to your software engineering colleagues, because I'm trying to teach them about security at the same time. So, I'll put that out there. So, first question was tips for software resilience.

Like a practical one.

Practical one.

Or a weird one.

Or a weird one. Oh, there are definitely lots of weird ones. Or a weird one, yeah. I think if you— I really like the security-by-design angle, so I'll bring up things that's both weird and practical, things like type systems. A lot of security people don't know about them, but actually, with static typing, you can encode a lot of, like, intention for the code into the type system. I'm, like, grossly oversimplifying, and I have more of an explanation in the book. But there are a lot of opportunities like that, again, with things like memory safety too, where if you talk to your, you know, really nerdy software engineering colleagues, and you say, like, hey, what do you think would help, like, reduce exploitation? They might come up with things like either one of those, or things like immutable infrastructure.

Mm-hmm.

That might be from an SRE, might think that. Same with ephemeral infrastructure that vanishes. It means it's a nightmare for persistence. Again, there are a lot of opportunities that actually don't require you to buy, like, a blinky, super-hot AI security tool. It's actually just choosing, making different infrastructure or coding choices. And so, have an open mindset and allow your colleagues to teach you about some of those opportunities. Obviously, they're in the book too, but that would be my practical advice is, like, keep an open mind that maybe the answer to security isn't always within the security industry. Second thing was recommended reading, obviously, aside from my own book.

Hmm.

I really like, actually, Gene Spafford and Josiah Dykstra, and I think Lee Metcalf recently came out with a book about cybersecurity myths.

Hmm.

I feel like it very much aligns with my book, where it kind of dismantles a lot of the folk wisdom that pervades the industry that generally leads us astray. So, I think that's a good one. And then, if we're talking about, like, not necessarily in security, I always recommend Martin Kleppmann's Designing Data-Intensive Applications. If you want a crash course in, again, how the software sausage is made, that is an excellent one, because he talks about all the trade-offs of different approaches. So, that's a fantastic deep dive into things. And then, if we're talking about, like, we wanna get really weird in terms of reading and what might help, I think any sort of book that kind of touches on resilience in a different domain is interesting. I highly recommend Normal Accents by Charles Perrow. I use that book extensively throughout, especially in the 4x4 matrix in chapter 3, talking about different kinds of failure and how complexity and coupling can conspire to lead to catastrophic failure. And Pero wrote about all sorts of systems, like chemical plants, nuclear plants, air transportation, all sorts of stuff, I think, back in the '80s, in thinking about resilience and how do we avoid these kinds of catastrophic meltdowns. So, that's a really great one if you wanna understand kind of how it looks in other disciplines. Because frankly, when I've been reading across all these disciplines, the main takeaway I have is, oh my gosh, we have it so Easy in cybersecurity. Software is so much easier.

That's an unusual one. Usually we get to the opposite.

It's so easy. No, no, no, no.

So okay, take an airplane compared to like nuclear safety. I believe you.

Yeah, take an airplane, right? There is an incident.

I think it's in the Charles Parrow book where I don't because you fight physics here. You fight humans. I disagree. Let's have a fight. Where's my pillow?

No, a coffee maker led to like some sort of explosion in an airplane. It was. There's just this totally baffling interaction. How do you simulate that? However, in software, we have virtualization. We actually can conduct simulations much more easily. There's a lot of recent scholarship. I'm talking about the last 5 years across these domains, which is basically, like, how do we get over the enormous expense involved in modeling our systems so we can conduct resilience stress tests? Guess what? Resilience stress tests are the exact same thing as chaos experiments. That's exactly what they mean. It's the exact same goal outcome. They don't know how to do it without this, like, incredibly complex, like, supercomputers and, like, crazy math and modeling software. We can do it, and we choose not to. It's the same thing with things like integration tests. Airplanes don't have integration tests where they can simulate, you know, 1,000 times in less than a second how the coffee maker is interacting with the electrical equipment. A lot of our teams just choose not to do integration tests. Like, it's a luxury. And I think in part, it's because people don't die generally when there are software problems. But I view it very much as it's a humbling thing when we realize, like, there are a lot of engineering decisions that have to be made in terms of safety that are just way more difficult and much more problematic trade-offs than we deal with. And we're a little spoiled, I think. So, that's a good way to close with a hot take, I guess.

Well, that is by any measure a hot take. Kelly, I can't thank you enough for joining us today. This was one of the widest-ranging, most delightful conversations. Listeners, the book is available wherever you get your books. Is that right?

Yes. Amazon, Bookshop, Target, a bunch of places.

Amazing. Kelly, thank you so much for joining us today.

Thank you for having me.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @Anton_Chuvakin and _TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.