EP133 The Shared Problem of Alerting: More SRE Lessons for Security

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit the subscribe button in your podcasting app of choice. You can follow the show and argue with us and the rest of our Cloud Security Podcast listeners on LinkedIn. This is a special episode because it was live streamed first on that same LinkedIn community page, and we had great engagement as always, and you can find the video content on our YouTube page. So Anton, what are we talking about today? This is a bit of a part 2 for us, which is kind of fun.

Indeed. And I think it's a really good episode with extracting more SRE lessons, Site Reliability Engineering lessons, as they apply to security. So you can read the SRE book, I can read the SRE book, we both did. We can extract some lessons, but it's usually better to get an actual SRE, in this case 2, and ask them how it really works. And we got some really solid stories and kind of really crisp metaphors. From the episode?

Some surprisingly crisp metaphors. And I think listeners, not to make you hang on the whole way through, but I think the most insightful thing happened almost at the very end.

Yes, but I think your metaphor about digging and then sighting the portal to hell was also pretty robust.

Listeners, this was recorded just 2 days after I had a D&D session involving gateways to the underworld. So shout out to my friends at Northreach. And perhaps with that, Let's turn things over to today's guests. Our guests today, Steve Magee, reliability advocate, and Aaron Eitelman, an operations advocate. We are full of advocacy today, which is awesome, and so excited to have everybody here. So I think what's really interesting is in the past when we've had Steve on the show talking about the SRE lessons, they've felt like truly mind-blowing to me and Anton. We kind of sat through that episode each just like with our brains way out here going, what, how, how is there all this wisdom? And we immediately agreed, like as soon as we hit end recording, that Steve would come back. And so Steve, thank you for coming back.

Made it.

We're delighted to have you.

Yes.

I think I want to give you the first question of the day, which is when we look at security people and we look at SRE people, there's kind of a common thread in their operational life, especially when it comes to alerting. And so they both suffer from noise. Tell us about the noise problem.

Yeah. So when it comes to alerting, noise is basically like the worst case of noise is, you know, getting woken up when I didn't need to be woken up. And that's the worst. And like, now I can't get to sleep. And like, it causes like, you know, personal problems, it causes financial problems, can cause health problems. So it's not just like noise noise, it's like pain, like human suffering. Often when we talk about alerting and separating the signal from the noise, it feels a little like nerdy and like not that big of a deal, but like it's actually a really big deal because it's like these are the humans that you brought into your company to do stuff for you that you're paying a lot of money and you're causing them suffering because you can't be bothered to fix the thing, right? So like, you know, top of the line, like this is a serious thing. This isn't just like a tiny improvement. So what is the noise generally in SRE and reliability? It's anything where I got pinged or paged or woken up or whatever, and I can't take action. Like, we call it, we call it non-actionable. So it's like either it's too vague or it's like, you know, stuff is bad, kind of, maybe, we're not sure. Or like the thing isn't well described. Like, computers are warm. So what, right?

So they usually are.

They're supposed to be warm.

I can tell you a security equivalent of computers are warm. Like, Network traffic is up.

Yeah, or that's good. That means we're making money, right?

Anonymous process.

Yeah, I mean, we wouldn't want the computers to be cold, which means we spent a lot of money on computers and we're not sending any electricity through them, right? The whole point of the computers is to warm up anyway, or at least that's when you know that they're doing their job. So like making sure that it's A, like it's a real thing, B, like I can understand it, and C, like I can do something about it. The last one actually kind of D or Z or something is like, is this something that I already know about that like we're already on top of and it's just like re-pinging me again because like the last person couldn't be bothered to do it? Or like we have a dumb policy where you're not allowed to silence a thing after a certain amount of time, or like those are the worst, right?

Mm-hmm.

So one of the things we recommend to operational teams across the board, whether it's reliability or security, is just keep track of this stuff, right? And just have a Notepad on the wall if you have to, or whatever, like a way to keep track of, of all the alerts you got, how many of them were actionable, right? And—

Is it because of the feedback loop?

Yeah, yeah, yeah, exactly. Yeah, you want to develop a feedback loop over time because improving that number over time is going to be different every single time depending on your environment and what you've changed and whatever. And like, it's going to get better and then it's just going to get worse again, right? So it's not— you're never done. So you always have to keep measuring this. And so this is a superset of what we call toil. Or sorry, this is a subset. So the superset is called toil, right? And the only way to measure toil is to ask people how they're feeling. I'm sorry, nerds in the world, including myself. I wish it was automated, but it involves talking and asking people about their feelings, and that's it. And you do it all the time.

It sounds like this is actually a subset of something that we also call entropy. The thing you described there, like you fix it and then it gets worse, or you're, you know, you're pushing this rock up the hill. This feels like just kind of the chaos of systems. So there's lots of things we've developed as the world to deal with entropy. What technology exists to help people here? What should people invest in to make their precious human lives not suck?

Actually, wait, before you go there, I want to briefly detour to the fact that a lot of people live like this for decades. So, to me, when I see this, when I read the first SRE book about toil for the first time, I kind of realized that a lot of what they say is toil is what every SOC does every year for decades. So, why is there reluctance to reduce noise? Like, I think before we talk about the magical tech that would magically fix it, is there a real reluctance to fight the problem? Like, I'm quite, you know, pained to even say that, but it sounds like there's reluctance to reduce this, right?

Absolutely. Yeah. And it actually, at least at first, looks very similar between security and reliability engineering, what that reluctance is. So, Stephen and I see this quite a bit, but it starts off with this sense that, no, no, no, we have to do the manual operation to fix the problem, right? That is our job. We are firefighters, we're heroes, you know, we've got to get in and do it ourselves. There is a difference between them, which I hope we get a chance to get to. But before that, when people say, I have too many alerts, that can really mean one of two things. One, we've been talking about, which is the signal-to-noise ratio. A lot of false positives, a lot of things I shouldn't be worried about. But in some cases, They're legitimately getting a lot of alerts for a lot of problems. And so that is a separate class of issue. But in both cases, having a robust system that you can count on, you know, you trust the automation that you have in place, et cetera. This is kind of how you start to move towards the solution to address Tim's question.

When you say system, by the way, you don't mean a computer system. You mean like a system plus approach, right? Just to confirm.

So Steve can catch me on this. So when I talk about robustness, I am talking about a technical system, right? Where you can have automation in place When we talk about resilience, that is this broader social technical system where you have people working together, responding as an organization to a failure, right?

Nailed it. You got it. Yeah, the other thing that I think Anton was alluding to was, like, often when we talk about toil and reducing toil, like, sometimes people define their jobs as the toil person. Like, right, why? What are you gonna do? You're gonna reduce the thing that I'm good at? Like, I'm good at that, dude. Like, and I need a paycheck, and I've been doing this for 25 years, so I'm gonna put up all the roadblocks in the world to make sure that you don't get rid of my job.

So is there like an anti-automation camp? The camp that says, you have an excavator, but I'm going to dig with my shovel because I'm good at digging?

Yes.

I'm sorry, but I'm—

The shovel lobby.

Big shovel lobby. I'm playing the naive guy who just showed up. I'm sorry, but I always assumed that everybody wants to automate, but there are challenges. So my mental model is that Everybody wants to automate. Everybody is the let's get machines to do it brigade. But there are these enormous challenges that they're facing. But you opened my eyes to the fact that some people say, no, actually, I don't want to automate. I'm going to keep digging manually because I'm good at it. Like, psychotherapy, please help.

Yeah. To extend the metaphor, like, use a shovel for a long time and someone comes in with a backhoe and they start going after whatever it is you were digging and they take up a gas line because you never in the past had the power to burst a gas line with your shovel. So you didn't have to mark the gas lines, then they'd be like, look, that backhoe is super dangerous. Like, what are you doing? Get out of here with your backhoes. We got a team full of shovels over here. And they're missing the entire point of like, we've got 16 miles of trench to dig here, guys. Right. You know, it's like, let's just mark the gas lines and the next water line we come up with, like, let's think ahead a little bit because we're going to be using the backhoe from now on. And so often it's just, it's just short-term versus long-term thinking. Wow.

But like in that example, The discovery of a gas line implies the existence of water lines. What about unknown risks that automation exposes us to? Like, maybe there's a gateway to hell down there or some other planar portal that the backhoe will discover, but the shovel certainly will not, because there's—

Someone watched Buffy. I can tell.

I've been playing too much D&D lately, is the truth of the matter.

Yeah, that's a good question. The sort of Esri approach to this is gradual change. Right. So the answer to this is don't just bring in the backhoe and YOLO, just like, let's do it, guys, you know, just go for it. Like, just dig a little bit first. Like, come up with like, hey, ask around. Like, what do you guys think? Is there anything down there? Should we dig a little bit for a while with the, you know, should we poke holes first? Should we look around to see if there's buildings that might have feeds to them? Or, you know, so basically it's like, think ahead, right? Uh, one obnoxious way of putting this is shifting security and reliability left. People hate that now. I think that just means think ahead. It just means plan a little bit.

Plan a little bit.

It's a radical concept, Steve, thinking ahead.

And in terms of novelty, like, when you do run into, you know, this risk, right, Tim, the one that's unpredicted, it should be truly surprising when you get to that point. If there's something that you can anticipate ahead of time, that's what you're targeting with automation.

So, the gateway to hell scenario is the risk that Tim brought up. Like, this is on the risk register now.

Well, yeah, I mean, when you look at risk management in general, I think this is one way that security may want to learn from SRE. SRE in general, when it's defining SLOs, tends to do, you know, is doing a better job at looking at things like probability. Steve gave a great talk at SLOConf about probability, how it's calculated. This is part of an SLO. There's a 0.01% chance of this happening over the course of a year. Whereas in security, you still have a lot of teams and organizations stuck using CVSS, something very subjective. The likelihood is 5. You know, there's not really this seriousness of like, over time, how can this happen? Same with financial implications of the impact. Criticality, oh, it feels like a 9 out of 10. What does that mean? It's at least $10 million in legal liability. That is something more mature. So, as security gets more to a mature model of risk, it has an easier way to prioritize, has a more consistent way to do it. then alerting also becomes easier. You know kind of why you need to treat this impact surgically.

That actually makes sense, which brings us to the next point I want to make about the differences. Some of the stuff from the world of reliability and SRE feels like surreally similar, but I also know that there are scenarios where the same words like incident and risk kind of mean different things. And if you just straight lock a SOC analyst and an SRE in a room, without giving many contexts, there's a decent chance they would confuse each other. So how do we approach this part? How do we approach the fact that SREs versus security practitioners approach risk differently? And how do we deal with alerting because of that?

That's a good question.

Don't define risk for the security guy, by the way, because this is like, it's going to cause major mayhem in pretty much like, hey, let's define risk. Then suddenly 6 hours later, people are drunk and some are beat up, bloody, and it's just not good.

But so are we back to talking about what we discussed during our one-on-one yesterday with drunk and beat-up Luddy?

I'm afraid so. So, let's just avoid defining risk from the security side because that's a danger area.

We will not define risk. Don't worry. Yeah. But yeah, at least in terms of incident, right? We've got this kind of definition given to us. It's an event that turns out to have an impact. That's an incident. But when you talk to developers, certainly when I was a software developer, primarily for us, you know, anything above CVSS 7 that we introduced into production, That was an incident for us because it meant we had to stop everything and fix this and roll it out. Now, in that case, you know, you're just talking about the potential exposure. You've introduced a risk and now it's very highly exposed, right? So, for some, that counts as, that counts as one. You do build alerts around that. You know, you alert people when they have this in production. So, I don't know how useful it would be to try to create a one-size-fits-all. definition, but certainly something Steve mentioned earlier, which is taking action on the alert. The 2 key components here are how relevant is the alert, like how accurate is it describing a potential impact? And then also, what action are you expected to take, right? Is it just informational? Can't that just be a ticket or a weekly report? Why do I need to wake up at 3 o'clock AM to get this done? Steve, as far as impact goes though, You've taught me quite a bit about impact over time and how that's calculated and considered. And I think security could actually take some from them.

Yeah. So, the tool that we use that is in the books and everything, it's the part of SLOs that everyone doesn't get to. They read about SLOs and they read the whole thing, and then they skip the last chapter, or they start implementing the basis of SLOs and they forget about this last thing, which is actually the most important part. And that's the error budget. So, there's 2 parts of the error budget, right? One is like, A, be aware of the concept of an error budget. Like, just, it exists, it's a thing. The idea of that is we are going to allow ourselves some amount of failure, and that some amount of failure is not instantaneous, it's over time. And so the kind of like base way of looking at it, they're like, oh, let's measure this on a 24-hour period or something. No, we're talking like a 4-week period, like a pretty long amount of time. So now we have 4 weeks worth of error that we're allowing ourselves. At some rate. So if we're talking about 4 nines, then we're having like 0.01% of error is allowed over 4 weeks. Do the math, you get this curve, you could do some integration, blah, blah, blah, right? So you have this bucket of money, this bucket of error that you're allowed to spend. So you can spend it any way you want. You can accidentally spend it because something bad happened. You can spend it because you're trying something new and fancy and it didn't quite work out, and that's fine, right? But you're keeping track of it. So The important part about this is knowing that it's over time, knowing that the time is actually kind of long. Right. And so now the real question is like, how do we know when to take action? Like, how do we tie this back to alerts? And so what we say is you should have a measurement against your error budget, which is, you know, by definition tied to your SLO. And what we really care about is not the really the height of that error budget. Right. We don't care about how much is in the bank account. We actually care about how fast you're spending it. Oh, right. And so if you can say right now, so this, like, let's get into calculus a little bit. We're talking about like errors, right? And then we derive errors and we get rate of errors. And so now we have our SLO, which is a rate. Now we're going to talk about deriving that thing itself. So what is the rate of the rate? Like, how fast are we spending the stuff?

Right.

So this is like the curvature of, you know, if you remember math, right? So this is basically the second derivative of error.

I'm sorry, listeners. I'm sorry.

It's the best. Math is cool. Everyone loves it. But like, at the end of the day, what really matters is like, we have all this money, all these errors that we're allowed to spend. Are we spending it fast? Are we spending it slow? Right? Are we spending it so slow that by the time we get a new paycheck at the end of the month, it'll be fine? Like, we'll just, you know, it's just kind of dropping and dropping, and then it goes back up and dropping and dropping. So really, you graph this number over time, right? And you say, how fast is it going down? And if it's going down so fast that we can predict In one hour, we will be out of money. If that's true, which is a very easy piece of math that all computers can do, you just say, okay, tell a human. This is literally— it is now actionable. It is like, you need to step in and do a thing. There's no ambiguity about it. We will run out of money in an hour. So you, SRE, get to fix it before that hour, hopefully.

There's a security example of that sort, but I'm kind of struggling to immediately come up with one because it's almost like the nature of the alert is not X happened and it's bad, but it's like something is happening and it's burning our error budget at the speed that would run out our monthly budget in an hour. But what do I tell the human? I tell the human about the thing that's burning the budget or the fact that it's been burned? Like, what do I do? It's almost like— Tim, help me with the security example. You need detection.

I think Aaron has one. Aaron, do you have one?

Yeah. Yes. I was going to translate this into securities, but I'm going to have to mention the R. The R word, unfortunately. So what this really translates to—

The blockchain?

No, what it translates to is risk tolerance, right? And risk acceptance and the entire understanding of risk and how much of it you're willing to, you know, how much you've budgeted for taking place. That may not, though, translate to alerting, right? Because conceivably, you want to know about this stuff a little bit earlier. What we would be describing would be an indicator of breach and how bad that breach is getting over time, right? You do need a way to project and measure that to tell a board, look, ransomware is this bad for us because it could cause this much disruption, right? So you are thinking about these things, but whether that converts to an alert, I'm not sure. What it does allow you to do though is to say, hey, if this thing that seems like it's imminently about to happen does happen, here's, you know, how long and how much it could hurt us, right? So you're still treating more—

Here's the price.

Exactly. You're treating it more proactively though in security, I think.

Yeah. The other terms to be aware of, and Aaron wrote a whole article about this that we should link to, which is—

We'll have it in the show notes, listeners.

The idea of symptoms versus causes, right? So lots of times we talk about if this thing happened, it would be bad. That's a cause, right? If we ran out of inodes on the machine, we would no longer be able to serve. Okay, true. Yeah, but you don't alert on that, right? Like historically we have, we would be like, so in that case, like let's set an alert for at 90% of inode, you know, email the person of the thing. And this is not a good idea. Like it sounds like a good idea. It's a bad idea, right? What we want to do is we want to separate symptoms from causes. So SLOs and error budget, that whole thing I was talking about, that's measuring symptoms. So one symptom is like hard to dispute. It's like, it's definitely there. And it can have many, many, many, many causes, right? Otherwise, if we have a bunch of causes, you can have a cause go off with no symptoms, right? And that's because you have a robust system, which is actually what we want. So if we have a bunch of causes going off and they have no symptoms because we have such a great robust system, what are we doing?

Nothing.

We're burning out the operators, right? We're just making them mad. We're causing suffering. Wait, what?

Wait, wait, wait, wait, wait.

Right, we're going back to the beginning, Anton. It's great. This is podcast gold. This is called a callback.

This is a callback. Great callback. I like that. Say more. Keep going.

So yeah, we want to be able to separate these 2 things, right? So if we have a symptom, the tricky thing about symptoms is they can either be leading indicators or lagging indicators, right? So if you're in like the best possible position, you have a symptom which is a leading indicator of a bigger problem, and then you can measure it and you can say that it's true and you can fix it very quickly and you're great. However, most symptoms are actually lagging indicators. Most symptoms are like, doctor, I have a rash. Like, I'm sorry, you already have the problem, right? Like, we can't prevent you from getting the rash at this point because you already have it. So, I mean, at the end of the day, like, it's important to be able to separate these symptoms versus causes and recognize that some symptoms are signs that damage has already been done, but now it's a matter of kind of pocketing the damage, right? So it's like damage control, right? So how do we mitigate it? How do we prevent it in the future? How do we make sure that we have all of the stuff, blah, blah, blah, right?

I've seen some interesting cases where security professionals hope to identify leading indicators, which end up being terrible indicators for anything that you would ever respond to.

Yeah.

So, you know, it's like when somebody's like, tell me when I'm getting port scanned. It's like, well, if your thing's connected to the internet, I can make the light turn red, but that's dumb and unactionable. So given this kind of framing of leading indicators in security might be really hard, How do we take this SRE thinking about incidents and adapt it to the world of security? Because they feel different. And importantly, in security, you have an adversary. Like in the world of SRE, you don't have a bad guy out there who's trying to break your iNode availability. You don't have a bad guy out there who's stealthily trying to break your iNode availability. And so how do you take this like random occurrence thinking or bad code deployment thinking Which is like not driven by a sentient being and apply it to a world where we're combating quite evil people.

Absolutely. Yeah. This is the difficulty and kind of the point of demarcation.

Oh, it's hard. That makes sense.

Where you start to say like, yeah, security actually does have a bigger challenge here. You brought up something in particular with an adversary who's able to willingly deceive you and intelligently respond to your attempts to stop that, right? This is not A problem that you have with random systems failure. Again, though, what you can learn from SRE is maybe not exactly what to alert on. This is going to be different. It's really the method of prioritization, because what SRE has done is consolidated causes into symptoms, and then those symptoms are consolidated into one key principle, which is user happiness. In this case, availability, latency, all these things affect user happiness. What is it for security? I would say security, if you ask people who got their certification recently, it's the CIA triad. It's confidentiality, integrity, and availability. But what does that support? Because you clearly don't care about all those things for all different types of data. Those only matter in certain situations when it can affect shareholder value, your customers, et cetera. So, there is a key principle in security too. more accurately addressing how your technical system with its metrics affects that principle, affects shareholder value, et cetera, in financial terms, in terms that are discrete and consistent, as opposed to 9 out of 10, 5 out of 10, right? This more subjective way. At the very least, it gives you a better way to prioritize. Okay.

So, I totally agree that prioritization should be driven by the outcomes that security professionals are trying to guard against, not just adherence to blind application of CIA trad. But I'm still struggling to understand how the presence of the adversary changes the picture. Like, I agree that we care about a different set of outcomes, but what does having an active adversary change?

It forces you to consider more conditions in your detection, right? And it also forces you to include more steps in your response. If you have a case where there's a sensitive action that could be legitimate user behavior. You're in a really tough bind there, right? But in terms of the response, in terms of what you do with the information that you have, part of the alert and part of the implication of the alert has to be multiple steps, like checking that the person, you know, the principal identity that's doing this action is actually online by sending a message saying, hey, is this you? If they say yes, checking that it's actually that employee that's logged in by Giving them a USB OTP challenge or something to prove that they are who they say they are. There's more steps involved in the response. It's not just a purely technical response, like, oh, let's jump in and see what's going on.

The thing that I would take from Esri to this is that the generic models that we've talked about with error budgets and alerting and all this kind of stuff, that's the stuff that's really easy to talk about because it applies in many situations. But the actual details of how do we fix the problem, around this one incident, it's always unique. It's unique to the environment and it's unique to the architecture and it's unique to the, you know, the date time, like the moment in which the event happened because of we made changes last week that we're not going to make next week, right? You know, so there's all sorts of, you know, on the macro scale, things look like they follow a really simple pattern and all you need is like an SLO and an error budget and some, do some calculus or whatever. But like in the micro level, every single time is different. We have a phrase in SRE, we say, may all your incidents be novel, because we want to have the case where every time we solve a problem, we prevent future occurrences of it. That's kind of an ideal, never really happens.

Or build an automated process to respond, either prevent or automate the response, presumably. You don't just mean prevent.

So the thing that I would bring from SRE to this situation where you have these persistent threats and you have these deep actors and these many-step programs, each time you do that, you have to bring software engineering in and you have to have someone like the model that we use that's simplified, but you get the idea is like every time you have an operator who knows how the system breaks and just feels it in their bones, they know what's going to happen next. Make a software engineer sit next to them the entire time and just take notes, right?

Mm-hmm.

And then that system administrator or that operator might not be like into building software or even aware of what's really automatable and what isn't, but they know this problem in their core. And having the software engineer sit next to them and like take notes and come up with ideas and, and then say like, hey, what if we built this thing that made it so the customer couldn't do this and they had to do this instead and blah, blah, blah, blah, blah. And the sysadmin or the, in this case, like the security engineer would be like, you can do that. You know, like often there's just like, it's purely like, this is just like product engineering, like where you go and you sit with a customer and you say, hey customer, what are your problems? You know, instead of me just giving you something that I think you need, I should go sit in your room and understand your problems and and watch you suffer, and then be like, ah, I can fix that one. I can fix that one. I can fix that one. This is just software engineering, right?

Yeah. And the key problem with an adversary is that, unlike a random systems failure where it happens once, and you fix it and move on, you will have multiple attempts and iterations, and they'll keep trying something new. There's this problem of persistence.

And the degree of persistence may vary. Some would be, like, really persistent. I think that makes sense. Right.

Yeah.

But I think what's interesting about that, in my mind, is that it points to the fact that in security, we have something that's kind of different, which is the opportunity for a defender or a vendor of defense to gain leveraged protection across their fleet of customers. And so, like in SRE, everybody's stuff is going to fail differently. But if there's a limited number of actors out there, in theory, as a defender, you could see them in one place and defend against them in other places.

Right.

And so, as much as people say, oh, Tim, what's so hard about security? And I explain, well, we're securing systems and the intersection of systems that we're not responsible for. And by the way, there's a bear chasing us. The nice news is if we see the bear try to climb one tree, we now know one of the ways the bear likes to climb trees, and that's generalizable whether we're defending, say, these trees or those trees. So I do think there's actually kind of a really interesting difference there on the degree to which lessons are transportable as a vendor of defense.

So are there lessons, other areas where the lessons don't apply? Because when I started thinking about which SRE alerting lessons may not apply to security, my mind went down to the stealthy threats that quietly steal data. You don't know about this, and you feel the damage to your business in 5 years. But that's not really a scenario that's in your charter necessarily. That's in your boss's boss's charter, perhaps. So, are there any other pointers or scenarios where somebody who is reading the SRE book or talking to you guys or talking to somebody else wants to apply the lessons for security, whether he should not? He should really do things differently. Are there— what's the diff?

I think there's a hard rule of thumb that you can follow when looking at SRE versus security. SRE is about risk mitigation, right? But importantly, when we talk about what type of risk, it's usually opportunity risk, growth risk, right? In order to expand my services, to treat more customers, in order to grow globally, in order to keep things running, I naturally We'll run into some risks. There's an upside to these risks though, right? That's why I'm investing against them. With security, there's something more mysterious about it. And if we pick into why, it's because security deals with pure risk. There is no upside to these bad things happening to you, right?

This is a profound thought of the day, actually. I sort of knew that, and I think I've used it in the past, but it's really cool how you structured it is that the There's some business risk has upside, downside. Security risk has no upside. I wonder what my boss's boss thinks about it, though. I have a suspicion.

I don't know if I agree with that. I think there are cases in security where choosing to have less protection does have upside.

Oh, no.

Yes.

Not deploying an agent, you get 5% of your compute budget back.

Sure.

Not requiring MFA, you get 20% lower shopping cart abandonment.

Excellent point. Yeah, there are certainly these trade-offs. What I mean is the risk that it's meant to treat. the pure risk presented by the threat actor. That is pure risk. But all the other things you mentioned, and then some more, right? Security has this unfortunate, you know, risk itself of creating, say, project or product risk. It can slow down a team's ability to release something, which entails competitive risk. You know, security can— Those are just a few. Security touches a lot, right? Yeah. Security touches a lot of other parts of the organization that You know, it needs to be careful of how it interacts with them.

Reliability tends to be temporal, I think, is what you're getting at, Aaron, where it's like, it's, you know, it's down for this hour that it's down, and we lost money during that hour, or we lost opportunities during that hour. But like, IP theft is forever, right? Like, you know, someone walks away with your designs, they're gone. They're not gone for that hour during the incident, they're gone gone.

This is such a nice depressing take. I think we should give him a security professional card. He would be an honorary security professional because Steve has this great depressing take, which I think makes him one of us, right?

Right.

One of us. One of us.

Cynicism is one of my best assets.

No, we're not joining your cult.

So unfortunately, we are just about out of time. And so it is my lucky duty to ask our traditional closing questions. First to you, Steve. Do you have one tip to help security professionals improve their outcomes? And 2, do you have recommended reading? And that recommended reading could be Anything except Anton's blog.

So I would say a tip for kind of anyone to improve their outcomes, just write down what it is that you want to do and have other people look at it and just kind of like start off kind of vague, get a little more detailed, get people to say like, yeah, good idea, right? Get some wind behind your back or get people to be like, no, dude, we already tried that. Look over here. If they don't have the thing written down, don't believe them. But like, if someone can present to you with like, here's what we tried, here's what we wrote down, here's why it didn't work. Then that makes a lot more sense. So basically, write more docs. That's my TL;DR.

Write more docs. I like that.

For reading, I would say it's kind of a category of work, but there's kind of a new side of reliability. We've talked about postmortems in the past when it comes to writing down the stuff that went bad and everything. There's this kind of new way of looking at it called learning from incidents. So shorthand is just LFI. I think there's learningfromincidents.org. .io, maybe, something. We'll put it in the show notes. They've got a lot of stuff. And so these are folks who've been talking about like airplane crashes and fires and like people dying from like nuclear whatevers. And like, so it's like real risk, you know, like in the real world, like atoms, not bits. Yeah. So we can learn a lot of things from them and apply it to our stuff. So that's a whole— like you can get a PhD in that. It's a whole big thing.

And, you know, I love that answer because one of the other people on the show who really changed my perspective was Tim when he was talking about safety versus security and the relative success of safety people compared to security people. And the fact that we don't die in airplanes very often actually comes down to a lot of learning from the times that we did die in airplanes. So that, I think, is really great advice.

So the critical thing to point out about that community is it's called learning from incidents, not learning about incidents. So we're not actually caring about the incident itself. We're using the incident as a lens into the system.

Oh, that's interesting. That's, that's really interesting.

That's really interesting. I mean, it's really interesting.

Like, that's all I can say. It's kind of interesting. That bit alone is worth the price of this episode, I think.

Yeah, I agree. Listeners, the best bit came at 34 minutes in. You better listen to the whole thing.

Hard one to follow. I will say though, my takeaways are if there's anything that's subjective or fuzzy, that is definitely going to aggravate having bad alerting. So Prioritizing by measuring everything, measuring everything. And that will entail talking to other stakeholders outside of security to understand impact. That'll entail talking to engineering, legal, finance, a lot of different stakeholders. And on that note, books I'd recommend, one is How to Measure Anything in Cybersecurity Risk. It gives you a good introduction to the FAIR method and, you know, other ways of calculating risk and probability. And the other is Security Chaos Engineering by Kelly Shortridge with Aaron Reinhardt.

That one is epic. We'll have Kelly soon.

Awesome. Oh, I can't wait.

Now listeners, not to give away our guest list, but we are excited to have Kelly join us.

We hope. Awesome. Awesome. Do not miss that.

Awesome. Well, hey, great recommendations, great tips, and a lively, interesting discussion about SRE and security. Steve, Aaron, thank you so much for joining us today.

You got it. Thanks for having me.

Cheers. Pleasure. And now we are at time. Thank you very much for listening and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter @anton_chuvakin and _timpeagle. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.