EP174 How to Measure and Improve Your Cloud Incident Response Readiness: A New Framework

Security running. Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior PM for Risk Analytics and Cloud Threat in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website. cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners all on our community page. Anton, we've got a super fun topic today. It's about preparedness for incident response. Is that right?

Indeed.

It's one of those IR episodes that are not about IR at all. It's about what happens before the incident because ultimately, as many, you know, books, guides, papers about IR indicate, Most of the important stuff you need to do for incident response happens before the incident, not during incidents.

So this is real like prepper mentality, but we're not moving to a bunker in Colorado just yet.

That's correct. And also for a global company, the bunker in Colorado would not quite cut it because we have like—

That's right.

Too many locations. And you also can't move cloud in a bunker, can you?

I've talked to some users who would like to, but maybe with that, listeners, let's turn things over to today's guest. I'm delighted to introduce today's guest, although I'm not going to try to mess up her last name. We have Angelica R., who is a senior program manager in cybersecurity response at Alphabet. Angelica, thank you so much for joining us. Cyber and IR, we have this interesting problem that incident response is inherently reactive. We're responding to an incident that's happened, but it's preparedness that determines our success. So, How do we think about preparedness? What's the scope of this problem? How do we prepare for something that you almost certainly can't really be prepared for?

Well, hello, Tim. Glad to be here. And you're starting off with a tricky question. Literally every incident responder knows that it's very hard to be prepared, but our goal is to always be ready to respond to any incident that comes in. So we look at it at Multiple levels, but at the core of it, it's meticulous preparation and planning that needs to be done to be ready to respond. Plus, you need to always have coffee and keep calm.

Ooh, okay, so coffee is important. That makes sense. What, what goes into the planning? What's the planning look like that's so important?

For incident response, you need to have the right team, the right tools, you need to have the right documentation and playbooks, and you need to be able to trust your team 100%.

And if I interject with something, I think that with IR, apart from burnout and some other risks, the interesting bit that comes up is the primacy of people. It's like, I don't want to go with security clichés, people process tech, but it's interesting how in SOC and in IR even more so, the primacy of people just comes up to the front. It's not about, well, but I have this forensic tool and you have this other forensic tool. It ends up being not about that. It is interesting that it's coffee, trust, and just goes way high on the list compared to the rest of the stuff.

Yes, that's very true. At the end of the day, we are all people. We try to do the right thing. We try to solve a problem and we try to do good. So we all have the same goal and everyone brings something to the table. I might have coordination skills. Someone else knows some pieces of information that's useful. So it's always good to have tribe knowledge and know whom to ask. Because at the end of the day, me as an incident responder, I'm not here to give you the answer. I'm here to find you the answer. So, I'm always telling everyone, I'm the most important router you will ever have.

Isn't there a classic quote from the early days of Google? You know, this is like totally on a tangent, but at one point, I bought a book, How Google Works, which was written, I think, 15 years ago or something. Don't— Tim is making the funny faces at me. The point is that, yes, fine, no comment. The point is that I think it had the router quote in that book. That's why I'm remembering this, because it was one of the early PM leads or something said something about, but you're just a router. And he says, yeah, that's kind of the point. I'm just a router and this is huge.

I think a lot of product managers feel that way.

But do they feel proud for being routers or do they feel sort of embarrassed for being routers? That's important. I think it's the pride. matters more than routing quality, I guess. Maybe. Okay, fine. Forget it. Forget I said that. Back to IR.

No, I think this is a really interesting line of discussion, though, Angelica. Aside from what Anton was saying, the part about it being a team sport, you know, we always say security is a team sport, but it sounds like here, incident response, it's really a team sport. And listeners, this really ties in, I think, to the conversation we had actually in the same room not too long ago with Robin about expeditionary behavior and how you should work as a team and what leads to that. So before we get into the next thing, can I ask a spontaneous question of how do you foster that team spirit and healthy team dynamics before an incident? What goes into making that successful?

So I think for incident response, one important thing is that we all want to do the right thing, that we're all driven by, we want to help. That's the first thing. The second thing is that we don't have revenue or metric we can kind of like show our value. Because I can never tell you how much fraud I prevented or how much losses I saved. So we don't have that pressure. We have different types of pressure to deal with. And I think that makes the difference because, again, I said I'm the most expensive router. And yeah, it might be fun. It might not be the nicest thing. But at the end of the day, it gives me a clear expectation. I do need to know the answer, and I need to find someone who knows the answer, but that makes it easier in general for everyone. And I think that, that also helps them with the team spirit because everyone knows what they're expected to be doing, and everyone knows you need to pull together in order to bring the case to a close. And you always celebrate as a team as well. There is no— which is really nice.

That's really interesting that you don't have a revenue metric or a losses avoided metric. Yeah, I hadn't thought about that aspect of your role.

Yeah, but how will you put that together? I cannot tell you I saved X amount of money because I'm here.

Of course.

And I think that this probably adds to the sanity of the team doing it. I think that's my impression because, I mean, I don't want to go on another tangent, but I think that sometimes the obsession with— if people demand the loss amount saved, the pressure to invent it is very strong. And I've seen it by security leaders where if they have to present money, they present money, but there's no methodology behind it. Oh, by the way, methodology. We are here in this room, in 3 rooms, because you created a framework which I felt is very cool for measuring how ready you are for an incident. So, can you tell us more, well, us and the listeners, about the approach you've taken and just like principles, ideas for the framework?

It all started a couple of years ago when my manager asked us, how do we actually know we are always ready to respond? And that was a very, very, very good question because we know we are ready during an incident, we can do postmortems, but we never know are we actually prepared? So, when you think of it as you're a fire commander, you know you can put out the fire. And while the fire is happening, you also know what you have to do. But the prep work, the underlying, do you have the right people? Do you have the right clothes? Do you have the right hose? How do you measure all of these things? Like, do you have enough air in your tires? You know, these are all pre-checks.

Mm-hmm.

But because we are in a very fast-paced environment, we often forget about the underlying infrastructure that needs to be in place and needs to be there. And you need to know how to use it as well. So that was the starting point. And from there we came up, okay, how can we slice and dice incident response? And we came up with— we name it verticals and horizontals.

Mm-hmm.

The verticals are the different process areas and tools. It's like playbooks, partnerships, or the general tooling. And then we have our horizontals. We do have buckets of responses, like we have a specific phishing response, a specific malware response, so we can measure each response or investigate how well we are doing for each response in terms of tooling, in terms of partnerships, in terms of intake. And that way We can identify very, very quickly where we are very good and where we are lacking.

So, it's interesting that my reaction to reading the paper and also to hearing you talk about it now had the same, maybe not hype cycle curve, but I'm trying to make a graph maybe in my hand. It's like I was reading it and my impression was like, oh, that's pretty obvious. And it's like, oh, wait a second. Like I had this, it's like, this is interesting. And I think that Not that I tried to psychoanalyze my reaction on the air, but it's interesting how it started from like, yeah, sure. Well, if you have a fire truck, you need to have air in your wheels and you have to have water in the tank and blah, blah, blah. And it's like, oh, I guess that if you present it that way, you have a systematic way to measure it as opposed to just a checklist or just an intuitive way to say, oh yeah, I'm ready. It's like, well, what do you mean by that? And suddenly there are pillars. Am I ready for ransomware but not for—

Yeah.

for a fire in a data center, to really mix metaphors here.

Those metaphors maybe shouldn't be mixed. I don't want to think about data center fires. I heard— actually, I heard about some of what went into the aftermath of the fire in the Paris region. And I was shocked by some of the things that we weren't ready for as contingencies in that case. And the efforts that Googlers, including Googlers involved in the Security Voices Project, by the way, were doing on the fly as part of the response there. It turns out we weren't ready for some of that, and hearing about non-readiness was quite interesting. So maybe speaking of that, one of the important things in incident preparedness is, you know, continuous improvement. And so how are we doing that in cloud? What does that look like today?

First of all, we measure the baseline, right? Like, where do we stand? What types of incidents do we get? What types of tooling do we need? What types of tooling do we have? Do we know our partnership teams? Do we have strong relationships with our partners? Do we have access to the right data? Do we have the right playbooks in place? And then also, how do we get informed about these types of incidents that need to be reported and escalated to us?

Now, that's an interesting one. Are there cases where you worry about bad things happening that you don't know about? I would have assumed that Google was really well instrumented for Knowing about bad things and getting them to the right places?

Well, there's different methods of detection, right? Some things we can detect, others customers or internal people need to report to us in order to be aware of it. So it really depends on like which type of incident and how well everyone is educated on how to escalate cases.

Well, that makes sense.

So wait, wait, wait, I want to stick my nose in this question a little bit and say, well, What if somebody from a less enlightened organization, aka not us, comes up and says, well, I have a consultant draft the playbooks this year and I'm going to use them. And I would just like, I'm going to pay for the best damn playbooks for IR the money can buy. But I would then stick to them over time rather than run a tight improvement loop. What is your advice to them? Like, why are they wrong specifically?

Because we're living in a fast-paced environment and things change so much, especially in cloud. So for us, with cloud, the most important piece is established partnerships and working with people to know about all of the new features, about the new developments, about new collaborations. Because whenever something changes, we need to adapt. We need to adapt our tools, we need to adapt our processes. So Having a really nice playbook, but you never audit it, or you never check if your escalation points are still up to date, or if certain tools are still working, that will not work because you will have a downfall somewhere, or a pity fall.

Yeah. So how do you make sure it's all working? Do you test this stuff regularly? Do you simulate stuff regularly?

Yes, we do have exercises, internal exercises, and we also have at least once a year a full audit. But also, whenever we use a playbook and in an incident, we do postmortems. So we check afterwards, was everything in place? And then on top of that, with the CI framework, we are actually double-checking where is there additional holes or gaps we were not aware of and where can we improve even further. So having postmortems in place and reactive reviews is nice, but we could never quantify our efforts. So we would launch a new tool or we would launch a new playbook, but The person making the changes could never quantify the impact. What does this change in the playbook mean for the next incident? With the CI framework, I can now put a measurement to it by, like, establishing relationships, having the right points of contact, knowing whom to talk to. Incidents become faster to resolve. So, we can measure that, or we have less pitfalls, we have more tooling, so it becomes all measurable. And it doesn't only serve the team. In the sense of I'm more ready, but also the individual because they can quantify the impact.

That's really cool that you're able to put those metrics in place for something that, you know, I think of metrics as useful for things that happen all the time or high frequency. I would hope that the number of times you're using these playbooks is not high. And so, it's cool that you're able to get meaningful metrics out of that. That's really great.

So, I got a strange question connected to it, but Wouldn't you be kind of tempted to focus on the easy metrics? Because I can see how people obsess about time to close. And when we had an episode with our detection leads, they kind of reminded us that the narrow focus on time to close, time to triage, time to blah leads you astray in most cases. So how do you resist the temptation to just measure things that are easy and instead measure things that are needed? That's a hard question, I think.

I think it's in our culture to try to be the best and to investigate and not to give up until you have the answers and be the best you can be. So I think it's actually easy because you always want to improve, you always want to be better, you always want to have this additional advantage because the industry is changing so fast. So you always want to be a couple of steps ahead, and that makes it easy.

It's easy, really?

Well, it makes it easy to drive the changes. I'm not saying that implementation is easy or that—

Ah, yeah, that's fair. Okay. So—

Developing new things, thinking about new things, coming up with new ideas, having brainstormings, that's easy. The implementation and execution is a different ballgame.

Yes, yes. They say ideas are cheap and execution is hard. I want to take a step back and zoom out a little. When we look at Google and incident response preparedness overall, What do you think we're particularly good at, and what do you think others should learn from how we do it here?

By the way, sorry for interruption. Tim is asking this question without any hubris, and this is important because Tim has this magic. Like, instead of saying, everybody knows we're the best in this area, what are the lessons we can teach others? That's how I would ask it. But Tim asked it in a way more subtle way, and I love it.

You know, Anton, there's a lot of people in my childhood education who would be happy to hear you say Tim asked this without hubris.

I just need to look at my day-to-day. I mean, incident response for Alphabet, right? I'm dealing with every product in Google. I'm dealing with every kind of like infrastructure there is within our company. So having that amount of different types of incidents, having to work globally, having to work with people from at least 107 different nations.

Wow.

Yeah. In Zurich alone, we have 107 different nationalities.

No way. There's 107 different nationalities represented in the Zurich office? That's incredible. Yeah.

Could be more now.

Wow.

Yeah.

Okay. But like, listeners, I want to illustrate a point that Angelica is making here. Like, Alphabet incident response ranges from Waymo cars catching on fire to data centers catching on fire to undersea cables breaking because ships sink and land on them. What are some other crazy incidents that we have to deal with at Alphabet? Like, what else goes into that list?

Oh man. I try to think what I can mention.

Yeah.

One time we had ponies and reindeers, so they need to be herded as well. You know, they were at an offsite. So someone— yeah, back in the days.

Pony-related incident. So is there a category for pony-related incident?

Did you have a playbook for that, or did you have to adapt the horse playbook to deal with the ponies? You had to get a ticket for it. Well, okay. I hate to rein us back in on the, on the horse topic and saddle us with some serious questions here, but what advice would you give to other organizations looking to implement a CI framework? Like, where should people start with this? How do other people get the Google magic going when it comes to improving their incident response?

When you read the paper, at the bottom, I left a couple of templates, which I started out with, and they are fairly simple, hopefully. You fill them out, and initially you come up with your vertical and your horizontal. Your vertical is basically which types of processes do you want to measure, and what is your categories of responses. And you can always categorize them. So I started out with, I think, 174 different playbooks and response categories because I get told every incident is different.

Mm-hmm.

But you manage to find a common nominative. So you just start kind of like collapsing.

Hmm.

And the core is the important part. So not every phishing response is the same, but I know the goal of the phishing response is to ensure that, like, we know what actually happened, close the— stop the phishing email coming in, or protecting customers. So the goal is clear, and that's where you start out from. And then when you start understanding the concept of verticals and horizontals, everything else becomes easy because you know what you want to measure and which pieces you want to measure. And then you can apply an initial— we use CMM for it, very simple, 5-tier bucket— how well you're doing in each of those, how well each of those areas is developed. If you have that initial structure, you can expand. Going further and further, but being clear on the initial structure of, like, which processes you want to measure and which areas you want to measure, that's the main part. Once you understood that, everything else will fall into place.

I really like the end of that, of the understand what you want to measure and things fall into place. I hate to do this because this has been fun, and how often do we get to talk about ponies on the show? I have to ask our traditional closing questions. One, Do you have recommended reading for people? And 2, do you have a single piece of advice to help people improve their incident response readiness?

Well, of course you should read my white paper.

Okay. That was the softball question of the episode, listeners.

And in order to be ready to respond, trust people because people always want to do the best.

It's a good answer. I really like that. Angelika, thank you so much for joining us today.

Thank you for having me.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter, @Anton_Chuvakin and @SaraKovacs. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.