EP191 Why Aren't More Defenders Winning? Defender's Advantage and How to Gain it!

Secure running. Hi there.

Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior PM for Risk Analytics and Cloud Threat in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us, and the rest of the Cloud Security Podcast listeners on our Google Cloud Security Community page. Without any further ado, let's turn things over to today's guest.

Hello, and the guest today, Dan Nutting, Manager of Cyber Defense for Google Cloud. Welcome to the show.

Hey, thank you, Anton. Good to be here.

So Defender's Advantage, uh, I remember it as a paper book that Mandiant used to distribute on the shows, but I think we evolved and we now have digital copy, papers, but the book is still alive and well, right?

Oh, absolutely. I mean, I keep a printed version on my desk all the time because I reference it quite a bit, but it's digital. It's alive.

Yes. So, can you tell me a bit more of the story, like how it happened? I mean, this is the second edition, so there are probably some fun changes. So, just give me a context on the book and on the changes to the book.

Absolutely. So a lot of this was born out of working with big clients who want to do the right thing by security, and they're hiring the right people, but those people get so focused on what they do well, they don't necessarily interoperate very, very well.

Hmm.

One of the things we saw, for example, with several of our clients is they would build out a dedicated detection engineering team, And they would have a dedicated intel team, and those 2 teams wouldn't talk.

Hmm.

So you'd end up having the detection engineers running their own intelligence so that they could figure out what new detections to build because the intel products that the intel team was producing wasn't productive for them and they weren't communicating. Moreover, a lot of that wasn't going back to the response team so they could update their playbooks and be proactive, et cetera, et cetera. So we saw this behavior and we said, all right, we need to come up with a framework in order to Identify what those critical services are, and then how do they work together? How do they communicate? Who's overseeing those communications?

But the book was the— not just the book in this case, but the framework for doing this together, right? Like, ultimately, to me, it's kind of a framework.

Exactly.

The answer, not you do a good job here, I do a good job here, but we don't work together. So the company overall does a bad job, right? That's probably what we've seen, right?

Exactly. Yeah, exactly. And one of the things we realized as we were putting this framework together is, you know, the defender actually does have an advantage because if you are intending to be proactive, then you can manipulate the attack surface. You can manipulate the environment that the threat is going to be operating in. You have that control, and a lot of organizations just don't do that.

So this is probably my next and admittedly somewhat painful question about this, because while the concept is very clear to me, it's my land, I set it up so that the attacker— if the attacker shows up or when the attacker shows up, I can detect better. I can defend better. So, philosophically, it's very clear that the defenders have an advantage. But practically, we still see an occasional attacker who literally knows the land better than defenders. In my PCI DSS land, I had people who stole card numbers from a dev environment because they knew that the dev has live numbers in violation of PCI.

Right.

But the defenders assumed that the live numbers are only in prod, and the dev doesn't have them. But the attackers show up and they have still the numbers and go. So to me, that was like classic, admittedly dated example of defenders knew the environment worse than the attacker. So why do we suck?

I mean, well, I think you had that, you captured it with the word assumption. So defenders are assuming that they know the environment and the attackers are assuming they don't. And attackers have that perspective that where they're constantly questioning the results that they are getting from their tools and their scans. And defenders are presuming that what was true a month ago is still true today. And that's something that really has to shift. Assume you're wrong. Assume you need to reassess what that environment is. Are you— did you actually patch those vulnerabilities? Did you actually deploy those detections? Do you really have logs coming from those critical servers? Assume you don't. And validate continuously.

So that means that the attacker assumes they have to discover to know, but the defenders assume that the network diagram from 1998 is still correct.

Bingo.

Like, that's the cynical example, admittedly. But I mean, we've seen people with IMDb being out of date by decades. Years, sure, but also decades occasionally.

That's exactly it. I've had conversations with clients where we ask, hey, do you have this technology? And they immediately say no. And I say, are you sure? Because I see evidence in the SIEM that you do. Oh, and it's from some latent acquisition that happened years ago that never got fully brought into the ecosystem and the updated security policies, et cetera, et cetera.

So, this is another lesson. I mean, admittedly, it's maybe, you know, a tad boring, know your environment, but it's also something that a reminder is very useful because people don't know the environment, and people who think they know their environment still don't. So, it's almost like the second layer behind know your environment.

Exactly.

Kind of like assuming less. One more thing I've picked in the book, in the second edition especially, is of course the whole intelligence-led approach to cyber defense. And I've met enough, and I've kind of ranted enough about how we are not, you know, threat-centric very often, how we are trying to rely on kind of like flat controls, the controls that aren't very threat-aware. So why, in theory, security people are supposed to be about threats? But we very clearly don't do threats when we plan defenses in many cases. So, what's the story and how do we fix it? How do we become more threat aware without drowning in the stupid bad IP lists?

So, I think a lot of this goes back to that siloization that the specialties have. So, for example, Log4j, that hit everybody, right? Everybody had some instance of that. It was pursued as a vulnerability problem rather than a threat problem. So a lot of the defenders weren't proactively building detections. They weren't proactively preparing hunts. They weren't setting up playbooks on how do they contain and eradicate from these sensitive critical web applications. They were just letting their vulnerability management team patch the Log4j. But the intelligence was saying, If the attacker is not already here, they're coming. You should prepare. And if you haven't done the effort to hunt, Mm-hmm. you don't know that they didn't already show up. And we saw this again with MOVEit. We've seen this over and over and over with all of these large zero-day vulnerabilities that everybody has, which means that everybody within that defense organization has a role to play. And that intelligence-led is that intelligence is seeing that the threat actor is coming. We've got instances of the TTPs. We know what they're going to do. So let's prepare, again, that defender's advantage right now, knowing they're showing up.

I mean, treating Log4j as a just patch faster problem would produce just hilariously bad results.

Right.

It's sort of like, first thing is you won't patch faster. And Obviously, you won't just patch faster because this word just is always the trigger word for me because it's like, oh, just patch faster. Well, like, I didn't spend, whoever says, 30 years in IT trying to patch faster, and now you show up and say, hey, I'm a consultant, just patch faster. It's like, we sort of know it, we can't do it. And I think the intelligence-led approach to, say, Log4j would have saved them quite a bit.

Right, right.

And any practical steps? I mean, I know this is kind of very fun and we're enjoying ourselves and hopefully the audience would too. How do you start that journey to becoming more intelligence-led in what you do? Not just have an intel team, have them go at it.

So I think the first step is teaching the intel team what the different teams actually need from them, and also teaching the other teams how do they actually generate requests for intelligence. There's a lot of organizations out there that their version of intelligence is putting together a list of news articles that happened between last night and this morning for their CISO to read.

Oh, depressing.

That's not productive. That's not actionable. Whereas the detection team, they need sample command line arguments that they can roll into their SIEM instances or their EDR instances to build alerts from. The response team needs to know what kind of privileges or what kind of data was searched for so they can build investigation queries and pre-built dashboards to aid with those investigations. And they need to know what type of critical platforms may be targeted so they can work with those application teams to say, hey, we need to isolate this box. What's the business impact? What's that going to look like? Let's prepare for that today. An organization I worked with a while ago, we were working on mass password reset and recognizing this one critical application had 40, 50 different accounts and like, it's going to take us 3 months to change all these passwords.

Yep.

Well, let's figure out that process today. Let's fix it today.

That part is hard. It's hard. Yeah, it's easy to conceptualize, but it's hard to do, right? Like, that's right. That's the vibe I'm getting here is that that's like, sure, everybody agrees, but like, how do I grow up to be that?

So that intel needs to be actionable. It needs to recommend actions and preparations for each of the other core teams within Defender's Advantage.

I think that's useful advice as long as people can turn it into the operational reality in their environments. And yes, of course, I can put my vendor hat on and say, hey, hire Mandiant, they would help you. But even this being kind of an obvious thing to say, there is a maturity growth to that. It isn't a secret trick. You just do this and you have it. It's kind of like The journey. So now we're going to go to a dangerously fun topic of detection engineering, and I think that you warned me that it's a topic we can only spend maybe at least 6 hours on, so let's try to condense it. So one thing is that we teach people how to do detection engineering. Occasionally there is resistance. Some people say they don't want to engineer, but ultimately we do realize that detection engineering lifecycle have to run pretty quick because attackers change. You know, some attackers maybe change less, but How do you do detection engineering in the face of this rapidly changing attacker? How do you ensure that you don't just build the DE team, but the team survives the test of time? Like, that's maybe the juicy question here.

Yeah.

So there's multiple components of that. Let me attack that in a couple different ways. First is the test of time aspect. So a lot of organizations will build a detection, let it go, and they'll never revisit it. They don't make sure the detection is still working. They don't make sure it's still targeting the right things. They don't make sure that the attackers are even still using those TTPs. And it's easy to fall, for example, within MITRE ATT&CK to say, yes, we've got this technique covered, but there's 5, 10 sub-techniques. And even within the sub-techniques, there's different permutations of how to realize those techniques.

Mm-hmm.

And the detections may be looking for just the existence of a process name or a process hash when a new tool may have come out that does that same technique. So that needs to be regularly revisited and tested and validated, and the dependencies need to be tracked. And then it goes back to, are your servers even logging the right signals so that those detections can activate it? And don't assume what was true last year, last month, is still true today. Validate again and again and again.

Yeah.

On the other side, with detection engineering, I understand why a lot of organizations say, hey, leave it to the vendor. The vendor can do this, right? EDR can do this, and they're going to do it better and faster than— and more accurately than we can. Yes, but the vendor, in its need to be precise, in its need to not be overloading their customers with false positives, is going to be a little slow, and they're going to make sure that their detections work for Yes. You have the opportunity for your detection engineers to actually target your organization, build baselines for your organization. And then lastly, the area that nobody's touching, your web applications, your layer 7 stuff. The web application firewall is hitting generic threats. It's not actually targeting the business logic threats, those evil user stories. And this is where your detection engineers need to work with your developers. How is a malicious user going to steal from the organization by taking advantage of legitimate authentications, legitimate logins, and they're just abusing the way the application works.

This is dramatically hard for a lot of orgs. I mean, this part is kind of the scary underbelly of detection engineering. I don't know, I just made it up, where it isn't just about, hey, I'm going to lightly tune the detections you send me, but it's like, sorry, buddy, I'm a vendor, and I can't help you with this because I don't know what app you wrote before your morning coffee. I don't know the threat model. I don't know what you loaded up. I don't know if you use PHP version from 2004. Like, it's work that you have to do, or somebody you would hire, a consultant or a full-time person, would need to do. A vendor cannot build detections for unique stuff. And there's a lot more unique stuff in the cloud.

Exactly. And the cloud in particular, cloud detection and response, that missing link. We've got EDR, NDR, XDR even. CDR is still new. It's still infantile.

Yeah.

You're going to have to bolster where those gaps are because the industry just hasn't quite finessed it quite yet.

Yes.

And I think this part, probably this part, detection engineering, is probably worth the price of a podcast on its own because this, to me, is some of the hard-won wisdom that's super hard to summarize. I still want to have an addendum to this question because I still occasionally hit a complete wall in my conversations when people say, I don't want to hear the word engineering. I don't want to engineer. I want to pay somebody and the magic happens. I'm paying you the EDR vendor, big bucks, you detect my threats. And you and me can argue with them if we sit in their conference room, we drink coffee, and you say, hey guys, you bring up an example about an app. But is there some kind of a middle ground, maybe not so offensive, like a magic put-down?

Well, so I think organizations like ours need to recognize that not every organization has the luxury of hiring Literally the best in the world.

Best of the best, yes.

So what is that middle ground? What's appropriate for them to target the EDR? And I would say, you know what, leverage your EDR aggressively. Hold them accountable as well. Do you have some sort of validation mechanism, a way to do a breach attack simulation, some sort of security validation that actually runs realistic attacks? And then go to your vendor and say, your tool did not detect this.

Yes.

How do we shore this up?

So, it's like an informed complaining approach. Yes, yes, yes. Yes. Yes, because, I mean, false positives are a lot easier to complain about because people say, oh, you're flooding me with alerts, please stop. And of course, the vendor would, well, then just stop. But you don't really want that. You want them to detect correctly and without noise, but also not miss.

And it's a dance that you have to blend between the false positive and the false negative. And no organization is reliably looking at their false negatives, but those are arguably the more dangerous ones.

Yes, I think that part is definitely a big part of the fun. So, fine, we managed not to talk for 6 hours about the DE. So, the one thing that also caught my attention in the book is this whole thing called mission control. And it's almost like when you look at the diagrams in the book, all of them make sense— detection, hunting, response— but then there is mission control. And that part is probably going to be most mysterious to many people in the audience. So, can you Clarify what it is and why it matters.

Absolutely. So Mission Control was particularly born out of— and I had described where those silos were not talking to one another, and it's important to pull those insights out of the detection engineers, out of the intel team, out of the response team, and so on, and force those to be acknowledged and be actioned by those other teams. So you have the Mission Control that's looking at the big picture And saying, all right, we've got this inbound problem even before it's a problem. Here's all of the things that we need to do. So a lot of organizations are practiced with an incident response, that right of boom, all right, we've got a problem, let's marshal all our resources.

Mm-hmm.

And all that ends up really being is a very urgent project that was unfunded that you're having to slam resources into to resolve this problem. Well, just bring that same approach, more structured, more organized, left of boom. Your mission control is effectively your PM in that regard. They're managing the threat project, and they're taking these resources from these different critical functions to address that.

Hmm. So, it's not the silo breaker team, it's also almost like a silo commander team, if it makes sense. So, it's the— maybe the concept of glue here is not Super, you know, matching the reality, but it's kind of almost like it makes me think of incident commander role when you don't necessarily get a shovel and start shoveling evidence, but you have the role of connecting the dots between orgs, teams, functions. Yeah, makes sense.

For the listeners that have experience with the government's incident command structure, ICS, a lot of it's structured just like that, that incident commander, somebody who's aware of all those moving parts and knows how to move those resources back and forth. Now, some organizations may not need a dedicated team to act as mission control. They may be small, they may be agile, or they may not necessarily have the silo problem so that that role can be distributed across those other critical functions.

Hmm.

So incident commander, I mean, I realize that it's the sort of like a magical role when if you look at the, I don't know, SANS 7 steps or 6 stages of IR, it's almost like it's an elephant in the room. that's not in the picture because there is no one person who kind of like makes sure lessons are learned, makes sure Team X versus Team Y. To me, this is interesting how even in a SOC it's missed, and that ruins intel experience, it ruins the flow of intel, ruins the flow of lessons from incidents, and pretty much ruins everything in the office if you don't have it.

And a lot of that's because we expect these forensics experts or these reverse engineer experts to also be really good at communicating with executives. And more often than not, that's a unicorn.

So it's a unicorn. Yes.

Yeah. So let the people who are really good at communications, really good at translating the technical to the non-technical, and vice versa, and tracking and organizing, let them be good at that.

Hmm. What's your best advice on building that function, especially for people who aren't attuned to its existence? Like, you don't expect that you need it and you show up, you literally, or a book shows up and they say, oh, we don't have that. And I feel like the Mandiant team here explains that we must have it. What would you do next? How would you produce it?

I would put together what I call a service catalog. So across the different functions that your organization is choosing to implement, what are the services that they provide to the other functions? Those other functions are their customers. And then how are those services triggered? Is it through a request? Is it through the existence of the threat? Is it through an initiation from a designated team, individual, vice president, et cetera? And then track how those flows are operating back and forth because everything really should be a data flow as it's moving through every single one of those critical functions to make sure that all of them have had a chance to support the organization against the inbound threat.

That actually makes sense. And I think this alone is also worth the, you know, price of the show. We have a couple minutes left, so I want to go to the traditional closing questions. Usually, we ask every guest for one piece of reading. In this case, it would probably be the Defender's Advantage book, and maybe one tip on making use of the resource.

I think the biggest thing to recognize at a defender's advantage is you have the power, you have the influence in order to support an oncoming threat. And that's something that a lot of people miss when they're reading the book. They focus on, again, the critical functions, and they look, all right, how do I do detection engineering better? Okay, yes, you can do detection engineering better. You can apply this methodology, but it's more of how are you actually working with the other teams and what kind of influence are you applying to your organization? If your champion is not giving you the authority to make changes, then you're not actually achieving that defender's advantage.

Yes, this actually took a good number of seconds for me to absorb because I think this is actually super useful advice, even if you read the book and match the elements of your company to that book, but you lack the power to move the pieces around, you would not really benefit from defender's advantage. You may still have— you may still run an environment where attacker has an advantage.

Exactly.

Yeah, perfect. Thank you very much. On this profound note, uh, very much appreciate your answers and of course the resource. Thank you for being on the show.

Anytime, Anton. Thank you.

And now we are at time. Thank you very much for listening, and of course for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton_Chuvaikinen. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode. Security.