EP264 Measuring Your (Agentic) SOC: Two Security Leaders Walk into a Podcast

Security.

Hi there, listeners. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here in a random office in Seattle this week are myself, Tim Peacock, the Senior PM for, well, a bunch of stuff in Google PsychOps, and of course, Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as our new video format on YouTube. youtube.com/@cloudsecpodcast. If you enjoy our content and want it delivered to you piping hot, don't forget to hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on the Google Cloud Security Community page. Anton, we have 2 returning guests today and we are talking about, well, not perf reviews, but SOC metrics, which I guess kind of look like a perf review of your SOC performance.

I think this was one of the most surprising episodes in recent history because it combined some of the profound insights about how Google does DNR, detection and response, with the client experience. So to me, this was, I don't know, I think you had the idea to combine 2 of these return guests in the same episode, and the result is just epic. This is the lovers of how Google does things would love it, and customer story lovers would love it as well. So a lot of love going on. I don't know.

I think there's a lot of love, and we have, of course, a lot of love for these 2 guests. But what I love about this episode is that we get really practical advice on how to think about specific kinds of metrics, how to think about your metrics program overall, and how to think about how your metrics might or might not change with the arrival of AI in your SOC. And so maybe before Anton and I give any other spoilers, listeners, let's turn things over to today's guests. Today we are lucky enough to have 2 returning guests. Fortunately, this time not trapped on an island. and, well, somewhere in Europe. We're joined today by Alex Pabst, Deputy CISO at Allianz, and Mike Cyno, Director DNR here at Google. Super stoked to have the two of you on the show. This is an episode that is a long time coming. We want to talk about SOC metrics today, and I think I'm going to steal the first question. Sorry, Anton, because I'm already talking.

Yeah.

I want to start with, are we here to bury Caesar or to praise Caesar? when it comes to the death of legacy metrics like mean time to detect and mean time to respond? How do these change in an AI world? Are they still relevant? Tell us the fortune unto Caesar, please.

I think it's evolve Caesar. I don't think it's bury Caesar or praise Caesar. I feel like metrics have to constantly evolve with the technology and just the situation, right? There are times we add additional metrics to my reviews for my team. when certain projects are going on or whatever, and then we'll drop them. So I think we need to evolve them for the new era and think possibly in different terms as things get faster and we have to meet the moment.

Yeah, I think I can echo that, right? So to me, there's kind of like 3 levels of metrics that we look at. We've got the technical metrics that we look at in the SOC, which is like all the how fast is your— well, it's probably still mean time or 95% response time or whatever you look at. And then the further out you go from there to what I would call operational or strategic metrics, I think they are more stable, right? If you talk to a board and you talk— I don't know, if I talk to our board and tell them about mean time to respond and mean time to detect, their eyes glaze over. So I think depending on the level of metric you look at, you see some move faster versus others are more stable.

But I want to sort of introduce a very brief rant. I hoped Michael would kind of hammer it harder, so I'm going to give him a bigger hammer. I feel like people who obsess about MTTD and MTTR and just speed and time metrics occasionally teach to the test. They say, well, can you click the button faster? And people are like, click, click, click, click, and everything is faster. But everything is 10% faster and 2 times worse. So, I mean, I've been kind of, in theory, understanding of why people like MTTD and MTTR, But is there any way to sort of do a modernized version of that that doesn't motivate people to just do things a little faster and much worse?

Let me steal that one, right? So there's a— Oh, lovely.

Yeah.

Oh, wow.

Alex wants the hammer.

I want the hammer. No, there's a great quote that I wrote down. It's by Charlie Munger, the vice chairman of Berkshire Hathaway, right? So the quote is, show me the incentive, I'll show you the outcome. Exactly what you described, right? That's why to me, there's different levels of metrics. To me, it makes sense to collect all the metrics and all the data that we can For SOC internal use, because even there, it's still useful to know how fast do you respond, what type of, what percentage of your incidents are fully automated, how much AI do you use, all those things, right? As soon as you start reporting them out and they become an incentive, which I think at Google is slightly different from a traditional company where I think you're used to dealing with data and having certain numbers without necessarily having them as an incentive. For us, any number you give out, people go like, oh, you need to increase that by 10% or decrease it by 10% or whatever happens. And the whole mean time, as you said, I think we're going to go away from that, right? One thing is that mean is a very, very bad metric overall in the first place.

Yes.

Right? Because you can have one incident that takes 3 days to resolve, and then you have a million incidents that take 30 seconds because they're automated, and you have a great mean time. So I think we're going to see that being adjusted. We're probably also going to see the base on which we take those numbers change, right? If it's fully automated end-to-end and it's done in seconds, I don't care. That's not the thing I need to worry about, right? I want to look at the stuff that gets stuck, where no analyst is assigned because something broke, where the incident resolution takes forever.

Although I do think you should care eventually, right? Because as the SOC transforms and we do more and more automated, we need to follow along. Right. So what's happening there? It might be— it's completely different metrics. Yeah, because I, I'm kind of halfway there of mean time to detect, mean time to respond are bad incentives because it's how fast can you click a button. But at the same time, I think they need to be paired with other things, right? So it's how fast did you click the button and did you do a good job on the ticket?

Correct.

Right. So we're actually like, we're using AI to actually evaluate subsets of tickets and reopen them if they don't meet certain criteria. Now it's all very early, so I won't say we're doing it on every ticket, but it's— so now it's, did you meet the SLO? Because we have SLOs as well. And then it reviews and says, you did everything you were supposed to, the summary is great, close the ticket or move it forward, or it kicks it back to the person and they look at it. So we're looking at a quality metric and a time metric. Not just time, because I think time is a bad incentive, right? Because people will try to game the system. My first job was at a call center, right?

Mm-hmm.

And we were tracked, like how long could you stay on a call? And you could be on for 5 minutes or whatever. And the first thing everybody did was figure out how to game the system so you could get the best metrics. But we were teaching to the test, right? We were managing to the metrics, not actually the quality of what we were doing. Nobody cared, right?

Right.

Of course I did because I was the model employee, but that's the difference.

Yeah, obviously.

Obviously. That's how you ended up here. Yeah. So Anton, do you want to ask the next question then?

No, apparently you witnessed the situation described in that question, so you should probably go with that because the whole maturity triangle, I may become too obsessed. So please ask.

Oh no, I, I didn't witness this.

This is, uh, well, we, we, I witnessed this and Mike witnessed this and I'm hoping we can share this with everybody.

Okay, so can guests ask the question to each other perhaps? Yes.

Yeah. Yeah.

Do our job.

Anton and I are going to replaced by AI anyway, so you too.

I'll try.

No.

So, Mike, one of the things— we met in Zurich last year, and we had a great conversation on how to measure maturity of the SOC in general, which to me is a great metric. Obviously, it's very hard to put hard data behind it. And the great thing that Mike and his team drew for us or kind of built out was what I would call a triangle or a pyramid, where you look at kind of the infrastructure that goes across everything at the bottom. And then, as you go further up, you go through the different layers. Let's call it operating system, application, data on top, depending on how you look at it. And then, you look at the maturity of each of those individual pockets in terms of detection, in terms of response, in terms of whatever else. And I think the question, Anton or Tim, that one of you was going to ask is, how does that triangle change now that we get more automation and AI into the game? How does that change? I think it's a great question. I don't have the answer. So, Mike, I'm looking at you for that one.

Yeah, I don't have the full answer. So, you're talking about the triangle with all the circles?

Yes, correct.

Service maturity, like base level all the way up, right? Or as I think about it sometimes, the triangle of doom. But I think with automation, there's another layer in there, right? Because you're now managing that automation or AI, let's just say AI, but it affects each of the points in the triangle independently as well, right? Because we still end up with possibly one more mature than the other.

Yeah.

Right. But I don't have the full answer. It's actually something we're debating right now, is, like, what are really the implications here as we're doing more AI-related automation? Because, I mean, honestly, like, we've done automation for 15 years. We're, you know, well further than a lot of teams on kind of what we're doing. And, you know, when I look at it, some of the industry is trying to invent solutions for things we've been doing for a long time. But differently, right? So, you know, some of it's inbuilt into that model, and some of it, I think, is really up for debate right now and how we deal with it as we use, you know, more agentic solutions, as we modernize the SOC, as we have agents writing detections, because all that goes into that pyramid.

And maybe my hypothesis would be it's not necessarily the metrics that change, it's probably the goalposts that you put on those metrics, right? Because as you automate more— and this hypothesis, I think, we'll see if it comes true or not— but as you put more automation in, And if you do put proper quality automation in, to go to Anton's point, you'll see that all the metrics you take, that they're getting better, right? So you kind of start moving goalposts. At least that's what I try to do to the team.

Well, the goalposts are going to move and they're really going to move. It's going to be— I don't even know if we're going to have— we're moving the field across the world. Across the world, yeah. It's not even pushing them forward because As we do more and go from hours to minutes to seconds to less than seconds in this, everything changes.

Yeah.

And that's kind of where we need to be. I mean, we're even looking at, we're working on models that we want to be there as it's happening. So rather than detect and respond, infer and interrupt and say, hey, Tim's doing something sus because he always does sus things. And we're going to send them a challenge, right? So we're going to say, why are you doing this? Or we need an ID challenge to make sure it's you or something like that. So it's, you know, it's not omniscient, but it's, it's there much faster than detecting it after the fact and then going and talking to Tim and saying, why did you do this? We're saying, why are you doing this?

Right.

So it's, you know, changing the goalposts, moving the goalposts and moving the field all at once. Moving the field.

I like that.

Yeah. I mean, if you're going from a world of following up hours later to a world of interrupting, that is like a different planet you're playing the game on now, not just moving goalposts.

Now, it's not a panacea, right? Like, detection and response will always be necessary, but what can we shift left to be earlier in the process and say, we don't even need to detect this because we already saw it while it was happening and asked them and we know this is legit, or this was sketchy, so We took a mitigative action, right? We contained them until we found out more and actually did talk to them because robots aren't going to be able to figure out everything. So we might still need to talk to the person, but it's better than knowing it after the fact. Yeah.

And I think that brings us back to metrics to some extent. And by the way, before I go ask the next question, just a reminder, there are still organizations when the average detection is measured in hundreds of days, last time I checked. So if somebody is shifting from days to hours and then shifts from hours to minutes and then from minutes to seconds and then from seconds to real time, we should be somewhat supportive of the people who have shifted from 237 days to 122 days before they detect. And yeah, that's just a minor side reminder so that we don't get too elitist here.

Yeah. Well, it's also what constraints do you have as an organization?

Yeah.

Right?

Yeah.

To be fair, you're 100% right, Anton, right? Like, it's what constraints do you have funding-wise, staffing-wise, and everything else? That helps set the bar.

Or regulation-wise, right?

Or regulation-wise.

Let's not forget that.

Yeah.

I don't want to steal the topic and shift us to vulnerability management, but it reminds me of my many Gartner experiences when no amount of beating can accelerate patching beyond a certain timeframe at a company. I kind of called it the patch sound barrier. Like, there's nothing you can do. You can order everybody, you can threaten to fire, you can give 5,000% bonuses, but patching cannot go faster than certain X. And that X just seemed to be, like, an immovable object. And I think in DNR, it's a bit more fluid, at least that's my impression. But in the MVM, it was just like, there's a hard limit. And if you discover it, that's it. You have to transform or you're stuck. In any case, sorry, what I want to actually ask is not that.

That's quite a side quest there, Anton.

It was a bit of a side quest, yes. Direct question to Mike about metrics that we use, right? Kind of toil reduction, how do we measure the humans versus machines part? I remember from one of the earlier diagrams your team shared was the whole cost of a detection in terms of time, because if a human does shoveling, human charges human rates, but if the machine does it, machine charges robot rates, and that's cheaper. So I really was impressed at the time by that. But what's the current state of affairs? I mean, whatever you can share, of course.

Currently, for the agentic work that we're doing, we have several agents running. I'll give an example of one that's deduping tickets because you get tons of duplicate tickets. So what we're tracking really is the operational reduction. So what percent of the tickets were closed and touched without a human having to interact with it at all?

Right.

So we look at the percentages We have a dashboard for the percentages, but we look at the percentages. We also look at how much of it, because we still have humans looking after the fact and kind of sampling to make sure that it's not going completely off track, what was the result of that? So we'll look at the accuracy and then how many incidents or how many of them were wrong. So that's where we are. It's a very simple metric to track right now, but that's the report I get. We go over it monthly, but we also look at it weekly and in some cases even daily, depending. It's been very largely successful, but we're also taking a very cautious approach in how we implement agents because we also know that we basically have a zero-fail mission. So we need to ensure that we're as accurate as possible. We know we'll never be 100%. Humans aren't 100%.

Yeah.

But we take it carefully. And we start with, you know, humans in the loop and then humans over the loop and then some checking. And then, you know, maybe eventually one day we'll set it free because we love it, but that's where we are right now. But it's some, some very basic metrics that we're tracking right now there.

Perfect.

I wanted to pick on this regulation thing as well because Anton can't talk about regulations.

No, no, no, I would be triggered by auditors.

Yes.

Yeah, yeah.

So I'll, I'll ask Alex the AI question, which is the AI regulation question, which is like, How are you navigating the regulation side of this? And what are the metrics around that? Like, how does that come together for you? Because I think you're maybe our most regulated guest.

Maybe.

That's a questionable honor. Questionable honor.

That's quite a title.

Look, I mean, regulation on that side, like the written regulation, as we all know, is normally behind what's happening on the technology side. So, the regulation, including DORA, that we got last year in Europe, that was actually created 4 years ago. So, they normally either regulate high-level concepts, right? So, you need to have a SOC, you need to respond in some decent amount of time, you need to report within 24 hours. They don't go into the details in the regulation itself. That doesn't mean that the regulator themselves don't come and audit you and tell you, hey, the spirit of the regulation in our mind is XYZ. To this point, they Look at outcomes, at least for us, right? So they want us to be able to detect at a certain time. We have these, I think I talked about them in the last podcast, we have these TLPT, threat-led penetration testing exercises, where basically we get a red team in and the red team reports directly to the regulator. And they're actually happy if we detect them earlier. We are happy if we detect them earlier and the regulators as well. So I don't think we have direct regulation on that yet. What we do have regulation on is, or what the regulator's interested in, if an incident happened, it needs to be auditable. Mm-hmm. And doing something in an auditable way with a non-deterministic AI sometimes is tricky, especially if something goes wrong, right? How do you explain to the auditor that that alert that popped wasn't looked at because the AI decided it was not interesting? And that makes it harder for us to scale, right? So, I mean, we use it in testing, we see a lot of value from it, but we probably have more of a human shoveling, to take Anton's analogy. than Google does, right? Also because, I mean, I think that the scale of data we look at is 10x or 100x less than what Google collects today, right? So we don't have enough shovels. Our goal right now is to make the human shoveling more efficient instead of just purely replacing it.

Mike, I'm curious for you, have you thought about how you would explain to, say, Google's board or Google's regulators if the AI missed it because it thought it wasn't interesting?

I mean, right now we have decent observability. I would say it's still non-deterministic. So, we've had to. I would have to bring in Madhu, who leads a lot of our regulatory work for DNR, on how we have. But it's still taking them on the journey of how we got to the outcome. I think it's gonna become a bigger, like Alex was saying, it's still a tough sell because it's non-deterministic. It's like, well, the AI told us. You're like, but— How did the AI arrive at that? I mean, we know how it was constructed, we know what we put into it, we know all the golden data, so we will show that data. But there's still a point of fuzzy logic there. So that's how we've dealt with it. But I think we're gonna really hit it this year, next year, and in the coming years of, like, as we do more with it, how do we justify what was done with it to auditors, regulators, customers, right?

Yep.

Because eventually Alex probably wants to know, As our customer, how are you arriving at these conclusions and how do I know that that's right? Because there's probably a transitive.

Yeah. Or one step further, right? So we had this incident because something wasn't detected.

Yep.

Why?

Why?

Yep. So this reminds me, and this I think is useful in the whole AI readiness for SOC work that I've been doing, kind of points out that being okay with uncertainty. And people, of course, have trouble with explaining it to auditors, again, not to go on that tangent again. But the fact remains, we'll have to be okay with uncertainty. And it's kind of like, if auditors argue with physics, physics still wins. You know, like auditor may win over some non-deterministic argument, but in case of physics, gravity would beat the auditor. Sorry for going a little morbid.

You don't know our auditors.

Yeah, I would, I would. Anton, I don't, I don't know if that's, that's the case, depending on the region you're in. Like, we've been in some really, really Interesting audits that I, I would say they beat physics, right?

Like, okay, they—

I, I—

okay, I withdraw my question. I'm gonna ask a different question.

Sorry, I, I just have an image of like an auditor going out a window in Anton's mind. I think that's what he's actually getting at.

He's going up though, out of the window and up, because he beat gravity.

Oh, that's scary.

It's basically The Matrix, right? We can change it a little bit, but we can only change it so much.

Yeah, so I want to latch on to something Alex mentioned at the beginning that And it was a very wise point about if you show MTTD to a board, you're going to be in a different type of trouble because it's just not clear. So are there board-level DNR metrics?

Like—

Yes.

What are the board-level DNR metrics? That's a great question, Anton.

Did we get hacked, I guess? But it's kind of boring a little bit, right?

That's like a zero metric. That's easy.

It's essentially MTDD, MTTR, And our pipeline latency to a point.

Pipeline latency cannot possibly be a board metric.

Apparently, that works at Google. Does that work at Allianz?

I think we have slightly different boards, and I'm slightly jealous of your board, Michael. So, for us, I mean, the language of our board is risk, right? And it's risk in terms, normally in terms of dollars or euros, right? So, to me, the further you'll go up the stack from tactical to operational to strategic metric, the more fuzzy the metric gets on the underlying components of that metric, at least in Allianz and in my world. So, yes, if I talk to the board and tell them, hey, we have a pipeline latency of 3 milliseconds, they'll probably shake my hand and at the same time guide me out of the room. So, the discussions we have is mostly around how much risk did we reduce, what incidents did actually happen. I mean, some people try to tell them how many incidents or attacks you blocked, which I think is the worst metric you can come up with.

Firewalls, port scans hitting your firewall is the board metric. Hey, hey, hey.

Hopefully not, right? But there are people out there that try to sell fear, right? It's like, we blocked 3 gazillion packets last year. It's like, well, congratulations, you put a device on the internet. No. So for us, it translates up, right? So it's risk exposure. And then the question the board ultimately wants to know is, is our risk exposure within our risk appetite? And if not, what are we doing to either get it back in or stop it from moving out of that?

And let me be clear, those are not the only metrics that roll up to the board. They roll into the board-level metrics that cover the risk. So those are all presented to the board, but from DNR-specific, it is the incidents, but it's the pipeline latency, and then where are we in terms of our detection infrastructure. So it's, those are a couple, but there is a ton that gets rolled up to the Google board. So it's not that easy. They prepare the deck every quarter and then my boss goes and presents all of that to the board. But I don't know, it's like 30 slides of various things.

Wow.

So while we're on the topic of boards and people who care about economics, dollars and euros and what have you, how are you two thinking about the unit economics of AI in the SOC? Is it labor replacing, labor augmentation? Is it obvious 1-to-1 ROI like we had in the discussion a couple of weeks ago with our friend from UKG? Like, how are you approaching the unit economics here and modeling that out for the future?

I mean, for us, it's not replacing labor. It's— we look at it as turbocharging labor or getting rid of their triage toil or whatever it is, because we can't get enough good analysts, right? So I don't know, Mike, I think your automation ratio is slightly higher than ours. We're not even at a 50% ratio, which I think would kill Google if it were that low. But for us, it's as I said before, right? So we need that auditability. We need people to look at pretty much any alert, even if AI went over it. But now they can just glance at it and say like, yep, this feels right, or this is right, this is wrong. Actually, this IP address is internal or external or the other way around. That's how we use it. There are some automations, not necessarily AI-driven, but just like pure automation. If you report a phishing mail, that automatically gets processed, blocked, yada, yada, yada. But that's deterministic flows. So that's how we look at it. So the unit economics, it's just allowing us to do more with less. And we need to do that, right?

Human productivity enhancement.

Yeah, human productivity enhancement overall, also human. But what we are seeing is, I think we're seeing a 30% increase in incidents or alerts, alerts actually year over year, right? We can't keep growing our— and I mean, as much as we'd love to, but like in 10 years, we'd deploy half the world.

Yeah. I mean, it's the pursuit of sublinear growth, right?

Correct. Thank you.

I look at it very similarly. I think it's augmentation right now. Like, I don't think it replaces humans, and I also think it doesn't replace the critical thinking skills of a human.

Yeah, right.

I was on another podcast on Friday, and I kind of gave this advice that, you know, one of the biggest skills for an analyst, right, or, you know, incident responder, you know, in security, is critical thinking.

Yeah.

AI does a lot of things well, but I don't think it understands nuance well. And we need that nuanced view still. So, you know, with our detection operations team, we still need them to be there. This is augmenting the work that they do and taking away the things that they don't need to do. I think that's the big key is instead of piling everything at them and saying, just deal with it, We can say, you don't need to do this anymore, you don't need to do this in there. Focus on the big chunks of things that humans still need to do.

Focus on the hard parts, right?

Yeah, the hard parts, the things that require deep dives, right? Like pre-long-tail forensics, but the harder jobs. And from that, I mean, the economics are hard, right? Eventually, space, disk, RAM, processor, all of this just in general, Come at a premium, add on TPUs, et cetera. But then, moving back to detection engineering, it's kind of a similar thing. We still need humans doing detection engineering, but at the same time, I want to get to a point where a lot of detections, we're not touching. They're writing themselves. The AI is writing them.

Or updating or adjusting them or fixing them or something.

Yeah, the AI is managing all of that, and we're writing the 20 or 30% that I want to touch on the economics very briefly.

Initially, I wanted to have some kind of a pithy, mildly disdainful remark about this, but now I feel like it's a very exciting topic. And the reason for this— I'm sorry, I'm laughing at my own jokes.

Somebody has to laugh at them.

Well, there's that, yeah. So, some of the AI SOC vendors would show up, and somebody shocked me by saying there are more than 40 vendors, but then the other person said there are even more than that. A lot of them secretly try to sell savings by getting rid of humans.

Yes.

Right? And it doesn't really kind of work out well for them. And, you know, sad to be them in that sense. But the alternative model is more difficult because, like, if we say, hey, same humans can do more, same humans can do better, same humans can cover the increase in volume of threats. Now we're going to have agents hacking us, so we need more So to me, these are good arguments, but they're not very economical. They're not very economic. So how would you turn this productivity into something that strikes almost as hard as get rid of the humans, which is false, but it's striking?

I think the economics will change over time, right? If you look at like when, you know, calculators were first introduced, they were ridiculously expensive, and now you get them as a token out of a machine for 25 cents. I think over time, some of the economics will change. And the view is too that the normal SOC— my view is that the normal SOC operations, the security analyst job is shifting. We talked to the entire team last year, the year before. Everybody's jobs are changing and we need to change with it. So I do think while it doesn't— it's not like, well, we don't need the SOC anymore and we don't need detection engineers and we don't need incident responders anymore.

Right.

I think the job changes over time and we're navigating that change now. So we may need less humans, yes. And what those humans do might be different. So I think the companies that have jumped in and said, well, we just got rid of all our customer service agents and we don't need them anymore, I think that's premature. Yeah.

And there are enough stories of them hiring back the agents at twice the cost or whatever. Exactly.

But I don't think it's a I don't think it's all or nothing. I think right now we're in this human-in-the-loop, human-assisted, assisting humans model. And as the technology matures, maybe we don't grow the team and the team's mission shifts to other things, other related things. But I don't think it's a, we don't need anybody either. I think it's kind of a middle ground there.

I think the economics, Anton, is what Mike said earlier. It's the pursuit of sublinear growth. If you can bend that curve for an organization long-term planning, there's ROI to be had there.

Well, there's also a second level of ROI, right? If you can prove risk reduction through this or risk increase mitigation, that's also something that on a higher level, people understand, right? If you can say, look, we changed our cyber risk from, I don't know, 100 to 90 by investing X amount while the threat landscape outside increased. I think that's also a story that works, at least for our board.

Well, I hate to do this, but we're coming up on time, and I think I have to ask our traditional closing questions. So for each of you, do you have a piece of advice to help people, you know, better plan out their SOC metrics and particularly how those metrics are going to evolve in the age of AI? And then 2, do you have recommended reading for people? And that could be anything except Anton's blog.

Uh, what about my blog on SOC metrics? Come on.

No, no.

That you can read. No, look, I mean, to me, I like to think about metrics as a storytelling tool, right? Because they should tell a story. It's either to your own team, to the board, or to someone else. They need to be understandable and simple to measure. In terms of something to read, so I don't know, Anton, if you're gonna love this, but Ellie Mellon from Forrester actually wrote a nice report that's like 5 Steps to Better SOC Metrics. Yeah, love it. That I really liked. And it's kind of helpful to give some perspective.

We should probably link to it as well. Alli can buy us beer at Next.

We actually, we both like Alli and we'd like to have her on the show, but we don't have that kind of money.

Yeah, well.

I wanna say, we all have to remember that metrics are living things, right? That we're not chiseling them in stone like the Ten Commandments. Metrics need to evolve as our operations evolve. And I think a lot of organizations fall into, well, this was always the metric and I always did it this way, so this is the metric we should follow. We need to be self-critical about the metrics we are tracking and feel free to drop ones that we can, because there are some that we need to track for regulatory or other reasons, and add new ones as necessary. AI is a perfect example of we need to rethink about how we look at metrics for AI. So I think that's my advice is remember this is It's living and we need to evolve how we track as the technology, as our operations, as the regulatory requirements and everything else evolve. And I would say recommended reading are various Google Threat Intelligence threat reports because I think everybody's super excited about AI, but we also need to keep the focus on what the end of that funnel is, and it's catching the badness and catching the bad guys. Right. So, let's not forget to do that.

It is indeed that. Yeah, that's the critical bit. The thing you said about evolving and living, that takes me back to some of our conversations you were having with Royal not too long ago as well, of how do we approach this as more of an organism that we're trying to foster, not as a set of systems we're designing like gods. And Alex, what you said about metrics are telling a story reminds me of the famous quote that you can torture data into saying almost anything.

Yeah, but that's why you should figure out what story you wanna tell and then find the metrics to support it, right?

Ooh, listeners, I can't endorse that. They will take my PM license away.

This just goes into a slippery slope here.

What's that saying? Like, 60% of liars use metrics or something like that?

Yeah, exactly.

No, there's a Mark Twain quote. There's lies, damn lies, and statistics.

No, I mean, I agree with you on that one, Tim, right? So, like, make sure you do this the right way and then don't find data to support what you're trying to tell.

Yes. Awesome. Well, super fun conversation. I'm so glad we finally had the time to do this. Listeners, thank you so much for joining us today. Alex, Mike, thank you for coming back on the show.

Thanks for having us.

Yes, thanks for having me.

And now we are at time. Thank you very much for listening and of course for subscribing. Please subscribe so you can get new episodes piping hot. And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcast. You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us, and if we like or hate what we hear, we can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast. By Google.

Security.