EP236 Accelerated SIEM Journey: A SOC Leader's Playbook for Modernization and AI

Security running.

Hi there. Welcome to Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, Senior PM for Detection, UEBA, and Risk in Google SecOps. And of course, Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on the Google Cloud Security Community page. Anton, we have a truly epic episode today.

Yes.

And I think what I love most about it is how much of our guest's thinking boils down to a single question. And that question, I think, is to ask of everything in their sim, what is it in service of?

I think that bit is quite magical. I agree. I mean, there are actually other magical bits, not just that one, but my impression is it was just a really, really good episode. And I think that with our gushing about how epic it was, it may come through in the show itself. But it is about sim.

No spoiler, not migration.

It's about SIEM, not migration. Yes, correct. But given that the blogs we've written on SIEM migration have gathered like tens of thousands of views, I bet this episode would be one of the big winners, especially given how amazing our guest was.

So there's another thing that comes through throughout her thinking that I think is really worth highlighting, which is how human-centric it is. Even when talking about AI and how to use AI, her thinking there is all about her people. Like, we've had episodes about leadership and we don't talk explicitly about leadership on this episode, but I think this is an amazing example of exactly that as well. And so I don't know, I could gush about this episode probably all day, but unless you want to gush more, maybe we just turn things over to today's guest.

I think I want to briefly gush for one more second.

Oh, good. Gush more. Gush more, please.

I think that the episode is very fun, very good energy, very good ideas, but it also has a lot of things that you can use, the audience can use. Like, if you are in the middle of the hairy SOC project, whether it's transformation, recreation, creation from scratch, just major tool migration, there are ideas that are really battle-tested, that are really battle-hardened at a major institution with a lot of procedural burden. I'll be very soft-spoken here. And these ideas are just pure gold and they have worked. That to me is the critical one. They are not coming from a vendor. They are not coming from a quote-unquote expert. They're coming from somebody doing it today.

Yes.

Through hard work and with really good results. That's the magic.

And on what tool? Our tool.

What tool? I have no idea.

Oh yeah.

Yeah.

It's our tool.

Okay. Yeah.

This is really like, we're so rarely self-serving and I don't mean to make this self-serving at all, but it is worth highlighting. It's all about doing it on our tool. And so maybe with that, let's turn things over to today's guest. Today we are joined by Maneja Politova, the Director of Security Engineering and Operations at Lloyds Bank, which I understand is like, A pretty old, meaningful, big bank. So this is, this is really great. We're talking about one of our favorite topics, and I never thought I would say that and mean it, but we're talking about SIEM migration today, and you're doing a SIEM migration right now. We all know that SIEM migration is hard. It can take ages. And given the scale and complexity and age and doubtless, like, suit-wearing London people of Lloyds Bank, this probably would take a long time, but you planned it for 9 months. And so has it gone that fast? Has it gone? And how did you plan it that quick?

Hi, everyone. Thank you, Tim. Thank you, Anton, for having me. Super excited to be here. And Tim, let me just start with saying the first thing is don't do migration, do transformation and transformation only. And let me tell you. Yeah. Yeah. So, hey, here's—

No migration.

No migration. And let me tell you why. So when we do a typical migration, what tends to happen to us is we look at our detection rules, We'll look at our response rules, data sources. We apply some filters, which is really hard to let go, by the way. And then you try to—

Yeah.

Have a set of things that you just want to move over. And there's definitely, like, an incremental improvement you will see, 100%. However, the problem with this approach is you don't actually solve the fundamental problem. You don't allow yourself to step back and think about, is there a way to do better detection altogether and response? You know, we're really bought into this continuous detection, continuous response. And by the way, to implement that, you have to rethink how we do security altogether. So to your question, to answer to your question, I was brought in to do a SOC transformation a year and a half ago, and it's been one of the most challenging projects that I've done, but it's been one of the most rewarding. And when we talk about transformation in Lloyds Banking Group, we actually mean the way we approached it is So, I was thinking about it, and I thought, if I were to have a blank sheet of paper, what does a modern SOC look like? So, from that moment, we've never looked at what we have, what are the current processes, what are the current detection rules, and we just built everything from scratch.

What I want to say about that is, like, I totally get this no migration thing. That's the case of, like, wherever you go, there you are. If what you have doesn't work or you don't like what you have, of course you shouldn't do a migration. This seems so obvious. Anton, am I just not undercaffeinated right now?

I also like the whole point about not basing what you will have based on what you do have today. And it's interesting, I was discussing it with some colleague from a consulting firm, let's put it this way. And we thought, let's write a paper about building SOC from scratch. And he said, nobody who needs a SOC would build it from scratch in 2025. And embarrassingly to both of us, We haven't thought about your case where it's not from scratch, but it's being built as if it's from scratch. So, if you see a paper from us in the next—

we are slow—

6 months, 9 months, 9 months, I'll just do it out of solidarity. If in the next 9 months you see a paper from us about how to build a SOC from scratch, you were sort of a partial inspiration for it, even though some of it's already written. We just weren't sure there is a case for migration by starting from scratch as opposed to migration by Gently tweaking everything that you have.

Yeah. So definitely, Anton. And the way— another thing to add to this transformation is, so the first thing you have to answer is, what is a modern SOC, right? And I've seen a lot of people, like, overcomplexifying this approach. It's like, what is it? And it's really simple. A modern SOC is the SOC that can detect any and every attack, especially the sophisticated attack. And then you go from there and you start thinking about, And what does need to be true for that to happen? So this is where we shifted our mindset from a traditional SOC to it has to be an engineering-led SOC. And we even have this mantra today, my team tired of hearing me say the same thing, everything as code approach. So we're trying to codify everything that is humanly possible and in service of what, right? You have to always think about in service of what. So in our modern SOC or new SOC, what we want our people to spend time on is really mainly 2 things. So the first one is you want to create a condition where your people are proactively hunting, and that's the majority of their time. So they're looking for things that could go wrong, new attack paths, different attack paths, and then based on that, they're improving their detection and response continuously, right? And everything is code. And then the second part is if something fires, so it's a high-fidelity alert, Because that's what your SOC does. A modern SOC fires high-fidelity alerts, so we're most likely in IR mode. So these are the concepts that we've kind of had in the beginning, and then we started executing.

That actually makes sense. And I think the as-code part makes some people think about the tech, but I want to put my reformed analyst hat— and I mean, I have a hat, but it's also— I only have a Google hat. I don't have a reformed analyst hat. So it's a metaphorical hat. Fine. So I'm going to put a metaphorical hat on. and bring up the whole 3-legged stool of people, process, tech. And when some people hear as code, their minds immediately go to as tech, to tech. My mind goes to process because to me, the engineering lead is mostly process. And of course, to an extent about skills. But with this meandering intro, from the people, process, tech, what was the hardest and what helped the most with your sort of transformation? But I think we need a new word for that because transformation does make me think about anchor into the old but changing dramatically. So it's almost like rebirth of SOC, maybe, like in your SOC recreation. No, that sounds like it was done for fun. Okay, Tim, help me with your English.

Meninja, you know what he's asking. Just answer the question for the poor guy. Put him out of his misery.

Especially rebirth. The first meeting when we had, we had a whole team together. So I put the picture of what would represent our new SOC and it was literally a phoenix Wow. So reverse from actors, you know. So yeah, I get, Anton, I understand where you're going with this. But from the people and process and technology part, I think you're absolutely right. These are the 3 pillars you have to transform to transform the entire approach itself. What was the hardest? I think every of them have their own challenges. As of today, if I think about where we are today, by the way, we're 8 months in, 1 to go, and we're on target. So woohoo, in August we will have our new SOC. So the problem we have today is the data onboarding. It's probably one of the biggest challenges because we are thinking about, well, what is the right way to onboard the data? We don't want to be another dumping ground for logs again, right? So what are the actual process of identifying what is the log source do you need?

Yeah.

And then on the tech side, it's actually because of the complexity of organization, you know, you do have to think, write a lot of parsers, you have to do parser extensions, And that does take time. So that's like the things that we're dealing with right now. But I think the biggest challenge that every organization will face with any transformation is the mindset and skill set challenge. And let me just expand a little bit on that. So when we talk about the thing that we're trying to create, the new thing, you know, you have to start thinking about the skill set that you need to achieve this and the mindset. So a lot of organizations will have the same SOC, the team that is running the SOC, doing the transformation, and that is super hard from kind of like the stretch perspective. We don't have enough people as is, but also with the thinking perspective, because if you want a detection type of engineering type of SOC, you have to start bringing in different talent. And the way we solve these challenges is probably the most interesting. So we put together a SWAT team, and that is my biggest recommendation. So a year and a half ago, when I just joined Lloyds Banking Group, we started running hackathons. So every quarter we get together, anybody in security can sign up, and we, 2 days, for 2 days we just hack. We hack in the teams. And through that process, we actually started identifying top talent, like amazing people with the right mindset, people that think outside of the box. Some of them, you know, never done hacking and all of a sudden were leading teams in these challenges. So we picked those people and we gave them opportunity to drive this transformation. So in the SWAT team, we had a combination of DevOps, you know, infrastructure engineers, data scientists, data engineers, cloud engineers, and SOC analysts and detection engineers. So it was just such a healthy mix of thoughts and background that when you started looking at these problems that we've been trying to solve for so long now in the industry, actually we started solving them quite differently. So that is how I would say put together a SWAT team. And then another thing is, of course, partners. So we did partner with PwC and Google Mandiant team.

Mm-hmm.

to drive this transformation as well. And that is super important.

That makes sense to me.

Yeah.

I want to pick at something you were saying earlier about high-fidelity alerts. And one of the things that I've been working on lately is what we're calling composite alerting techniques. And so, have you adopted the same kind of funnel model? And are you doing this in SecOps? Like, how's that going?

Before you answer, I'm sorry, like, I just give these horrible flashbacks of funnels because I think ever since then, since the birth of SIEM, roughly around 1987, Have had a funnel in the slides. So funnel, I'm just allergic to funnels.

All right, Anton. All right.

Okay. Help me with the funnel.

Okay. So we won't call it funnel because we're not doing funnel.

Thank you. You saved us.

See, it's transformation, remember? So the thing that we are doing is that right now we're doing a lot of composite rule detection writing, right? So, and it's probably one of the biggest things for us, the feature to achieve the high fidelity alerts. There's actually 2. So the first one is this ability to combine a logic and actually start detecting for a logic. Right now, we're testing with endpoints, so like our use cases. The thing that we're looking for is, so endpoints are getting really good at picking up different things. But there's low and medium activities that individually you actually don't really want analysts to ever look at because that's the noise. You don't want to do that anymore. How do you create a high fidelity is by behavior. In this specific case, it's like if this and this and this happened, then this actually is interesting. But individually, they're low events and/or medium or high and medium combination. Right now we're playing with combination and looking how do we build. The way we're building it, we're creating feeder rules and then that feeds into the composite rule. That's number 1. Number 2, actually what it does really improve is managing your detection rules because the feeder rules are like your building blocks, so you can reuse them. for other detection rules. So instead of like approaching a detection rule as one rule that is very complicated and complex and then becomes this monolithic thing, you can actually divide this into small logics that you can easily pick and choose from as you build new rules as well. So that's going really well. And then on the high-fidelity alerts, another thing that I would recommend everybody to do is actually do purple teaming against the detection rules.

Mm.

So as soon as we write a detection rule that says, what we do now, we will do a purple team exercise against that rule. So that has dramatically—

Not against individual rule, presumably against the detection domain. Like, it sounds kind of like very heavy duty to do an exercise over a single rule.

Yeah, no, we're doing it. So again, when we talk about rule, because we're doing composite, so it's a combination of multiple rules, right? But yes, you are definitely doing that against that logic of a rule or many rules depending what you're writing. So yeah, you're right, it's a bit time-consuming, but actually it does speed up the process And then the output of that is amazing because when one of those fires, we know we are not in a good place.

Yeah, something interesting is afoot.

So, let me ask a detailed question. I mean, a slightly more detailed question on this. I mean, multistage, avoiding the final trigger word. I mean, sure, I can understand multistage detection when you have rules that produce signals and then rules that produce alerts from that. But to me, that process only guarantees high-quality detection if your second stage is done well. And how do you do the second stage well? You know, you can still do good signal rules and then screw up the actual alerting rules. So, what's the secret to doing the second stage well?

So, I think the first and the second stage, it's really important to do a lot of testing. So, this is adoption of CI/CD, right? So, or CTCR, which requires CI/CD approach. So, that is it. And that has been the biggest shift. and like the time spent on how we do detection and response engineering because everything has to be tested and it has to be tested through pipeline. And nothing makes it into production unless these tests are done and done properly. And then of course we have a peer review process, right? So part of the testing, like you create this, you've tested it, these are the automated checks, and then you have a peer review to just say, okay, does it look good or not before it makes it out there? So you test both, you test the detection and the alerting at the same time. pipeline, this CDCR pipeline.

That does make sense. And I think the fact that the other secret thing that you didn't mention is that with CDCR, your tests can run fast. It wouldn't be like sending an email to a consultant so he comes next quarter and tests the rule. It's, oh, yeah, you don't see this in audio, but her eyes just got double in size. No, no, no, not that. It has to run at the S-code speed, hence pointing out that S-code is as much about the process than about little characters typed in text files.

And that was the point that David French made on that episode, listeners, that I think is so important. Detection as code doesn't mean we wrote our detection rule in Python. It means we adopted the mindset and process and control environments that software has used for 20 years to be more reliable. Yep.

100%. That is right.

So, I want to step away from the funnel. Sorry. Now you made me obsessed about the funnel. Sorry.

I know.

But the thing is, the funnel thing is, I remember one particular slide from one long-dead SIEM vendor that had a huge rainbow-colored funnel. They had like 6 stages and all colors of the rainbow. In 2003, somehow the date is seared in my brain. And I was just like, not another funnel.

I love when we get a peek inside Anton's head. Like inside Anton's head apparently, listeners, are images of funnels from 22 years ago still seared in there. It's great.

Yeah, there is that. And also, okay, so funnels aside.

Funnels aside.

You had a plan. Like, 9 months, we opened this with saying that 9 months is quite fast. And of course, for a large global corporation like yours, 9 months is good. I'm impressed. I mean, in my analyst days, we've seen people run migrations for half a year.

It's not a migration, it's transformation.

That's my trigger word, Anton, so please.

That's right. Yeah, they've processed this even with more trivial steps, almost like lift and shift from SIM A, I did say SIM A, to a more modern SIM, still many months. So, what was your plan for that 9 months? Like, how did you plan your activities? What was the structure of that plan?

Yeah, so I think, again, like, going back to the fundamentals of adopting CI/CD approach, that is how you execute at high speed and with accuracy. So, the first thing you need to do is actually have a clear vision and understanding what are the main capabilities you want to build, right? spend some time on it, not too much time, because we do love hypotheses. And then where the failing comes is in actual execution.

Mm-hmm.

And then the biggest recommendation I would give to anybody driving such a huge transformation is learn how to break down your work and break it down into— so the way we've approached it is into 2-week sprints. And every end of every sprint, we have show and tell. So every time we delivered a 2-week work, we will show, and then we will talk about it. And then if we have to kind of course correct, because sometimes there was instances where the old ways of thinking did come across and we were like, no, no, no, no, we don't want to do that.

Yeah.

So that was perfect, right? Because instead of doing this kind of long-term approaches, you break it down and you can really, you become agile and you can course correct really fast. So that is my biggest recommendation, adopt the agile methodology of delivering, right? So make sure you break down the work and you look at Run show-and-tells, invite your organization. We invited everybody within security teams just to show and get more opinion as well and be open to feedback because there was a lot of times where some people just not even part of the SWAT team would come to us and be like, hey, actually, we've done this maybe better. This is a piece of code you could use to automate this part of approach. So, it was very useful.

That's awesome. I feel like D&R teams often get flak for being so secretive and closed off and not connected with the rest of the org. So I have to do a thing that I find slightly distasteful still, but it's 2025, and so I have to ask an AI question. Have you all started using AI? What's your agentic future looking like? And what do you think you want to tell the rest of the world about how to be successful with AI?

So it's a great question, actually.

Oh, no.

Yes. No.

They've gotten you too.

They did. And the reason is we just had a 2-day Google Next in London. So all my team went there, lots of AI talks, and everybody came back with, like, inspired thinking of, hey, agentic SOC, what are we doing about it? So there's 2 things I'm going to talk about. The first one is, we are still in the build phase, right? We have a month to go, so we have to get the fundamentals right. So the things that we talked about during this, like getting the processes right, getting the pipelines, the testing, that is the fundamental to adopt any technology, by the way, because if you have this thing, figured out, then it's just like, it's easier. So, and number 2 is I would say that with AI you have to be intentional.

Right.

The thing that we're going to do, so AI starts the phase of adoption of AI. So the answer is yes, we're definitely going to do that. Agentic SOC is next year. So the beginning of next year, because we will do a proper switchover. So the switchover is going to be delivered by the end of this year. So next year we're fully on the new SOC, which gives us ability and the space to now start looking at adopting AI agents and then always in service of what? So this is why you have to be really intentional. And for us, it would be in service of, again, creating even more space to take humans up the value chain so they can do even more important work. So we talked about, you know, proactive threat hunting. For us, we're envisioning a future where level 1, level 2 type of activities are taken by agentic software completely. The agents, they can do end-to-end. We don't even need human interaction. And then what do detection and response engineers do? They actually go into the space of pre- and post-breach. So they're looking at what could go wrong, the attack paths that maybe never existed and they found something. And then they also think about, and how do you fix it? And how do you go and actually proactively start fixing it? So we're now talking about improving the security posture. Now, we need to be careful because if you don't adopt it the right way, What could also happen is you become a clicking person. So you're just—

Yeah.

Yeah. And that's what we do want to avoid. I do not want to be in a place where, you know, we haven't achieved that up the value chain and people are just clicking yes or no, or is that a good answer, or write me a detection because— and then they forget how to write a detection rule. So this is why I always like to say to my team, we have to be intentional about it.

Man, I can't tell you how much I love that answer.

Yeah, same. I mean, I'm super impressed. I was briefly thinking it was going a little like in a traditional direction about like retiring ones and twos. But then later on, I think it came back to the beautiful new idea, well, beautiful idea that's new to many, where this is done well and it still follows the engineering-led approach.

But it's also so humanizing. I mean, like the sort of the core of that answer was, I have really talented people. My use of AI is to let those talented people do interesting work, not my use of those talented people is to click accept for the AI. That's so good. I love that.

Yes.

So hopefully we meet again next year and let's talk about how we've done that one.

Yes. Yes. You're gonna have to come back on the show a lot of times because I think we're, we're all enjoying this and there's lots of headroom. Phil's been on like 8 times, so you've got a lot of catching up to do.

I think so. We had to make a badge, a special badge for him because when he was a guest 7 times. So there's a memorial badge.

7 times?

7 times. Well, he has been, I think, 2 more after that.

Oh, God.

So I hate to do this, as Tim usually says, but we have to finish the episode. And for that, we have 2 closing questions. One is, please suggest a resource, piece of reading, a book, anything that you can read or perhaps listen, I don't know.

And also give us one tip for improving the odds of your sim So on the reading, I'm a very big fan of, for this type of work, you actually just do it, go and hack. So the best way you're gonna write amazing detection rules is by understanding what does hacking look like and actually doing that. So in Lloyds Banking Group, we do use HackTheBox, so everybody hacks. So that's my advice. And of course, there's a lot of Google security material that you can root through. But I do think just do it, just hack. On the second one, it's— I think I did mention this— is the odds is bring diversity of thoughts and skills into your team. And I know a lot of teams say, well, we don't have those people in the security organization, but actually go wider, go and ask for those resources, identify those resources across the organization because you do have, you know, data labs, you have other labs. So, why not pull individuals from those teams and actually build a team that can drive the transformation?

I love it.

I think that has proven to be quite effective when people unfortunately get stuck in the groupthink and everybody agrees, everybody loves each other, everybody is supportive, but they all flow in one direction and the attacker shows up from another direction and they're all cooked. Okay. So, this is like, I guess I made a negative spin on something very positive, but I think ultimately I do want to support the diversity of thought as a built-in principle for your detection response function. Tim, anything final, final, final thing to say?

I'm just sitting here blown away by how much amazing content we covered in a relatively tight episode, given the challenges getting started today. Aneesa, I cannot thank you enough. I feel like just as delighted right now as I did after our session together at Vegas Next. That was a high bar, and this has exceeded it.

Perfect.

Thank you, Tim. Thank you, Anton. It was great to talk to you.

And now we are out of time. Thank you very much for listening, and of course, for subscribing. Please subscribe so you can get new episodes piping hot. And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcasts. You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us, and if we like or hate what we hear, we can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast by Google.

Security.