EP184 One Week SIEM Migration: Fact or Fiction?

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Timothy Peacock, the Senior PM for Risk Analytics and Cloud Threat in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts. as well as at our website, cloud.google.com/podcasts. If you enjoy our content and want it delivered to you piping hot every Monday, just in time for your commute, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on the community page. Anton, we are talking about Google SecOps today. It's our second SecOps episode after having Chris on the show last year, and it features a week-long SIEM migration.

I think you provided a pretty massive spoiler. I think of this episode as more of the magic of a really well-put-together security engineering. I think it's much less about Google SecOps and it's a little less— I mean, it's about SIEM migration, but it's even less about SIEM migration. To me, it's an episode about— I mean, no jokes about 10x engineers, but it's about how security engineering Delivers magic.

Yes, yes. And I think what's interesting about that is it's about how security engineering is foundational to magic. And we draw an analogy later in the episode about the foundations at Google that enable magic. So I think with that, we can introduce our guest today. And I guess by what we just said, he is magic. So without further ado, let's turn things over to today's magical guest. Today we are joined by Manan Doshi, Senior Security Engineer at Etsy. Manan, thank you so much for joining us on the show today. Super stoked to have you here. I want to dive right in on the topic of SIEM migration, which is an evergreen perennial challenge. Having recently gone through one of these, in your experience, what are some of the biggest challenges that organizations tend to face when doing a SIEM migration?

Yeah, that's a good question. And first of all, in the last 3 years, I've actually done 3 SIEM migrations. Oh my God. Which has not been fun for me. But I've learned a lot of things along the way. So hopefully this is—

Wait, but your hair is still so dark. So you must have like a very good zen routine going on.

Yeah, but give it like another couple of years. You can already see a little bit of white that's coming in. This was not there before I, before I did a single SIM migration.

So you've done 3. What did you learn from doing 3 other than apparently not to do SIM migrations?

Okay. That's what I've learned now. But I'll tell you what the biggest thing I learned was. Um, I think when I, when we were doing our first SIM migrations at Etsy, or I would say even our first SIM adoption, cause we used to use mostly just homegrown tools. We had so many different criteria that we used to evaluate the SIEMs. Like we were thinking about what all products does it integrate with? Does it have nice dashboards? Does the language that it uses for searching, is it like comfortable to use? Do they have good training content on their website? Like all these various things that we listed out and we gave them weightings, right? We did like 100% score and there was like 17 categories all worth like 3 or 4% each. And in retrospect, I think we did that for the first SIEM migration. For the second one, we did something similar as well. And for the third one, you know, what we realized is just, let's just focus on the basics. Like, we got way too carried away trying to focus on a million features. You know, a lot of sims, especially older ones, have more features, but that doesn't necessarily mean they're good. Or even if they are good, it doesn't mean you'll necessarily use them. I think the thing we use 95% of our time is searching, detection writing, and then, you know, API capabilities. And we kind of just decided those are the most important things. So why don't we just pick a sim that does those things well? And then of course, like, price is always a factor. But what we find is that the older tools that do a million things tend to be more expensive as well, because they just are more robust and they have more teams working on features. But we just really don't care about them, maybe over here. Yeah, there's a few features I would say, and I'm not going to name any of our old sims by name, but some of them have these really fancy features that we realized that it takes so much time to set up. that it's either not worth the effort to set up or it's not worth the effort to maintain once you've set it up. And so our new motto is just keep it simple.

That's shocking to me in some ways of like, wait a minute, you're telling me that a more complex set of evaluation criteria might lead you to optimizing for the wrong things and that you could in fact just keep it simple and focus on what you actually need out of it?

Absolutely.

But you know what? This is actually not as simple because the only reason he and their team knew for sure what they want. That's a prerequisite for doing the top, the big 3 requirements. But knowing what you want may have required them to have the experience they had. Sorry, I'm going all philosophical and all that, but I think ultimately, if you know for sure, you can go the easy route. But to know for sure, you have to go the hard route a few times.

I heard something in the distance, actually, even though I have my studio headphones on. I heard in the distance a million analysts crying out in terror and suddenly silenced at the idea you don't need 17 checkboxes for evaluation.

Yeah, I mean, you definitely don't. I mean, I guess to your point, Anton, that is true. We did so many sim migrations that we had a good idea of what we wanted. But I would even say if you have not, like even if you're just using your one current sim, really try to think about, are you using every single feature that they— if you go on their website and you see every single feature they advertise, are you using all of them? Are, you know, all the fancy things that they say, look at these automatic extractions we can do for you and normalize your data, are you using that? Or at the end of the day, are you writing your own parsers because they work better when you write them? Are you writing your own detections because they work better when you write them? If that's the case, then you don't need any of the fancy out-of-box stuff anyway, do you?

That makes sense. And I think that boasting about the device integration list and all that starts to be kind of a bit of a strange game. So I digress, and it brings back scary memories of my childhood. So, why don't we focus on the—

Anton had a very strange childhood, listeners.

So, many SIEM projects have problems, and we hear about this. I've heard about it back in my analyst days. No way, right? Like, there's something broken. But in my mind, in my experience, people often assign the blame to a product, and they wanna migrate. But ultimately, something is broken in their process, or they just flat out don't have the team. Before the age of software as a service, SIEMs, sometimes the entire team was dedicated to keeping it up and nobody was using it. So, they were only judging how hard is it to keep up rather than how useful it is, all that stuff. But the challenge of when to migrate versus when to double down is a genuinely hard problem. And I've met people migrate too early, and I've met people who stayed for too long. So, help us decide when is it time to go. Sorry, I'm all romantic. When is it time to go?

No, I kind of love this question. This is the like, should I stay or should I go now? If I stay, it will be trouble. If I go, it will be double.

No, no, when you're sure you need to go.

Yeah, that's a good question. And I think what you said is kind of right in that it's— you don't want to just blame the tool. It'll never be as simple as, oh, this tool is bad, right? Otherwise, you would have never been using it in the first place, most likely. But what it usually tends to be is maybe there's a few things that the tool is not doing that you would like. And then you combine that with the fact that Maybe the pricing model is not working for your team very well and it's starting to get out of control. And combine that once more with, let's say your team is small and what I just talked about, some of these older products that they have a lot of features, but it takes time to set up these features. It takes time to maintain these things. Maybe you just realize my team is small, we don't have time to set up and maintain these features. I've definitely heard a lot of stories. I've read blog posts about some of these bigger companies that have like an entire team dedicated to just doing like log parsing and extraction, right? And then another team that's just focused on the testing data for the detections. That's great, but like Etsy's not that big of a company and certainly the security team's not that big. So sometimes you just say all these factors combined, we don't have the right team to support some of these type of tools and we don't just get enough value out of it. And so now it's time to move on and just try to pick, in our case, a simpler tool.

So I want to pick at this migration thing a little bit. I'm going to do the questions a little bit out of order because I— this is like just weighing on me so heavily. When you were migrating 3 times, how did you keep your detection content up in each platform? How did you do that? How did you not have visibility gaps and blind spots and all the issues that people worry about?

I would say the first time we did a migration, we really started from scratch again. Like, that was a bad time. That was like, we've lost our detections, we're writing new detections. I mean, of course we had a frame of reference for what our detections were, but we had to rewrite them in the new tool completely in a new language and from scratch. That was a bad time. But I think at that time when we were using homegrown SIEM, we kind of just decided that the detections we had were not very portable anywhere.

Huh.

Because the way we were doing it, um, maybe you're familiar with Airbnb's like StreamAlert. We had built a system similar to that based off of it. And essentially a lot of those detections are like SQL-based and they're based on streaming data. It's not really applicable as well to current SIEMs. So maybe I'm thinking at the time we just decided this is a lost cause anyway, we're going to have to rewrite all of these. It's okay. But for future migrations, we really started to invest heavily in detection as code.

Mm-hmm.

And certainly this is one that's helped us greatly in this recent migration. We— our previous platform, we had detection as code set up, and in our current one, which is Google, we have detection as code set up. And that's a lot easier to translate because, of course, the detections, the actual syntax has to change. But, you know, the way we store our rules in YARA, I'd say at least half of the YARA stays the same as far as metadata, MITRE category tagging, all of that good stuff. And you just have to learn the new syntax of the actual detection and change that part. But it was a lot easier this time for sure.

So wasn't there anybody who showed up out of the woodwork and said, hey, let's just write a converter, pile all the old content in there, crank the handle a few times, and then have new content come out? And what did you do to that person if that was there?

We did think about that actually. I think perhaps it wasn't— I think it's just not so easy to write a converter, to be honest. There's definitely at least one person on the team that suggested using AI to do the converter, and I was just like, please, let's not get too carried away here. I think where we're at is also we don't have like a million detections. You know, we have dozens of really good detections and then maybe another dozens of okay detections. When you're migrating, you have that chance to be like, we don't need to move the okay detections yet. Let's keep them in the backlog. But we have our good detections. Let's just move them over. It doesn't take that long to port them over, you know. You have to learn the new language, YARA-L in this case, anyway. You have to get comfortable with it anyway for the searching. So you may as well get some practice in writing the detections. I think that was part of the mentality. As we write the detections, we will self-train and get better at the searching angle as well.

Oh yeah, sure.

I suppose if we had a ton of detections, like hundreds of them, maybe we would have to invest the time into writing a converter. But honestly, it's easier said than done because it's not just a language difference. Like usually it's entirely structurally different in the logic itself. And at that point you can't just write a converter, right? So.

Sure. There's like a mindset difference between different languages. platforms even.

Yes. And this thing is just so— like, for us in this conversation, it took us a whopping 3 seconds to realize we have the shared understanding of this. But this is like years of advisory by all sorts of highly paid consultants to explain that point to many security leaders. So I'm like mildly depressed at this point.

Well, say the point again in case people missed it.

It's the point about that converting detections is much easier said than done because it's not only the code, But it's also the asset model, the logic, and even the mindset behind the products and the sort of like almost like spiritual descendants of the product designers matter in this case, how they thought it should run. And so, converter is really not a thing. And people show up with it and say, where's your conversion script? And you'd be like, okay, let's start this conversation. But the positive shocking part is that all of us here would be like, Yeah, that's for sure. But it's like so not for sure.

But I think there was another— that's all true. That's all completely true. There was one other interesting piece of lemonade in there, though, that Manan pointed out, which is the opportunity to practice and get familiar with the tool by doing the migration into the new spiritual framework, not just the new syntax. And I thought that was pretty interesting. People often miss out on the silver lining or the upside in an experience. And while I'm not as pathologically optimistic as some of the people I work with, I do try to see the best in the world. And that's pretty cool.

Okay. Another fun thing from the conversation was that you had this criteria or idea that you want a platform or a SIEM which would best enable your team to build on. And I thought, that's smart. And then immediately I thought, how do you test for that? Of course, you can say, does it have an API? And everybody's like, yeah, yeah, yeah, we have it. But how do you then test for that? Like, that's a useful criteria, but like, how do you make it practical?

That's such a good question. And the truth is, you can't test for it very easily in the span of time you usually have to try these products. I think we've been burned in the past for sure with older sims. The API documentation is very good. But then when we try to set up certain use cases, we realize that there's more blockers than we thought, or the API is more limited than we thought, at least. I think The best we can do, and I think what sold us on like Google SecOps in this case, is that we saw the detection as code pipeline that we're currently using was made by someone else. I think actually it is someone at Google, but it was made using the API. And the fact that it was, it looks like a much shorter amount of code. I almost like dropped the name of the previous SIEM, which I told my team I wouldn't do. But we use a detection as code pipeline for them as well, which is like 6 or 7 times as much code to do the same thing. And it was flakier and it was really frustrating. That's an example of us. We tried to build it. I actually made my own version of their detection as code pipeline, our previous SIEM, that I think was a lot better than theirs. And I was going to go maybe even talk about it at their conference, but never ended up happening. It's fine. We've moved on. The one we see for Google SecOps is significantly better. And to me, that was the indicator where I'm like, hey, people have their own projects that they've made that use the API and they use it in a powerful way. And it looks, it just looks like it's good. So that was kind of our mentality in this one. You can't really tell by testing, although we did, of course, test as well. We wrote some stuff too. It seems good on paper. So far, thus far, it's been good. But that one is a little bit more of a best guess. But the best thing you could do is just see what other people have built. If a lot of people are building things, it probably has a good API.

So I want to pick at this a little bit because this is such a good area and it's, it's so in line with how Google does things internally. Tell our listeners more about what detection as code means and what you've built around it and how that How does that make your life better? Share your wisdom, please.

Limit the response to 6 hours.

Yeah, yeah. No, detection as code, if you're not familiar at all, I guess, how can I start this? It's essentially instead of writing the logic of the detections and everything around it within your tool itself, you kind of adopt a code— like, it's like when you're writing code in GitHub, you know, code management. You want code reviews, you want people to learn off of each other, you want things to be reversible. So essentially it's that, right? You have your detection stored in a repo. You can do a lot of testing right before you even push it to the product within the repo itself to make sure people write them in the same style. They don't miss any important fields. They don't misconfigure anything. They don't accidentally set something to production mode before they've done testing first, right? You can get like very, very granular with all of that. And then you make sure that when someone's written a new detection, a bunch of others on the team get a chance to code review it and they say, okay, this looks good, or they have a chance to provide feedback. And then only once all the tests have passed, people have done their code reviews, you send it into the product. One thing we've not done yet, but we definitely want to do and we're working on soon, is even once you've pushed the detection into the product, like, do further testing. See if it's working as you expect. See if it made it to the right place. Yeah, it's just a lot better than having people write things right inside the UI where they can either be lost or people could make silly mistakes that just cause thousands of alerts to pop up. You paged someone at 3 AM by accident and then it's like, oh, what did that? I don't know because—

Yeah.

There's no trail of it anywhere. Let's just dig around in the tool.

Totally. And the last thing I want to do as a product manager of security tools is build a linter into my security tool. I want you to rely on the linter you already know and love and use everywhere else. So I love this framework of treating the tool more as the execution environment and less of the coding environment. I do think there's an interesting sort of boundary though of test mode. And so whether that's a soak test, Or a retro hunt, or just producing alerts in test mode so that your SOC starts to see them. You can play with your playbooks, see how your automation runs. Is that something you've seen handled well in any other tool? Is that something you'd like us to handle better? How are you doing the test mode today?

Yeah, I don't think a lot of tools handle test mode very well, to be honest. And I think that's a frustration we had with our previous tool as well. Something we asked one of our previous vendors was if they can give us essentially a second tenant Oh yeah, sure. Give us an entire second tenant so that when in our detection as code pipeline, everything that we write first gets like submitted there and we get a chance to see if it showed up in the UI correctly, right? All the obvious stuff. But then we could even send test data just to the second tenant to see if it triggers the alerts and move it over to the production.

Just a little high cost because it has to have the same data. If it doesn't have the same data—

What if you have 2 like tenants pointing at the same database underneath? Imagine that world.

Well, so actually the way we're doing this, so spoiler, we have a second tenant actually for Google SecOps. We have gotten that to happen, which is awesome.

Nice.

We have a dev tenant and what we're doing is we're not sending the production data to the dev tenant. What we're doing is just, we have essentially sample logs that we save. So like logs that we know were supposed to trigger certain detection in some cases, we don't always have that.

Oh, sure.

Sometimes we have to kind of like make logs up, otherwise we just don't have it. That's also okay. We're figuring that part out, but When we do have sample logs, we can send that, just the sample logs. So essentially right now the dev tenant just has a— index isn't the right word here, but whatever they call it, a feed of just sample logs. It just receives sample logs and that triggers the alert or it doesn't. And that's how we test.

And you're able to do both positive and negative testing on the detection content that way?

Yes, exactly.

Yeah, that's sick.

That is actually really cool. I mean, I'm not joking, but it's like, it's legit cool. And it's interesting how it's so obvious in retrospect, but like so Painfully not obvious until you hear about it, which is, I guess, how the best ideas originate, is then when they seem like dead obvious at the end, but like super not obvious in the beginning.

Wheels on luggage.

Yeah, like wheels on luggage. Oh, yeah.

Interesting.

True.

Never heard that one before, but that's cool.

Oh, we didn't put wheels on luggage until after we'd been to the moon.

Wow. Wow, that's cool.

That's how recent having wheels on your luggage is. Yeah, it's crazy.

So one more thing that blew our minds when we were sort of looking at the initial materials for this, and I'm almost like I'm afraid to say this because then people would Hundreds of people who I advised to the contrary would show up and beat me up, namely that you completed the SIEM migration in a week. And of course, I've written Gartner papers that—

Wait, you did what?

We completed the SIEM migration in a week. And of course, I wrote, you know, a dozen Gartner papers that say that you have to allocate 9 months for it at a large company. I mean, it doesn't matter, 6 months or whether it's even 4 months, whatnot, but it's not a week. And now, I have to eat all this paper, and some of them are 50 pages long. So, just like, High fiber diet. Save me from this mind explosion of migrating a SIEM at a fairly sizable company in a week. Like, like, help me here.

Yeah. So to be honest, I was also— so I talked with my coworkers and they were like, you're going to make everyone very upset if you say this, you know, everyone's going to get pressure from their management now that they have to do it in a week. So I hope I've not ruined anyone's life. I'm really sorry, guys. But yeah, what we essentially did in line with our, you know, we have such a strong engineering culture at Etsy. It's funny because we listened to a lot of— I've listened to a lot of discussions on your podcast as well, people talking about whether security teams really need to be like true engineers and stuff. We definitely are. And so, you know, we did the engineering thing and we turned it into a hackathon project. That's why we did it in a week is we—

Wow.

We did a lot of pre-planning ahead of time. So I guess it wouldn't necessarily be fair to say the process was only a week, right? Because we knew we were making this transition. We knew the hackathon was, let's say, a couple months out and we spent a lot of time pre-preparing. Not only listing out all the different data sources and content we had, like detections, but in many cases, like even before we got the environment of our new SIEM, we had either documentation links or in some cases I even had a lot of Terraform code prepared and ready to go. You know, Google SecOps uses Pub/Sub, for example, very— it's very easy to ingest from Pub/Sub. I had the Terraform ready for like 85% of our log types so that as soon as it's available, just apply the Terraform change and all those Pub/Sub pipelines are up and we just have to do whatever little bit you have to do in the UI. So essentially when the week actually came, we call it Code Mosaic. That's our like hackathon we do once a year at Etsy. We all just had, we had the spreadsheet of all the different log types. We already had Terraform code prepared. So we're really just applying things, making sure they come in properly, applying the right parsers to them, stuff like that. And yeah, by the end of the week we had all of our log types in, which is a lot of log types.

That's amazing.

Yeah. The detections of course take a little bit more time. So yeah, I'll be fair and say we didn't have all the detections in right away, but we at least had an idea of You know, we had the detection as code pipeline set up by the end of the week, and we were kind of starting to get that process going.

That's really incredible. I, I love hearing stories like this. I do think you are going to make all of the other, like, SOC directors out there jealous, and they're going to be, you know, saying to their SOC teams, hey, they did it, why can't you?

Yeah, well, I have a message for them.

But I think the prep work matters.

Oh, what's your message for them?

My message for the other SOC directors is just, hey, listen, like, Be realistic with— this is me shilling Etsy a bit too, because I love Etsy. We have a very strong engineering culture. We have a very strong engineering team. That's why things like this are possible. And our management made space for us to make this our hackathon project. So if you're not doing all of the above, if you've not built that culture, you can't just expect magic to happen on its own.

That's a great answer.

I think that reminds me of a lot of magical answers from our security when people, you know, come to us, specifically to my team, and they say, hey, how does Google do it? And we're like, we can tell you, but like, there's this other stuff that took us 10 years to build. And it's a lot of this is like freaking built from silicon up. And so, yeah, a lot of magic here would work really magically if you do this. But unless you do this, this magic is like, what is this? How can it ever work?

Well—

Yeah, you need hardware root of trust on every motherboard in your data center. Then you are starting off well.

Yeah, the magic suddenly works, right? Like, yeah, sure. Okay, so we digress. Tim, go with a sad Closing question.

I hate to do this because this has been— we're actually recording several episodes on this day, and I would be shocked if this is not the most fun episode we do, not just today but all month. And I hate to do it, but I have to ask our traditional closing questions. One, do you have advice for people on SIEM migration or choosing a SIEM or, you know, choosing SecOps? And then 2, do you have recommended reading?

All right, so first question, I guess I'll stick to what I said at the beginning, which is keep it simple. I think the lesson we've learned is trying to classify 25 different things that you need in a SIEM, you know, perhaps if you're a really, really big company with many people invested, that could work. But for most companies, I would say keep it simple. Just focus on the core functionality a SIEM is supposed to do— searching, detection, writing, things like that— and make sure that that works extremely well because that's just 85% of where the time is going to go in the first place. For recommended reading, I should have prepared some answers, shouldn't I have? Because I don't have anything off the top of my head.

It can be a fiction book.

You're not allowed to say Anton's blog. That goes in the swear jar.

Okay, let me think. Recommended reading.

Oh, we've had literally everything. We've had Mary Oliver, we've had The Art of War.

The Art of War is really good. I'll back The Art of War. Oh, I'm reading right now, I'm reading a book called Ancillary Justice by Ann Leckie. Really, really good book. Absolutely nothing to do with sim migration, but it does kind of talk about AI, which I know AI is a hot topic these days. So I guess that's— we'll go with that.

Love it. Well, with that, listeners, I think this is pretty far from ancillary justice. I think this is like Really good justice for SIEM migration advice everywhere. Manan, thank you so much for joining us.

Oh, thanks for having me. Really appreciate it.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter @Anton_Chuvakin and @_TimPico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Thank you, everybody.