EP250 The End of "Collect Everything"? Moving from Centralization to Data Access?

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Tim Peacock, the Senior PM for Detection, Risk Analytics, and UEBA in Google SecOps, and of course, Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this show wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want to get it piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners on the Google Cloud Security Community page. Anton, we are talking about not pipelines today. Is it oil tankers instead? What are we talking about?

We are talking about not pipelines, yes. And the reason we are talking about not pipelines is because we think that the pipelines episode would not have broad appeal. Would you agree? I think that not pipelines is a much broader topic than pipelines.

I think, definitionally, that's the case.

You have to agree.

Yeah, yeah. I have to agree.

Listeners, this is, you know, a quantum physics PhD, 20 years of Gartner. You have to agree, not pipelines is broader than pipelines. Can't argue with that. So what are we talking about, though? Because I think this episode had a couple of moments where you had a, like, once-in-a-year eureka moment today, didn't you?

Correct. And I think that the issue with this this is that however I describe it, it just doesn't sound as glamorous as what actually transpired. So, my next hypothesis was—

Oh, then don't describe it.

I'm gonna still try. Again, I have 8 years.

Okay, try. What do you got?

8 years at Gartner, not 20. But it does talk about the data collection and inputs for SIEM.

Yes.

But it's also kind of narrow. It's really bigger than that. I think, okay, okay, I'm gonna try one more time. Last one. It's about a modern way to get value out of SIEM. Nah, now it's too broad. Sorry, now it's annoyingly broad. Fine.

Listeners, we promise this is a great episode, but maybe rather than try to explain why, let's just turn things over to today's guest. Today we are joined by Balazs Scheidler, CEO at Axoflow and the original founder of SyslogNG. Balazs, thank you so much for joining us today. We are delighted to have you. Anton, do you want to ask the first question today? I feel like I always do it and it would be fun to make you do it for a change.

You know, and it's a question, and it's like, think about it as a meta question or an uber question. Because we— okay, I'm gonna go into banter with myself mode briefly. Because when we were planning this—

Oh, boy, listeners, Anton is talking to himself now.

Well, yes, through— well, I guess I'm addressing the audience, meaning you 2 guys. But the point is that when we were planning this episode, we were sort of like talking to each other and thinking, oh, this would be a pipelines episode. And do we really need another pipeline episode? And Tim said, Actually, we do need another pipeline episode because our previous pipeline episode is just performing so well. And I thought, no, I checked, and yeah, it kind of does. So maybe we do need another pipeline episode. But how about we start from the motivation and problems? Like, the whole pipeline thing, log pipelines, came from a belief or a view that instead of centralizing all the data and collecting it in one place, we can just provide access to data. And I don't know what to think about it. I'm nervous almost every time I hear it, but I'm also realizing that it's needed. So, what is your response to this sort of broad philosophical question about switching from centralizing log data and telemetry to access to data?

Yeah. I mean, we have centralized data forever, right? I wrote syslog-ng, and that was used for centralizing data in the '90s. And then we started using SIEMs and log management systems, and they all centralize data. And I think there are good reasons for centralizing. So, I mean, compliance is one of them, but also detections is the key reason we centralize data because we do correlation and, well, detections on a central repository.

Mm-hmm.

But I mean, today, I think bandwidths have increased a lot. Also, data management practices have evolved a lot. So, we might not need centralization, or we might not need a single repository. Maybe we need some repositories, like, you know, we have branch offices, we have data centers, we have central repositories, and maybe that should work in concert.

Mm-hmm.

What I'm still believing is that we need to get the data off the data sources, because that's not going to cut it, especially from a compliance perspective. But other than that, data could be anywhere, in transit or at rest. It doesn't really matter.

So, this is a bit— yeah, I think thanks for making the subtle point about you're not just suggesting we leave the data be and then we magically query it when we need it. You're describing something other than hope for the best, leave the data at source, and then distributed queries.

I think some people are describing that these days. Some people are describing that.

I feel like I've been seeing that pitch for a while now. Yeah. Moreover, lately, and again, I'm not going on another end. But they believe that the power of AI agents guarantees them that if the data is left at source, it would magically manifest where they need it because agents.

Balazs, what do you think about that?

Okay, God, fine. Yeah, let's say I agree.

I obviously disagree. So data sources usually have— they are not aimed at or designed to keep a lot of data. So they do have some storage usually, like maybe even a few gigs, but the amounts we produce is just not going to cut it. So data will be lost if you just leave it there. And that would not give us enough retention even for our detection purposes, let alone compliance. So we need to get the data off the data sources and put it somewhere. What I'm arguing is that at least some of the data doesn't need to be centralized. So debug logs, stuff we use for troubleshooting network incidents or— You know, I temporarily increased the log levels of various applications because this and this. We don't need that in the scene. That's what I'm arguing. And we need a space or something to store these kinds of data and obviously provide access to it, if that makes sense.

It does. It does. And it seems like—

It does, but would it push us? Yeah, go ahead.

Go ahead, Anton.

No, I was just afraid that you'd push us back into this whole, what do the people call it, land of lakes, data lakehouse. where they kind of hope the— I don't know, there was some kind of bizarre metaphors for this type of semi-distributed where the data is where it's needed, but not where it was. Is it?

I mean, it's not— I mean, the data is where the data sources generate them. So, if we leave it right next to that in a local temporal, whatever, cold storage, that's not a data lake. That's just a temporary storage for data that is already there. And then obviously you need federated search to make it useful. So that's the access part. And you can obviously rehydrate or move data from these local distributed storages, maybe even to the central storage because you need it for whatever reason. But the point is that we need a way to manage this kind of data, where we store it and how we move it. And it needs to be very easy. The problem with data pipelines, I mean, of the past, I mean, we call those log management systems.

Yeah, let's talk about pipelines because it sounds like you're about to describe pipelines. So tell us about the issues with pipelines of the past.

I mean, I wrote syslog-ng and that was considered a pipeline by many. I mean, people tried to use it for that purpose. But I think the key ingredient for a pipeline is not necessarily the ability to run regexes or transform data, but that's a key— I mean, we can talk about that too. but also the management aspect. So pipelines go into all locations of your organizations, impact every function in your organization. I mean, like various departments and stuff like that.

Mm-hmm.

So it is really a complex system. And previous pipelines or, you know, log management systems were really static. You needed to go there, change configuration, maybe even manually a number of times because you had 50 locations and you needed to configure all of those. So, I think the key aspect, or one of the key aspects of pipelines, is management. And, you know, we used to manage IT in a way 20 years ago, and we do it differently now. So, what I'm arguing is that the way we manage fleets of compute today should be applied to pipelines too. So, that's what I'm trying to describe when I'm talking about management.

Mm-hmm.

And that can include data at rest or data in motion. We don't really need to care if that's managed right.

Okay. So is this almost a cattle, not pets argument for how we're doing data pipeline management? Are we trying to get away from hard-to-manage manual configuration? Where are we going with this?

Yeah. I mean, automation, we do a lot of declarative methods for managing compute in Kubernetes or in the cloud. But still, when we go and configure pipelines, we— I mean, just check any of the pipeline vendors, and they will tell you to open port 514 and 515 and 516 and 517 to take care of various pieces of your infrastructure. And then you will need to go out and, I don't know, configure your Zscaler and Palo Alto and whatever devices to send to a specific port.

Mm-hmm.

And what happens if the port is you know, incorrect. It'll ingest it into the incorrect index, source type, whatever log type in SecOps. And that—

This is annoyingly manual.

Yeah, it's all manual. Yeah, yeah. That's— and I think I recently read one of your articles about output-driven SIEM. And I think there is a lot of problems around data collection and, I mean, managing data collection. And part of that is it's too manual. And we just don't know how to do it. And therefore, it's easier to collect them all rather than go out when we need them. I mean, need the data.

I feel like this is very useful, and I feel like people are listening to this and they're nodding and agreeing, but practicality of this is also very difficult. So, imagine that you have certain data sources where I want to retain data. For example, I want to have a guaranteed retention period. And there's other datasets where I want almost near real-time streaming. And there's yet another dataset where I want just for it to be there when I need it, but not necessarily care about the time. And I'm assuming that your technology and similar technologies deliver that. But deciding how to do it is a little tricky. And if it's manual configure logs by port, I'd rather go back to centralization. Sorry, guys. Like, I don't want that.

I think there is one. Whereas we built data governance for business data, we didn't do that for security data. And what does any data governance program start with? Identifying or classifying the data that you get. That's what we don't have with security data.

Mm-hmm.

You just get bytes on the wire. And then you need to go and, you know, tell using a regex or whatever search query, or I don't know what, to declare your retention period, for example. It's a very, I mean, a policy-level requirement is translated into very low-level regular expressions, and that doesn't work. So we need some bridge between those 2 layers so that it becomes usable. And I think the key element here is classification. And that's what we didn't or never really had for security data. And that's what we actually created so that by the end of the day, you can say stuff like, okay, I want my— I don't know. I mean, I'm going very, very high level now. I want my security data into my SIEM.

Right.

Okay? And that's a label whether something is security-related. And I know the complexities of that decision. But we can peel off the layers of the onion and go like, okay, my firewalls, any kind of firewall log is security data. So I have like 10 different firewalls, and it's easy to map that to the security bucket with that. So classification, understanding if data is coming from a Windows endpoint, a Linux server, a Palo Alto firewall, or a Zscaler is crucial And it needs to be automated. If the customer—

Because I'm not sorting through Windows events to decide security or not security. That's just, it happens. People come up with these questions and say, Anton, give me a list of security events from Windows.

Oh no.

And I'm like, it's for real. And they, because they don't want to retain the non-security events, but these are like agonizing decisions that nobody wants to make.

You know, people used to ask me to do that for cloud logs too. It was very painful.

I think history repeats itself. So cloud logs are not really different. The crucial element here is that no one wants to do the work. I mean, this is a lot of knowledge related to Windows logs or Palo Alto logs or Zscaler or whatever.

Yeah.

And I, as an organization, I'm running with a security stack that has 100 different security tools, and I would need to understand each and every one of those with their own respective schema or ideas about logging. Which is different, obviously.

And you have to stay up to date as those things change.

Absolutely. So, who is taking up on that challenge? I mean, data sources, I don't think, I don't know, 2 vendors of the firewall space, they would agree on a schema. They will just do whatever they did in the last 20 years. I mean, it seems they do. I mean, I know that SecOps does this, and I know that Splunk does it to a certain extent. But, you know, as long as the incentives are set in a way that if you send me more data, you pay me more, then after a point, you just don't care. Is my Apollo data correct? I mean, I can list you, like, in just the last 4 weeks, we have found fatal issues related to widespread security devices. that is actually collected into a SIEM and would completely prevent any detection to be done on that. And people have been doing this for years. No one cares. So—

Yeah.

What I'm trying to say is that there needs to be some piece of the infrastructure responsible for data quality, finding the issues with the data, highlighting it, showing that it's a problem, And obviously fixing most of it, because if we expect everyone to fix data sources or parsing rules in the SIEM, that's not going to happen. That's why we are where we are today, and we couldn't solve this in the last 20 years, at least.

So why can we solve it now?

Well, first of all, I think with the product we created, we can prove its possible.

Hmm.

Okay? Because previously, this was just the problem of the customer. So you should do it. Fix the regex and it's going to be done. And, you know, you do that once and then twice and then, you know, for the 100 different data sources that you have. And some of that will probably work. I mean, but there will be quirks and problems all the time. I have anecdotes here, but I don't want to bore you too much. What kind of— No, no, no, no.

These are useful. Yeah, yeah. That's actually best.

I think that these are—

yeah. I was about to ask for stories.

Yes. You know, there is a— one story is that some of the SIEMs out there, they prefer CEF. I know that you know CEF, and I know that SecOps does it too sometimes. I mean, there are alternatives. And you can set, for example, Palo Alto, to produce CEF format. And there is even an official documentation how to do that. Okay? You download the PDF, you set up Palo Alto, and then it produces CEF. And then if you do what is described in the PDF, obviously manually, because you need to copy-paste commands there, the log message that is created is incorrect. It doesn't meet CEF specifications. specification at all. It has multiple lines in them. CEF has key values, and sometimes there are 2 key values that have the same name with a different value, and it's clearly a mistake. And also, Palo Alto truncates log messages at 2,048 bytes.

Hmm.

And so it means that the data that you send to the SIEM is truncated. But then framing, I don't know how much you know about framing in syslog. Anyway, that means they will send the right size and then truncate the message, meaning that it completely gets out of sync. So every data you ingest into the SIEM is bogus, or most of it anyway, completely unparsable.

That's depressing news for 2025. I feel like this story is just perfect if we had— if the calendar would show something like 2004, maybe. Like, I don't know. Like, I'd be like, oh yes, this log is too short. Messages get truncated. It's so sad. And you and me would look at each other and say, yeah, it's kind of sad. But I don't want this in 25. That's just insane. That is just utterly insane.

It is. And it is still happening. We find all these examples all the time. And the right example, I mean, What do we do about it as an industry? And I see a lot of tools that reject invalid data, but is that a good solution?

Oh.

No.

No.

Because then I'm losing the data, right? So I need to fix it up, but who is fixing it up?

Man, it's sad how desperate defenders are for data that we will take whatever we can get.

Yeah.

Yeah. Anyway, so I think the pipeline fixes it up, obviously. Because they are the only ones incentivized to do it.

So I am having an eureka moment, mind blown emoji, whatever.

He doesn't have these every day. This is big.

Yeah, no, no, I don't have them every year. And I feel like in a lot of cases, and I may name your competitor by name, but a certain vendor that name starts with a C would be presenting to SIM users and saying, save on your SIM bill, save on your SIM bill. And in everybody's head, Pipelines are often associated with using vendor C to cut the bill from vendor S. That's the entirety of the universe. That's what I've heard back in my analyst days. Buy the C to cut the bill on S. And I'm like, this isn't very strategic, is it?

No, I agree.

But your point almost made me see suddenly the higher-level value in pipelines beyond Cut the bill of ass using C. I don't know why, what got into me, but the point is that this is actually genuinely interesting.

You know the best part? If you start looking at your data and fixing up these kinds of issues, I mean, truncation, who cares? But yeah, if they do happen, volume solves itself because you will see stuff that's completely irrelevant to your security operations. And you don't want it. You may want it in an archive or a second tier of your storage. And then, you know, your compliance criteria is still a check, but you don't want it into your security operations because you don't need it for anything.

Mm-hmm.

And by the way, I think that cut the costs opened our eyes to new possibilities. Because people would be doing the same things they were doing in the last 20 years if this wasn't a problem.

Yes.

So in a way, I'm really grateful for this whole idea because this way we have a chance to redo a bit how we manage security data. And I mean, you asked why there is a solution now. I mean, yeah, because we came around and we can show that this is actually possible. And I'm sure that, I mean, other pipelines may see the value in this and replicate what we've ever done. But the cool part is data access, going back to the initial question.

I like that. And I think that solving the data quality for security telemetry is kind of a huge value in pipelines. It is not just cut the bill, or it's not just Reduce the volume by, well, cut the bill usually by aggregation, whatever.

Usually by reducing the volume.

But the quality, there's finally— and, you know, you know what, I would now admit to being a little bit part of the problem because the very blog you mentioned, the output-driven SIEM, it just makes pointers that you need to collect what you would use. But the decision to each log, it shoves back to a client. So, if I say, you have a SIEM, you have these use cases, these are your risks, and you need to collect logs that power those use cases, but how do I identify those logs and make sure those logs arrive alive, so to say, it's kind of your problem, buddy. And I feel like somebody has to build the tech to solve that. And I feel like pipelines are that tech and not just like cut-the-bill tech.

Yes. And, you know, the reason output-driven SIEM is so difficult, because this whole feedback loop, I do detect, I have a detection problem. I mean, my metrics are wrong. So impacting the data collection piece is just too difficult. Just imagine that I need to set up port numbers where I can set data and then go out to devices to change them to— so these things are too manual, too much work. They don't do it. That's why I said in the beginning that if this thing remains this difficult, we are never going to reach the output-driven scene. It's much easier to collect everything. That's what we do today.

Hmm.

And if I want to add another thing, so fixing up— Formats. I mean, I agree, that's crucial because we have all these examples. I mean, I have a war chest where I store these stories.

Okay.

But the other thing is normalization, right? And the schema. I mean, SecOps uses UDM, Elastic uses ECS, Splunk uses CIM, OCSF is in AWS Security Lake. There was CEF, there is LEAF. I mean, I can name you. A number of schemas out there. And the problem is that detection really is much better if you have schematized data, so data in an actual schema.

Mm-hmm.

But if you have classification automatically, you know what kind of data source you are processing, and you store that along with the raw data, you can always map to the right schema from that point. So this is not a schema-to-schema mapping. So not like, okay, CEF to OCSF, I don't think that would ever work, or OCSF to UDM, I don't think that works either. But if you have the source data, you have classification, you understand what it is, you can map it to anything. Going back to data access. So access to data wherever it is at rest or in transit in the schema you want it to be.

That's the dream, right?

But that's close to reality at this point. I feel like that's the dream is a good 2015 comment, but I think we are getting there, which is kind of interesting to me. You want to touch on the enrichment as well? And then we'll have to close because I feel like enrichment— the reason I'm going to touch on enrichment is this. I feel like it's SIEM's job and not the pipeline job. Convince me otherwise.

Okay. So, I think there are a number of reasons why enrichment in the SIEM makes sense. So I'm an analyst, I'm looking at an incident, and I want to know, I don't know, some additional data point that is slow to query. I can't do that at, I don't know, 50 terabytes a day ingestion. But I mean, let's take another enrichment example. I know that this server is in that data center, in the 4th row of racks, and at the unit whatever.

Mm-hmm.

That's very easy to enrich while the data is in the pipeline and actually nearly impossible to do that at the SIEM level.

Yes, I'm convinced by 2/3, yes. I think that you gave it a good shove and shifted me from this is SIEM's job to, hmm, okay, yeah. That's fair. And also it's not portable if you leave it with a SIEM, you kill portability.

Not just that. And that's why it's the pipeline. Moving stuff left makes a lot of sense because if you collect the data in the same data center as your data source is, you can still trust its IP address, which is not true when the data hits the SIEM. Also, so there is a lot of information available in real time, which is lost When it is ingested into the SIEM. So you cannot go back, basically. So I think the solution there is to enrich at the ingestion level, so at the pipeline level, and ingest that information into the SIEM as a part of the log message or the data entry.

That makes sense to me. I think I'm more than 2/3 convinced, maybe more like 4/5, but I hate to do this. I have to ask our traditional closing questions. One, do you have a piece of advice for people who want to do more than save on their SIEM costs? And 2, do you have recommended reading? And that can be anything except Anton's blog.

Which we already did.

Oh, yeah, that's true. Oh, boy.

I think, please, look at what you ingest. Just one, the biggest data source or the most important one or whatever. Try to understand what you get there. Not from the perspective, is it large, is it small? But I think there's a lot of learning once you go and see what's there. So, observability or seeing what the data you ingest is crucial. And a lot of people just don't understand what is there. Reading. Yeah, Anton's blog is a trivial example.

He was about to go there.

Okay. About this stuff. No, I can't give you very good pointers.

Well, you also have a podcast, right? So I think it may count as reading, right? Well, we'll add the link anyway.

Yes.

Yes. I mean, I do have a podcast and I try to explain whatever we try to explain here as well. And Anton was a great guest a few weeks ago. So yeah, I mean, that's where we are trying to explain how we think this should be done. And I mean, again, I've been doing this for 27 years. I mean, syslog-ng, I started in 1998 and was part of log management. And, you know, we were satisfied with just moving bits and we are not satisfied with that anymore.

Perfect.

Well, Balazs, thank you so much for joining us today. We recently had a guest who's been doing something for 25 years. It's even more fun to have somebody who's been doing something for 27 years.

Yeah. I mean, this was a great discussion.

Perfect.

Yeah, I really, I really like it.

Well, thanks for being on the show. Yeah.

And I hope you liked it too. Yep, definitely.

And now we are out of time. Thank you very much for listening and of course for subscribing. Please subscribe so you can get new episodes piping hot. And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcast. You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us. And if we like or hate what we hear, we can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast by Google.