EP239 Linux Security: The Detection and Response Disconnect and Where Is My Agentless EDR

Security running.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Tim Peacock, the Senior PM for Detection, UEBA, and Risk in Google SecOps, and of course, Anton Chuvakin, a reformed analyst and now senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us, and the rest of our Cloud Security Podcast listeners all on the Google Cloud Security Community page. Anton, this is tragically an episode without you.

Correct, but it's also my fault. Nothing terrible happened. I kind of like sort of like missed it, and I felt like it maybe is a good opportunity for you to record it and see what happens.

Well, I think it turned out to be pretty fun.

Yeah, we can add the jokes. At this stage, at the opener stage.

Yes. So, we're talking about agentless EDR. As the inventor of EDR, how do you feel about that?

I remember thinking about agentless EDR back in the old days. And weirdly enough, in the early years, there were a couple of companies that tried to create agentless EDR. And even more weirdly enough, half of my colleagues at the time, half of Gartner people thought that agentless EDR is not a thing. and it should not be called EDR. And the other half thought that actually it is EDR. And it was kind of very interesting because the point of the opposite party was that if you don't collect certain telemetry, like if you absolutely cannot collect certain telemetry, then you're not an EDR. So agentlessly you cannot collect live memory.

Sure.

Not that many EDRs do it, by the way.

Oh, that's not true. You just can't continuously collect it.

Yes, and actually even from like regular EDRs, very few do that.

Yes.

So, but they pointed out some other esoteric example, Maybe it wasn't live memory. And they said, oh, see, you can't collect that, so you're not EDR. To me, that's not a good argument. I almost, I believe in agentless EDR because I want to believe in agentless EDR.

Oh, boy.

And maybe I'm wrong, but I think that the idea of observing endpoint telemetry under constraints of remote access will give you some pretty massive Cons, not just pros, but there would be some pros.

Yes.

So I feel like if you're intimidated by the massive cons and you think that's not EDR, that's your choice. I feel like there are pros and cons, but then the result is still EDR.

Yeah, that's fair. I believe that. So maybe with this brief discussion of whether agentless EDR is a thing and our agreement that it is a thing, let's turn things over to somebody who's created agentless EDR. I am delighted to introduce today's guest. This episode has been a long time coming. We are delighted to introduce Craig Rowland, the founder and CEO of Sandfly Security. Craig, I'm so happy we're finally making this happen. You know, we, we spend so much time as an industry talking about, you know, Windows EDR, and I feel like we spend so little time as an industry talking about Linux security monitoring, especially for like cloud and hybrid. And so Tell us about your theory of defense and what you're doing there and why it's not really an agent, but it's kind of an agent. Like, tell us the Sandfly story, please.

Yeah, sure. So the basic premise that we operate under is that Linux is an umbrella term. Okay. Right. So it means everything from a big GPU cluster up in the cloud down to a Raspberry Pi someone shoved in a broom closet somewhere. Right. So what you typically run into when someone says, oh, we're going to protect Linux, we need to protect Linux, you run into a situation where it comes to What distribution, what kernel, what CPU, how much memory does the device have, and things of this nature. So even if you say, well, you know, we're going to run it up in the cloud, frequently what we find, the organizations often have very mixed environments.

Yeah.

So even giving an example in a data center, yeah, you're watching the hardware instances you have, but the data center cameras might be running Linux. And there was recently a ransomware attack that used an IP camera in order to do all the encryption because Even a camera is running a full-blown version of Linux and it was able to mount the Windows shares and perform the operation.

No way.

Yeah, Linux is a very broad term, right? And even the smallest camera can operate with nearly identical feature set that a server does. So it needs to be watched in an identical way. Plus you get also strange use cases too, where Linux can be operated in a variety of environments such as robotics. One of our customers is using remotely driven vehicles now. So we're watching that type of stuff. So Linux transitions a lot from between cloud to on-prem, and even this, I don't even wanna call on-prem, you know what, you know, there's a cell phone base station up the hill for me. It's guaranteed, it's almost certainly running Linux, you know, so I guess it's cloud, but it's on-prem 'cause it's at the top of a hill, right? So you get these weird environments where Linux tends to permeate everything around us.

So how do you protect the cell phone station on top of the hill?

When we looked at the problem, one of the issues we run into is that getting an agent that ties into the kernel, or even when using eBPF across all these systems in a compatible way, very difficult. So I said to myself, what if I just got rid of the agent, right?

Okay.

What do we gain and what do we lose? What we lose is we might not get real-time visibility into all the systems, but what we gain is total complete compatibility. So we could work on systems that are up to a decade old all the way through modern systems. We could work on Intel, AMD, ARM MIPS, IBM Power, and IBM S390 CPUs as well. And also we have total stability. So basically you could update those systems as much as you want. It's never going to affect us. And more importantly, we're never going to affect those systems. So for customers who are running mission-critical environments, we deploy without the agent. And what that means is they could do it in a very safe way. And frequently they could watch many, many, many more systems than they typically can with a traditional endpoint agent.

So if it's agentless, what is it? What are you actually doing?

What are we doing? We come in over SSH. We use a small Golang binary, which I'm sure you at Google will be happy to hear that we are huge fans of Golang.

Love it.

Our entire backend is written in Golang as well. And this binary is basically written specifically to investigate Linux systems and look for signs of attack, people hiding from us, Linux forensics and things like that. We instruct it what we want it to do in terms of collecting that evidence and data. And then we bring it back off that system basically for analysis to tell customers if we see anything wrong. We could do traditional EDR, but we also do SSH key tracking. So we'll track if SSH keys are moving about. password auditing and so-called drift detection, which is our ability to spot any change that would happen to that remote system, be it a new process, a new user added, new systemd service spins up, anything like that as well. So it's kind of a, it's more than just EDR. And then part of the benefit we get, even though we lose real-time visibility, the benefit we get is we get all these other components that you don't typically want or even can use real-time on. And that allows us to profile a lot of different things.

So help me understand the threat landscape here, because I think Some of our listeners might think that Linux is more secure or less malware targeting it than say Windows environments. Could you break down kind of the differences between how malware tends to operate in Windows world versus Linux world? Because I feel like this is something that's kind of opaque to a lot of people.

It is. Linux is a bit of the, you know, that ignored stepchild, right? Because people frequently in the InfoSec world, they spend a lot of time and money training people and dealing with Windows malware. And as a result, you frequently have very understaffed security teams in terms of watching after the Linux deployment. So the worst I ever heard, this was several years ago, we were talking to someone, they had a team of 3 people and they were responsible for watching around 100,000 Linux VMs.

Really?

Right? So this is the scale of badness that some of these companies, I think it's fixed now, but this at the time is what they were dealing with. So what you end up with is situations where people aren't monitoring Linux. And in a way, in a bit of human nature, you do what you're used to doing. So if you don't have anyone who understands Linux, they just, they won't look at it, right? So this is something we run into. Linux also operates under this premise of it's more secure. That's not true, right? Sure, if you put up a Debian host and somebody run an SSH, it's fine, but nobody deploys like that, right?

Right.

They put on products, they put on services, they make changes. There are lots of people on those boxes operating. Changes are being pushed out to production that developers might have done something to. So it rapidly disintegrates into something that, yeah, it would start out secure, but you have to keep an eye on it. And then like we talked about earlier, you frequently get a lot of these corner case devices that nobody is watching. So, you know, we had one customer, for instance, they had command and control software running on their Synology NAS, right? Because—

Of course.

Nobody's looking. And if you're an attacker, why wouldn't you go to the edge? Why would I want to go to an endpoint that's heavily monitored on Windows when I could go to the network attached storage system that has all the data for the company anyway and went to steal it.

Yeah, it's got all the data anyway, why not attack it?

Exactly. It's the same thing for VPN routers. We've seen that as well. They go to the VPN router, it's a choke point for the data before it's encrypted. So this is the type of thing we run into with Linux a lot.

Okay, so Linux to me is very interesting because in Windows land we talk about living off the land, but in Linux you have such a rich set of tools at your attacker's disposal. What are you seeing as some of the subtly abused or interestingly abused? What are some of the neat things you're seeing, maybe, is a fun way to ask that question.

Yeah. The way I explain it to people is Linux is not an operating system. Linux is a programming language.

Oh, that's nice.

Once you think of it from that perspective, you could chain together a lot of benign commands into something that's very malicious. You could build an entire scanner based off Linux command line tools. So you do get traditional malware on Linux the way you do on Windows, but the malware tends to be open source, so it frequently changes. Or it might just be done in a way that is not— it's very gray area, right? It's like, was that an admin who did that wget command, or was that malicious, right? It becomes very involved in terms of investigating it. So what we see with attackers, a lot of times when you go into a Linux system, part of the security comes from the fact that each Linux deployment is very bespoke to that company.

Oh, that's interesting. Say more about that.

Yeah. Yeah. So, you know, I'm sure Google, for instance, you have a very unique way that you deploy Linux systems. And that's the same way at other companies. Other companies don't have a unique way at all. They might just be throwing everything, everything into the blender and just letting it go because they're legacy companies and they buy other companies, they roll it all in. So what attackers frequently will have to do is maybe kind of move a little bit slower. It's harder, I think, for these automated ransomware attacks to get a toehold on Linux. And it's not— people think, is that because of the better security? I'm like, no, No, it's just because Linux doesn't work with itself, right? It's hard to find compatibility across a lot of different systems. So a lot of times what they'll do, they'll have to move on a host and do things a little bit more custom. They're also what you see living off the land will be just stolen keys, stolen credentials, stolen SSH keys. You know, you get onto the first host, they immediately will crawl the hard drive looking for private keys that someone's left around, and then they're off to the races. They're going to look at your SSH config, your history files. They're going to see where that user is connecting to. And then at that point they turn into a ghost because they look like a regular user. So the living off the land attack might be as simple as a find command on Linux. They just, and they find SSH key and they go. So we tend to see a blend of things. And then you do see some more advanced attackers, people like China was associated with some backdoors called BPFdoor that's been hanging around on telcos and stuff.

What is BPFdoor?

BPFdoor, boy, we first wrote about it in 2022, but it's come back again. It's been targeting a lot of telcos, a lot of critical infrastructure, and it uses a Berkeley packet filter. to look for magic packets. And then when the magic packets seen coming to the host, it activates the backdoor.

Oh, no way.

Yeah. And what's interesting about it is that let's say you have a firewall on port 443 to let in your encrypted traffic. You're like, hey, I'm good. If BPF Doors on that host, it's monitoring all the traffic coming to that host. If a magic packet goes to port 443, it'll say, right, someone wants to talk to me. It will reconfigure the firewall to forward all the packets from that attacker on port 443 to a shell that it opens at a higher port. So even if you're monitoring port 443 traffic in this example, it's going to look like regular traffic, but it's really being forwarded to the backdoor.

Wow.

But it's using the Berkeley Packet Filter as kind of a front-end filter to make sure it's not watching everything. So that's just one example, but there's lots of similar backdoors that do kind of variants of these things. But you'll find, I think, a blend of stuff tends to be hitting the Linux systems. And some of it's designed for specific tasks. Some of it might be stealing SSH keys. Some of it might be just to target a specific VPN router or things like that.

So that's such a great example of how you can use BPF for naughtiness. Listeners, you might not realize this, but Craig and I are in some ways like brothers in arms in having built agentless Linux security tools. I built a capability here at Google that we call VMTD a couple of years ago. And it was, you know, using the hypervisor access we have to look for malware at runtime in Linux hosts. Craig, when you were talking to customers, because I ran into this question all the time, when you talk to your customers, how do you help them reason about the tradeoffs between traditional real-time visibility of an agent-full system versus the specialized bespoke but not real-time visibility that your company or my old product were able to offer people? How do you help people through that gap? Because to me, that was so sticky.

It is sticky. A couple of things is, first of all, is I say Sandfly is where agents can't be. Right? So we might talk to a customer, they might have 10 or 20% coverage with an agent, but 80, 90%, they're doing maybe log monitoring.

Oh, that's not great.

You know what I mean? Yeah, you know what I mean? So stuff like that. So I'm like, you know, what are you going to do? I mean, is doing nothing a better idea? Well, no, it's not. No, right? Second of all, we find— and you've probably seen this too— I've been in the IDS industry a very long time. Intrusion detection systems. I shouldn't throw an acronym for saying what it is. And probably over 30 years. Customers love watching too much stuff. So you can have a real-time system and they're like, we need more telemetry. I'm like, well, how much telemetry do you want?

Right?

And most customers aren't prepared to answer that question. You know, so if they have a less mature security organization, they might be trying to collect this telemetry and they'll use it as a mop-up squad after incident. But I'm thinking, wouldn't it be better if you had alerts that maybe weren't real-time, but they're very certain that something happened? And so I think the advantage we give on the agentless side— is we can provide more context about the attack. And I always use an example of a log file being tampered with. So you get like a traditional log file on Linux, like /wtemp or /utemp, which logs who logged in and logged out. So an agent might say, hey, someone opened that file and wrote something to it and closed it, right? Okay.

Okay, that's fine.

That happens a lot, right? So then, but we'll look through that log file. We'll say someone opened the log and entry number 16 has been overwritten with nulls to erase that they were there, right? So this is, I think that's the context that we give by not being real-time is we're— we see the attack after it's fully completed. So you tend to get a more definite answer of what's going on. So you have, I feel, more actionable information versus the real-time telemetry. Now, real-time telemetry is useful. I'm not slamming it. I think it needs to be used diligently, and a lot of people use it too much. I think it needs to be used appropriately. But also, at the same time, we could provide telemetry as well. That's, again, just a bit different take. And that also doesn't include things like, again, our abilities to track the SSH keys and things like that, that off-of-real-time systems might not necessarily be looking at for you.

That makes a ton of sense, and it leads very naturally into the next thing I want to ask, which is, you know, imagine you're protecting a cloud environment, you've got a couple hundred hosts doing different stuff, you pick up evidence on a bunch of them that something is going on. How are you helping people prioritize and reason about and sort through what their next step should be?

Usually it's— so the larger the company, the more they're going to have a defined process typically.

Yeah, sure, sure.

Right. So most companies are very nervous about going onto Linux guns blazing and start doing stuff automated, right? It's, you know, so what we find generally is we will give them in-depth information about what's happening. We've now also added the ability to use AI analysis against our results as well, which provides very good information to customers on what they could do next. We'll make recommendation steps. And at that point, typically they want to call their incident response team and make a determination on how they want to go. If it's Linux, typically they're going to want to take a look at that system. They might not want to interfere at all because if they're dealing with somebody who's a nation-state or serious, you know, ransomware actor, they might want to watch them for a bit and see what they're trying to do.

Yeah.

So it kind of depends, and I'm sure you've run into this as well. I feel a little awkward talking about it with, you know, security experts from Google, right? So you're like an elite-level security team, but in general, you probably understand that usually they want to be a little bit more cautious. And the reason is for that, I always talk about Bob in accounting. So Bob in accounting, let's say he's got a Windows box and the antivirus does something crazy, like it quarantines a file, has a false positive or something. Yeah, Bob's mad because this machine got affected, but if you go and you shoot the Postgres database cluster in the head, Automatically, like people get fired, right? So it's a completely different level of risk. So we find most teams, we deal with a lot of critical infrastructure. Most teams want to go take a look, but we give them the information to know what to look for where they might not have anything prior. Right.

You must deal with a lot of teams that are not experts in Linux. And so how do you even like help people get their heads around this stuff?

Yeah. So we try to provide really simple explanations when we detect a problem along with MITRE ATT&CKs. Again, the, The new feature we added, the LLM integration as well, is also helping a lot. By the way, the Gemini LLM is excellent for handling our forensic data.

Oh, no way.

That's one of the tie-ins we have. Yeah, we tie into the Google LLM. It's a really great product. And it's actually my favorite one of the ones we use. That's what I typically default to the most. It gives very good information.

You know, listeners, I didn't, I didn't tell Craig to say that. That's not in the show notes. That's all in the book.

When I go to the next podcast after this for a competitor, I'm going to say the same exact thing about their product too. So no, but I'm dead serious. But the thing is with the LLMs is they could give you very confident wrong answers if you give them bad data.

Yes.

Right? So, but we get very, very good data because we source it all ourselves. Like we are the source. We're not looking at log files and just tossing them over the fence. Like we are looking specifically in the Linux system. If you pass that information into an AI system and you ask it the correct questions, It could be very, very good. In the case of Gemini, for instance, it can help a novice up to an expert-level person in terms of the types of details they can provide. And the more you understand Linux and you know what questions to ask, the better those results are. So these are some of the things we've been helping customers with. But the main thing we give customers is that visibility that they don't have today and really, really good forensic data. So they have everything they need right there to see what's going on.

So let me, let me pick at this AI thing and, and ask you, You know, as the world starts building AI applications, how are you thinking about helping people secure those? What's the impact of that migration on people's Linux security posture, and how are we gonna possibly keep all that safe?

So I'm not an expert in AI, like LLM security or anything like that, so I'm gonna defer answering specifics, but we do think that the AI systems are clearly being used now to help automate and make a lot of attacks better. Certainly now, start with phishing attacks. The Nigerian emails no longer contain misspellings, right? They're very good. I think the same thing will happen on the Linux side as well as people begin to understand these systems a little bit better. Like I said, we get good results passing on our forensic data asking about a problem that's happening. So let's say I were to pass that data in for nefarious purposes, you know, said, I saw, you know, these are the systems we found, these are all the keys, what should we be doing to target next, right? I think that the systems could be easily flipped to do that other side of the coin as well. So that this is, I think, if I were looking at it in the future, I think we're going to see AIs helping red teams out and bad guys as well, just being able to target their attacks and become more efficient in how they do it.

That makes a ton of sense. And then moving beyond the explanation side or the what questions do I ask next side, how are you thinking about using AI to help the blue team?

I think what you're going to see coming from our product is not only are we going to be recommending responses, but eventually get to the point where we'll actually be doing them if a customer wants us to. I think the sticking point we find right now, again, is I think a lot of Linux security teams in particular are nervous about having automation happen. So I think we'll be using it in that regard. The other problem too with pure automation is that just because you— I don't think you could clean up a Linux host because you got to find out how they got on. And until you figure that out, there's no point to trying to kick them off the box.

Right.

You don't know what they did. They could get right back on in 30 seconds and stuff like that. So I think part of the AI, the way we use it, is to be able to pass in multi-events and then being able to derive maybe thesis on how things might have gone off the rails that might help security teams respond faster. So that, I think that's one other area where we might see it. 2, we've also had good luck using the systems to kind of derive a bit about what they think the malware might be doing if you have incomplete information, which is frequently the case.

Hmm.

You might not have people who are reverse engineers. So for instance, but we found we could pass it a binary that's broken down or a BPF door as an example, right? We pass in the BPF bytecode and we said, decode what this is doing. All right. So Gemini system did it in about 30 seconds. It took one of my guys a half day to do it by hand.

No way.

Right. So yeah, it is. So it's one of these things where, you know, you just want to, you know, use them and leverage where you can, but, you know, people think it's going to replace the human. That's not true. Like we know what questions to ask. Right. So if you know what questions to ask, these systems can really accelerate your ability to get to a problem.

Yeah.

Yeah.

So I think it'd be an interesting balance how it works.

What's so funny about that example, and again, listeners, I swear this wasn't staged, is I think this episode is going to end up airing around the same week that I do a demo, which we recorded last week, of reverse engineering a piece of malware using Gemini as well.

Yeah.

So that's, that's really wonderful.

Yeah. I recommend everyone give it a go. You'll be surprised at how well it works. Maybe you won't be surprised. You won't be surprised. You work at Google.

But yeah, I was going to say, you know, yeah, I'm singularly not surprised because the thing that these systems are so good at is, you know, translating encoded meaning into understood intent.

Yes.

And when you can understand the intent of a file or of a binary, which is a file, it's much easier to, to really make a malware decision about it. Like, there are so many techniques the adversaries have come up with over the years against signatures and hashes and sandbox detonation. But once you're stripping it all away and saying, I don't care how you behave when I watch you, I am going to understand your intent, it's a totally different ballgame. And so I hate to do this, but I have to ask our traditional closing questions. One, do you have recommended reading for people? And I will actually pitch your LD preload blog, which I really enjoyed reading.

Oh, thank you.

I thought that was great. So listeners, you should go Check out Craig's blog on that. But do you have other recommended reading? And then 2, do you have a tip to help people level up their Linux endpoint security game?

Yeah. So thank you for the blog. By the way, the AI analysis we posted in that blog was from Gemini.

Nice.

Right. So there are a couple snippets in there that you see there. But I know I keep going on about the product, but I really do like it. Reading. Boy, I'm hardcore in InfoSec. I know I'm going to get grief. I'm just going to say it. If you're in InfoSec and you don't know how to code, you are seriously hamstrung. So I would say, I would recommend, if you're going into Linux security in particular, I would recommend reading a book on C and a book on Python and a book on shell scripting.

Okay.

So, right, if you're talking— we're just talking Linux security today, I'll keep it very focused. I, I'm going to give generic advice there. Those I think could be really good to understand. And again, a book on Linux fundamentals in terms of system Fundamentals would be good in terms of system networking as well.

Yeah.

So those are kind of my books. I'm more of a, if you're going into Linux, you're eventually going to run into C, Python, and shell scripting, and you're definitely going to run into malware communicating over the network. So I think you should understand TCP/IP pretty well.

I would agree with all that advice. Craig, thank you so much for joining us today. I can't wait to air this episode and listeners, let us know what you think.

Great. I appreciate your time today. Thank you.

And now we are out of time. Thank you very much for listening and, of course, for subscribing. Please subscribe so you can get new episodes piping hot. And if you love our content, please drop us a review on your platform of choice.

You can find this podcast on YouTube, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us on our website, cloud.withgoogle.com/cloudsecurity/podcasts. You can argue with us on the Google Cloud Security Community site, googlecloudcommunity.com.

You can follow us on X, x.com/cloudsecpodcast. Tweet at us, email us, argue with us, and if we like or hate what we hear, we can invite you to the next episode.

See you on the next episode of the Cloud Security Podcast by Google.