EP108 How to Hunt the Cloud: Lessons and Experiences from Years of Threat Hunting

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a Reformed Analyst and Senior Staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website. Cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Finally, you can also now find us at cloud.google.com/podcasts. We've been added to the official Google Cloud podcast page. And listeners, well, maybe you would believe it, But I have to say, you wouldn't believe the process of getting us there. Anton, we're talking about threat hunting today. This is a fun one.

This is a fun one. And I feel like maybe we are a couple of years late addressing the hunting topic.

No, we're in the far side of the hype cycle. We've passed the trough, right?

We— no, I feel like hunting, if you made me think of hype cycles, I think hunting is still falling into the trough. I don't think it passed.

Oh, no. Oh, so sad.

I feel like we are falling off the peak because more people realize that hunting is not just searching for indicators, and then they get upset, and then they go complain to their vendors. So I feel like we are still falling into the ditch on the hype cycle.

Got it. I certainly bring a bit of skepticism to this episode when it comes to what is hunting and what's its value for most organizations. And I wish we'd had more time to get into that, actually. I think that would've been fun.

With that in mind, let's get to our guest.

And with that, listeners, I'm delighted to introduce today's guest. Today we are joined by John Stoner, a principal security strategist here at Google Cloud. Anton, you have the privilege of asking the first question today.

Yes, and I am jumping in my chair because I really want to ask pretty much everybody that question.

Like people you meet on the street, everyone, or just everyone on the podcast?

Everyone who actually ever opined on the topic of threat hunting.

Okay, good.

Because whenever I hear somebody say threat hunting, threat hunting, I want to say, Stop. Can you please define it for me? Because the term has been corrupted by the industry a lot. And I don't want to say by the vendors, by the way. I just use this polite term by the industry, but of course, hey, it's by the vendors. And I've seen sometimes people even call searching for typing an IP address in search and pressing Enter, and they think, I just hunted for threats. And you want to, of course, slap those people, but—

Yeah.

Okay, John, what's your definition of threat hunting?

That's a great question to start with. It is a very abused term. I think what I like to fall back on, and I've used this for the past couple of years when I've talked about it elsewhere, is to leverage a document that SANS wrote called the Who, What, Where, When, Why, How of Threat Hunting or something like that. And the definition that they've got talks about this being a focused search for adversaries who have capabilities, intentions, and opportunity to do harm to the organization. And that's slightly paraphrased, but you get the idea. The word searchy in there is also a little bit of a loaded term because obviously hunt, investigation, you could use those terms interchangeably as well. But really the idea is you're focusing on adversaries, you're assuming breach, right? That therefore prevention and detection tools have been bypassed or you have low confidence alerts. And so you really need to kind of go digging in there. Now, that having also been said, The threats are not one-time things though, right? This is going to be something that you have to do as part of a programmatic approach. So there's a lot of different things I kind of threw together there, but hopefully that kind of helps a little bit to put a little bit better scope on what threat hunting is. And it's not just, yeah, literally running a search and saying, yep, we're good.

I love the bit in there about assuming breach and assuming that detection has failed. And maybe I'm feeling defensive because I'm a dirty, dirty detection vendor. But one of the things that's always struck me in public breach reporting on companies that have famously been breached is that the detection systems often did catch the bad guys, but it was low confidence or otherwise ignored. So what's the intersection between threat hunting and just reviewing the alerts you're already getting out of your detection systems?

Which is, of course, the whole point of the term, right? It's alert gathering versus hunting, at least in the version I've heard from somebody who's older than me.

Yeah.

So, I mean, I think from that perspective, the problem and the challenge with the detections is, is that you have so much there and you have a lot of goodness there, but it's prioritizing, right? And so if I have 10 things that kind of look alike or I have something that is recurring, but you know what, I've got to set these aside because there's something else that is much more high priority for whatever reason. I can't spend all my time everywhere, so I have to go ahead and say, maybe these things I'm going to come back to later. At least with the threat hunting concept and being able to put a programmatic approach around that, it allows me to maybe come back to that pile of detections that I had that maybe I wasn't as high confidence about, but gives me a chance to focus in on them a little bit more.

Mm-hmm.

And maybe approach them a little bit differently. So, one way to do hunting is to take a victim-centric approach. When I say victim-centric, we're talking about focusing on incidents and events around impacted systems or maybe doing something like a crown jewel analysis, if you will. What are my most crucial systems out there? Who are my most crucial users? And look at maybe that pile of those detections that you were talking about from that perspective and saying, hey, here are a bunch of things that are associated with systems that are really important to me. How can I derive value out of those things? Maybe they're, like I said, they're little atomic things that are like, oh, this is a little bad, this is a little bad. All of a sudden becomes a death by a thousand cuts. If I can put those kind of together and start looking at them and saying, does this tell me a bigger picture around this specific victim server or victim system that might help me be able to do this and not have those individual detections kind of fall by the wayside?

So I think I like this as a methodology, and I like, of course, the whole assuming the breach kind of victim-centric or target-centric thinking. But if I set up a new team or if I set up a new person who wants to hunt, do you have any favorite start their hunts to jumpstart the effort kind of more practically? Of course, people like to go look for persistence, and they're kind of classic choices, but what about John's choices?

I think it's highly dependent upon the dataset that you have to start with, right? And that's a trick. You know, that's kind of a bigger point that we'll probably come back to a little bit later. Getting started, understanding the space that you hunt in is its own first hurdle that you have to get over. You know, we see this even when we train folks on how to use a SIEM, right? The first thing you got to do is you got to understand the data that you have inside of it before you can really write rules or really search, right? You have to kind of understand that terrain. But that having been said, if I have network-centric data or I have somebody who's comfortable with network-centric data, being able to start in looking at the direction of the data flows, understanding the volume of data, understanding where data is starting and where it's going to, maybe looking at something like an autonomous system number. Hey, where, what, what net blocks is this going to in the world? And what kind of protocols am I using? Those are some good, nice, meaty kinds of things that somebody who's maybe coming out of a networking background could probably grab onto and start kind of working with. 2 really good examples that I like to look at from that perspective would be, hey, I've got a bunch of servers that are out there. Are those servers doing a lot of client web browsing traffic? Should I be surfing the web on a server?

This is such a '90s idea, which unfortunately is super relevant. It's like almost Sadly relevant in the 2020s.

It is. And, you know, I mean, it's one of those things that kind of falls into the compliance violation bucket, right? But at the same time, if you've got some level of persistence and you've got somebody going ahead and exfiltrating, that tends to look a little bit more like, you know, a web browser than it does a server, you know, serving up web traffic, right? So things like that, you know, again, kind of a fundamental one, but do you allow FTP within your organization? If you don't, do you see it, right? But if you do, Where do you see it? Who's using it? Who are they connecting to? And what does that traffic look like, right? I mean, and again, those are very low-hanging fruit, and those are things that, you know, you might get to the point where you're building detections around, but it could be a little noisy to start with. So you don't necessarily want those false positives in your SIEM. So maybe starting off with those kinds of things, hunting as you refine, as you understand what your datasets look like, and allow you to kind of build that over time to have a little more teeth in it.

Okay, so let's dig in on that one because you talked about building detection around this, and that's exactly where I wanted to go next. I wanted to talk about how we evolve from these manual hunts or like brain-driven things into systematized things. What's that journey look like? If I want to take this, say FTP in my org, for example, how do I turn that into a high-quality detection?

I kind of fall back on the wheel and the rinse and repeat method for things. The best one that I can point to, and it's not in hunting per se, But it's the NIST 861, right? It's the incident response lifecycle. If anybody's ever taken the SANS incident handling course, you've seen this. If you've ever read those NIST publications, you've seen it. It's pretty much everywhere. And so it starts about talking about preparation, moves into detection and analysis, moves to containment, eradication, recovery, rinse and repeat in that cycle between those things. And then you go to post-incident activity. And then post-incident activity, you learn from that and it flows back into your preparation. Okay. That same premise, right? And those boxes aren't exactly the same, but that same premise applies to threat hunting, right? You're gonna go ahead and prepare your threat hunt. You're gonna conduct that threat hunt, which is kind of that detection and analysis phase of that. You're gonna find certain interesting things and you're gonna kind of peel that onion with that and continue hunting against that as you continue to learn more in that cycle. And then as you come out of that, you need to learn something. Okay, now learning something could be, well, we hunted for something, we didn't find it. But even if you didn't find something based on your hunt, you've learned something. You've learned that— you've either learned that I built a hypothesis and I was not able to prove my hypothesis. Okay, I may not have the right datasets available to prove that hypothesis. So therefore, maybe I need to add additional data sources to my collection, or maybe I do have the right data sources, but I'm not auditing or logging the correct things to be able to answer the questions I'm trying to answer. So using that incident response lifecycle, I like that as kind of a foundational concept. And obviously you're changing the boxes from a threat hunting perspective, but the concept still applies. And then whatever you hunt for, when you complete that post-incident activity or that lessons learned piece of it, the question then becomes, can I build a detection upon it? Is that detection going to be high fidelity? If it is not high fidelity, that's okay. You can have what I would refer to as hunting queries, right? Hunting rules. You know, that would be something that maybe I wouldn't go ahead and generate an alert for in my incident queue, but I'd have those things available to me to be able to start hunting and kind of going, oh, okay, yeah, I got a couple nuggets here. Maybe that allows me to focus on a couple of specific assets, on a couple specific identities to hunt further. But you definitely want to learn things from your hunt. And you don't want to hunt for the same things over and over again, because if all you're doing is hunting the same things over and over again, you're not making any progress. Right.

And that's a common complaint I've heard in discussions on this, because if your weekly or monthly hunt look really the same, then whoa, wait, something's really wrong. Because if you're not finding stuff, maybe you're doing the right thing. But if you are finding stuff that maybe they should have been detections, and then in this case, they will be picked by the machines, not by humans running the queries. But it's easier said than done. I think that falls very nicely into the easier said than done bucket. And I do expect teams to do repeated hunts.

Yes.

Just not all repeated hunts.

Exactly.

See, when I talk about this with users, I talk about it as the hamster wheel versus the flywheel. You really want to get out of doing the same thing over and over again if you can and throw off— and this isn't really how a flywheel works, but forgive it— good improvements to the tooling, whether it's a preventive control or a process detection control.

For some reason, I'm still seeing you accelerate the hamster to a certain speed and then have it fly out and just depart from our industry, maybe leave Twitter altogether.

Oh, sorry, that's a different meme.

Centrifugal hamster launcher. This is terrible.

So, and this hamster perhaps would land somewhere else, and it would land in the cloud. So our next question, of course, is a tricky question. Which I haven't seen a lot of insight on this, and I'm nervous about it myself. So I haven't seen that many practical advice of hunting in cloud environments. Is it because people don't do it? Because people— detections built by team are so good that you don't need to hunt? Hahaha. So any tips on hunting in the cloud? Do you expect differences in these environments?

Yeah, hunting in the cloud. I've done a little bit of poking around here and there. And obviously, there are 3 major cloud providers out there, which obviously we're one of those. Yeah, it's a tough one. And the reason it's tough is that at one point in time, I had built a couple of slides and talked about understanding your terrain and talking about, you know, you have your, you know, the image of a dog walking out into a field with a hunter carrying a certain kind of gun as he's literally hunting, whether he's ducks or Well, you know, whatever it is, right? There is a certain level of terrain there that's going to be very different versus somebody who's in the Sahara and looking for some other sorts of animals, right? And that really does apply to this overall. The tricky part is that if I've been hunting in an on-premise kind of environment, I've gotten a feel for network devices, endpoint devices, authentications, and they look in a certain manner that I have some level of comfort in. Much like somebody leaving a job and going into a new premise where the business operations may be a little bit different and they may be using a CrowdStrike versus a SentinelOne, right? Things that change. That analogy goes into the cloud as well. So if I go into an AWS cloud, right, and I have EC2 buckets and S3 storage, and I'll probably mess up the naming conventions of these different pieces, I have to learn what they look like and what their logging looks like and what those common identifiers are. to be able to kind of start putting those pieces together to start understanding what good and bad looks like in their environment. If I move to a GCP environment, I have similar challenges. If I move to an Azure environment, I have additional challenges. And then when you have software as a service, when you have Salesforce or AD or these other kinds of things, I then have additional log formats that I have to worry about as well. So there's a lot of different pieces of that terrain that you have to put together, and that makes it challenging. Another challenge that I've seen is around the visibility of the logging fidelity that's available to you. You are a subscriber, right? You have access via APIs. Those APIs have changed over time. I've seen that with different services where it's like one day you have this set of data and that looks great, and next week that dataset is not exactly the same. Maybe it's a little bit better, maybe it's not, maybe it's a little bit more restricted. And so those are challenges that threat hunters have, as well. And then, the latency piece of this. Yeah.

It's like, can we be in denial about this one? I just feel like hiding under my desk when I hear the latency comment again because we did have logs that arrived 2 hours later, even here at the esteemed security-aware Google Cloud. We had this happen, right?

Mm-hmm. So I'd say latency has gotten better over the past few years. I don't think it is as long in general. That having been said, there are a couple of the major cloud providers in their application services that will say up to 24 hours. Now, for a hunting scenario, that may not be as bad, but for a SecOps scenario or potentially an IR that's fast moving, that could be very, very deadly. Right. So again, it depends.

I was going to say, isn't the point of hunting that it's retrospective almost necessarily?

Yeah. So if you've got that hooked up and you've got that logging set up, The latency probably won't be as painful for the threat hunter, but again, for IR and SecOps.

Yeah.

And I do think that there is a fairly tight join of these different pieces that you do need to keep that in mind from a latency perspective.

By the way, this is useful. I want to interject with a useful reminder. I don't want to have a fight about reactive/proactive, but at the same time, hopefully I want to add a useful comment that when people show up and say, hand-to-hand is proactive, they really mean before they alert. But then, people show up and say, no, no, no, no, no, you're stupid. Hunting is obviously reactive because you detect an attacker who is in your environment. I think they're both right. And can we please not have the debate?

I would also not have that debate.

Just say you agree with me, and we can proceed.

Let's move on. I agree with you.

I would agree with you, and I go back to the who, what, where, when that we started with. And hunting can occur throughout this entire cycle.

And also you described hunting in the context of an incident response lifecycle.

Yes.

So it seems the two are almost necessarily, if not cousins, siblings.

Yes.

So back to the data sources and logs and latency, what kind of logs, very tactical question, what kind of logs in cloud environments ought a threat hunter be looking at?

So I would wanna understand systems that are being spun up. I wanna understand when they're being spun down. Permissions that are being applied to buckets. If I could get fidelity into those systems, so if I'm spinning up more and more containers and I can go ahead and put some sort of endpoint monitoring tooling onto those, you know, at one point in time we've done a little bit of work and messed around with osquery to get better visibility into Linux systems. Now with Sysmon for Linux, that has definitely some appeal as well, particularly for those who are familiar with Sysmon on the Windows side. You know, that's a nice way of leveraging your existing knowledge and applying some of that from an engineering perspective into your new boxes there. I'd want to go ahead and look at flow. So again, looking at applying from a network perspective and looking at which way data's going, firewalls, as well. So there's a number of those pieces. When you start getting into, like, managed SaaS and Azure Active Directory, then there's a whole world of additional auditing controls that you want to look for at the application layer. And then if the database, having started off in my career in databases, being able to turn on a SQL audit or something like that, that I can look at the actual queries that are happening, getting down into that level would be appealing to understand who is accessing my data and in what manner.

We should probably write a guide about how to hunt in GCP. I'm just realizing, and of course I'm going to figure out who to volunteer for this task. Hey, but I think it's going to be a useful resource to write a guide on how to hunt in GCP, right?

You know what I love about this answer? Not only did you just come up with more work for somebody to do at the company, you also listed basically all the logs that my streaming detection service looks at today. So that's great. I love that.

You're welcome.

Yeah, I was going through that list. I'm like, oh yeah, we look at that. We look at that. We look at that.

Yeah, it's really, it's nice.

So maybe a less nice question next. I know Anton's been itching to ask this one.

Yes. And it's a kind of a painfully obvious question, and I think I know the answer, but I still want to ask it, is should every organization, perhaps every security organization, because if you have a team of zero, then the answer is no, you shouldn't hunt. But should every security organization pursue threat hunting? And if no, what's the criteria? Who should? Yeah.

So I'm not of the mind to say, no, you can't hunt. I'd rather approach it a little bit differently. I think all organizations probably should be hunting. But—

oh, you said that.

Every organization should say that.

I think they should, but that is not where we start our job. Okay? I think far too often, and this extends into SecOps and lots of other things, organizations want to jump right to the 400-level class, right? You think about schooling, you go to a 100-level class, you get your 200, you get your general arts and sciences, you build that criteria, and you work your way up. Threat hunting, I feel like, is one of those higher-level competencies, right? So if I'm not logging, if I have no idea what my assets are, I have no idea what my users are, no idea what common flow of information looks like, how can I hunt without those things? So let's start and build those foundational pieces, right? But there's no harm to me if I have those foundational pieces, having people start looking at data and being curious about the data that they're collecting and build hypotheses. And start looking and nudging at specific things and saying, is this something interesting that we can go after?

I think I like your answer. It's a tame but correct— it's a factually correct and fairly tame answer, right?

I don't think most organizations should be hunting. I think most organizations should be outsourcing their security ops to somebody else that does it.

That's, that's not unfair. That's not unfair. But I guess I would extend that out. If you're going to outsource it, I would expect that outsourcer to provide some level of hunt as well.

I agree with that.

So if you're not capable of doing it because you don't have the staffing, that's fine. But I also extend that into if you're not capable to operate your own security operations team, you're going to outsource that as well. This would be an extension of that. Yeah.

I'd rather avoid this. I mean, I think we run the risk of doing this whole lip service to should we outsource, how well outsourcers can hunt. I think I probably know too much about this to avoid this because it's a big question on its own, probably a separate episode. Can outsourced partners hunt? Some can, some can't. How do you tell? If you're clueless about hunting, can you pick a partner who can hunt well if you yourself have no idea how to do it? That's a tricky question, right?

Yeah.

So why don't we switch to a human, in a human direction, sort of, so to say, because you mentioned hunting is kind of a human-centric, analyst-centric process, right? What are the traits a threat hunter should have, a human threat hunter? In the future, probably a robot, but in the far future at that.

Sure. I mean, I think the most basic fundamental piece of this is you need to have somebody who has some intellectual curiosity around their data and willing to explore that data. I also think really, really important that that individual or individuals has to be willing to continually learn. And the reason I say that, and we talked about a couple of these things already, you have new terrains out there. Hey, I'm going to go ahead and move my workloads to GCP. Great. Somebody has to understand what does that GCP workload look like, the logging of that information, what, you know, good and not so good look like, right? They have to understand adversary techniques and understand what, if an adversary does X, what does Y look like, right? So you have to be willing to continue to learn as these things happen, as new data sources are onboarded, as the environment changes, as users are using these different technologies. And then you have to be curious to be able to keep looking at that data, to explore that data, to look for those outliers, to be able to identify those outliers. So it's continually poking, prodding at the data, asking questions of the data, and again, just being willing to, to interoperate with it in that manner.

Well, I like that answer on humans, and I think maybe Anton's hunter robots are further away than we would all like. I want to close now with our traditional closing questions, and I'm sure you knew these were coming. First, do you have one tip to improve your threat hunting? And then 2, do you have recommended reading for our audience following along at home?

Preferably Threat Hunting in the Cloud. Sorry for being a sneaky bastard here, but I just want to like tweak this a little bit.

Other than the blog post that apparently somebody has signed up to write now.

Apparently.

No, not the blog post. A longer white paper. A longer paper. Sorry, not a blog. Blog wouldn't quite do it.

Oh, oh, an even bigger project. That's great.

So I guess the things that I would fall back on in terms of reading is I would definitely start with that SANS Who, What, Where, When, Why, How of Effective Threat Hunting. as a good foundational paper. And yes, that isn't cloud-specific, but I think it pays off in terms of the overall thought process around and debunks, you know, a lot of the hype around what threat hunting is and what it isn't. I will also go out there and throw out there, I did a talk at SANS Threat Hunting IR before COVID that talked about building hypothesis-driven threat hunting. So I'll mention that one as well. So I know sometimes people throw out stuff that they've done in the past, so I'll go ahead and call that one out as well. And then the other thing was a tip to improve your threat hunting. I think I'm going to fall back on know your environment, learn your environment, take that time. That alone goes a long way. And then the other very important tip is when you do your threat hunting, don't do it at 3:00 PM on a Friday afternoon, because if you find something, you're probably going to be working through the weekend.

That is a very practical tip.

Okay.

With that, we're out of time. John, thank you so much for joining us today. This was, uh, A fun and engaging episode, and I will remember not to look for bad guys on a Friday afternoon.

Fair enough. Thank you, guys.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton underscore Chuvakin and underscore Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.