EP30 Malware Hunting with VirusTotal

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here are myself, Tim Peacock, the product manager for threat detection here in Google Cloud, and Anton Chuvakin, a reformed, tamed, and much better man than he was, former analyst and member of the cloud security team here at Google. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you like our content and want it delivered piping hot to your device every Monday, please do hit that subscribe button. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, what do we got today?

So this is a very fun one. It's fun for 2 reasons. First, It's with our friends at VirusTotal, which are, I guess, considered our fellow Googlers, right? In theory.

They are.

And the other one is, of course, that it's about threat hunting, which is a very fun and controversial topic, which has been kind of hijacked by, oh God, marketers.

Uh-oh. But we know that we like marketers on this show.

Well, there is that, yes. But we don't like them hijacking our specific terms and running with them, right?

I suppose not. It's important to maintain words meaning things. So what does threat hunting mean anyway?

Well, you will get our guest to define it for you on the episode, but to me, one thing I've encountered over the years, including when I wrote my first Gartner paper on threat hunting around 2017, is that people immediately assumed that hunting is this new cool thing that everybody must do. But straight in that paper, I kind of said, hey, hunting just is not for everybody. If you're facing certain types of attackers, if your defensive posture has certain features, and if certain top-notch people are out to get you, you probably need to hunt. But for many other organizations, they have many, many, many, many other priorities before they hunt.

Yes.

Like good detection.

Yeah, I think hunting is one of those things I imagine most organizations really ought not to invest in. There's probably a gazillion things to spend your money on before you spend money on that.

Correct. Gazillion at least, yes. Because this is where they end up doing the cool thing, trying to do the hunting before they did detection well. And frankly, I've seen some real hilarity ensue and some real money being wasted and people getting upset over it because they just did not have their priorities straight.

I think that's an interesting challenge for people is, you know, as an industry, we're really, credit to marketers, good at getting people excited about the new shiny approach to things. But really, a lot of our industry, I think, ought to just be eating their veggies.

Yes. And with that in mind, let's proceed to our guest. And today we have with us Vicente Diaz, a threat intel strategist at VirusTotal. Welcome, Vicente.

Thank you for having me. Sure.

And the topic today is really the one that fascinated me over the years, and that's threat hunting. Frankly, I would refuse to start a conversation on threat hunting without first listening for a definition. I know there are many definitions. So Vicente, what's your take on how to define threat hunting, say, versus detection or maybe versus other activities?

You know, to some point, I think this is both a science and an art to some degree, in the sense that you need some kind of inspiration to do it right. But in essence, this will be like how to define the process of finding something cool, interesting, or maybe scary, basically based on your previous knowledge of attacks. And I think that following what you said regarding detection is pretty different. Let's say detection, you are sitting and waiting for something to happen, and when you recognize something, you will flag that something happened, right?

Yeah.

But hunting is like a process, it's a procedure where you have some signals, you have some Previous knowledge is much more proactive. You are trying to find something that you had no idea it was there in the first time. So detection is more recognition. Hunting is more finding something unexpected.

That makes sense. And I think to me, this is actually a very useful start because I do still see people who sadly confuse them, or they think something like, oh, it's a historical detection. We're going to detect an attacker last July. To me, you have to uncover them. So to me, that's hunting, right? Like if you're looking at the past.

Yeah, actually, many times hunting starts not even based on some clues, and you have no idea what you are looking for. From there, you don't know what you will find. Sometimes you find nothing. Sometimes you uncover, like, one of the biggest APT operations. So it's pretty different.

So given that there's ambiguity in definitions, would you say that companies or organizations make some popular mistakes hunting? Do they confuse hunting with other practices? Like, what are your favorite mistakes or pitfalls that you've encountered with threat hunting?

Threat hunting is not easy. As we say, there is not a clear definition. First thing that happens when you start hunting is that you never know what you will find, so you never know when your investigation is finished, and that happens all the time. I think that the major mistakes are based with our own biases and also sometimes doing the hunting for the wrong reasons. So let me explain myself. First of all, we have all of our biases, and we quickly jump into them. Like survivor bias or just looking for confirming something that we already suspected and this kind of stuff. And I see many times people is dying to find a small clue to confirm what they were hypothesizing in the first place, which is not the right thing to do. We also see that in some occasions, hunting is more, let's say, for the hype or more for the PR than anything else. So they are very quick. to jump into creating a new APT group or defining a new attack or isolating this from any other activity. Well, maybe they don't legitimately know the connections, right? But many cases, they just want to jump with the big story. So in this case, hunting is not really trying to get more knowledge in a global sense. It's simply trying to find something cool to expose. But mistakes, we all make a lot, and sometimes for different reasons.

Right.

I like the confirmation bias mistake, frankly, because I've seen people who start hunting with somewhat of a clear idea of what they're looking for. But if they have a very clear idea of what they would find, then they're almost certain to find it, then why don't you write a detection rule? Maybe that's something that happens a lot in your environment. So it's a very kind of narrow-minded, I want to look for X, and I'm going to find X type of hunting.

Yeah, you think about that. If you think why all these APT groups are so isolated, for instance, when we make all this attribution, Why is that? In reality, maybe they are not. In reality, we are maybe artificially constructing this because different groups, they evolve in time and they are many different than they were like 5, 10 years ago. And we still keep hunting them the same way and we keep differentiating these artifacts even when these artifacts broadly used by many other different actors, et cetera, et cetera. So this is what I mean. If we are just keeping in mind these old paradigms and keeping doing the same over and over and keeping the same rules and doing things the same way, way, it leads to the same conclusion. So, this is a good example of this confirmation bias.

Interesting. That gets back to another theme we've had throughout the show of the way we've been doing things is maybe not the most successful way given everything we're seeing in the industry. So, speaking of ways of doing things and how things change, when I was growing up, hunting was something you did from a deer stand in the woods. I assume this doesn't involve deer stands in the woods. Could you describe maybe what the modern threat hunting process looks like these days for people that maybe learned about this 10 years ago and things have moved on?

Yeah, I think that, well, most people when they start doing threat hunting, first of all, you have different motivations and you have different reasons, and the process obviously is different, right? But most of the time what we have is something to start our investigation with, and many times this is something provided by some external provider, like threat intelligence provider, or maybe published by some security company. So we start with a bunch of indicators, a bunch of YARA rules. And this is where some people, they simply stop here and they do nothing. But usually we want to learn about the infrastructure. Obviously, it depends on the tools that we have. It depends on the database and the telemetry. But let's assume something like VirusTotal, right? So the first thing you will do is check the infrastructure. What else was related to this infrastructure in the past? You will see all the artifacts that are detonated and what is resulting in the sandboxes. And from here, well, you will see an idea of what's the whole picture, right? These days, we can use things like similarity, which is extremely useful, because this is the way to see all these relationships that initially are not so obvious. So with this, you can create a timeline as well, and see how things evolve from the original and looking back into the past. An important thing to do here is to keep monitoring any future activity just to see what you can discover. So you can start with the YARA rules that they are provided to you, and you can create your own. From here, basically, it's checking what else you can find depending how big is your collection and how inspired you are, you know, to find the right trigger to find more artifacts. And it's a bit tricky. Sometimes it's super quick. Sometimes you don't get the right idea.

So speaking of artifacts and interesting things, could you share a couple of examples to make this real for people?

Let's say that these days things are maybe not that exciting initially just because everybody is using the same stuff like lateral movement. You see Cobalt Strike, you see Mimikatz, and you see all these kind of artifacts in every single attack, right? But this is because it's working from one side for the other. It's good to avoid attribution. In case you care about that, which many people don't. I mean, in other cases, you simply don't have the telemetry or the means to check into, let's say, networking hardware. There is no way for you to check what's going on there. So it's unfortunate that you cannot use this. But some of my favorite cases, I remember one, it was related to some ransomware attack. And in this case, the attackers, they were in a very tightly secure environment where they couldn't execute their scripts. So probably you heard about that, but what they did is they download a full virtual machine. And the whole idea is that they just wanted to execute one script that was inside of the virtual machine just because it was the only way they had to execute it because the environment was very well secured. So these kind of crazy things that it will never occur to me, they occur to the attackers, right? I would say that these days everybody's very worried about supply chain attacks. We saw a good example last year for a good reason. Unfortunately, it's difficult to extrapolate lessons learned from these attacks in the sense that they are pretty unique in every case. We cannot say, like, doing this or that, we will avoid the next supply chain attack.

In your examples, you mentioned ransomware, right? Which in this case, I'm assuming you're talking about the malware component of ransomware business. I guess in the old days, unlike with deer and team, I assumed hunting implied you're hunting a human attacker. So I'm sort of a little bit lost as far as where do we go and hunt for malware? Like, shouldn't ransomware just announce itself, and then we wouldn't have to look for it? Or are you talking about detecting it early, but then you're detecting, not hunting? So could you clarify a bit more about hunting for malware versus hunting for human attackers?

And I know what you are saying here. In the end, it's human attackers what we have on the other side, right? This could be semi-automated attacks, but if we know how they are acting, we can prevent them from attacking us. And we have seen this in the past from these APT groups. We understand these are operators. Operators make OPSEC mistakes, and they basically use the same tools over and over. They have like their playbook, and they keep repeating. So if we know the playbook, we can somehow anticipate what is the next step. The thing is that I believe This is a personal opinion here, but I think that in terms of TTPs, there is still way to go. It's not so easy to understand and especially to characterize how an attack works. And we are doing big steps. We have ATT&CK, MITRE framework and everything, but still everybody is doing this in a different way and it's difficult to automate and it's difficult to find some steps that happen in every single attack into these huge metrics. The problem is that we are hunting for people. We are hunting for these attackers, and we want to disrupt them anyhow, but it's really difficult. It's really difficult to characterize and to put this information and to make it actionable. This is really tricky. So that's why I think it's not really working at this point. So these days we are hunting for malware, and unfortunately, in some cases, this is really tricky, especially as we said before, for lateral movement. Everybody's using the same stuff over and over. So how can you use this to characterize different attacks and different attackers, right? Yeah, so we should be hunting for people, I guess, but these days I think we are still not ready for that.

So when it comes to hunting for attackers, what does it look like when they know that they're being hunted? Are there tricks that the bad guys use to avoid being hunted? Deer just wander around the forest. They don't really worry about the hunters.

You said it. Some of them, they don't care. They are being detected, and so what? So in some cases they do. In these cases, we see them doing all kinds of strange tricks in order to avoid attribution.

Like what?

Let me give you a couple of examples, like out of my mind. If you are familiar with VirusTotal, let's say you find some malware in a document, and this is a macro which is executing and is doing some evil stuff. So in the old days, you go there, you check the content, And there is a limit in the number of lines of text code that you will see in VirusTotal, just because of efficiency. The attackers, they realized about this, so instead of starting their macro right away in the code, they put like 250 lines of trash.

No!

Yes, for avoiding the analyst to see the lines of original macro, you know, the malicious stuff there. And this is one example. I was talking to some researchers, and they said, hey, I'm having trouble because I don't find some stuff in RetroHunt, and I know it's there, and I cannot find it with my YARA rules, but the YARA rules, they work locally. So we were checking, and again, we had some limitations in the size of the files that are being scanned in RetroHunt, and they are abusing that. They are making them slightly bigger, yes, to avoid them to be found. So it's this kind of game. It's like anti-sandboxing evasion and things like that. So basically they are finding how we are detecting the tricks and they are finding the way to evade. This is something that for some attackers, they are aware of that and they are trying to exploit, let's say, our weaknesses of just checking the tool and believing what we see with our eyes. And there is a perfect example about this. Maybe you remember about Olympic Destroyer.

Yeah.

It happened in the Olympic Games in South Korea. It was in 2018, I believe. So if you didn't do it at the time, I highly recommend to go now and check every publication that was at the time from different security vendors. So basically every big security vendor was doing their own analysis, and all of them, they have different attribution. You may ask yourself why, and the thing is that the attackers were playing with security researchers, and they were aware of the tools that we are using just to provide different attribution for their malware. Actually, what they did is they forged the PE header to make it look like it was from Lazarus, which would make perfect sense because Lazarus is attributed to North Korea. So in this case, you can imagine that Lazarus would be interested in doing some badness in the Olympic Games in South Korea, right? You run this malware in some attribution engine, it will say, hey, this is Lazarus, just because they were forging the headers. But in reality, it was a different actor behind. So they were perfectly aware of the tools that we are using, and they were abusing that. At the end of the day, who is interested into this attribution, and who is ready to spend the time or the money just to exactly know who is behind? Of course, like, let's say, 3-letter agencies, they are interested because it's their job. But for, let's say, other companies with a more moderate budget and are interested just in being secure, probably attribution, they don't care about that.

It sounds like to actually get to the bottom of it, you do need a fair amount of research, just like you pointed out. So would you say that the threat research and deep threat research at that is kind of necessary to do your own threat hunting, or could you outsource it? Could you skip that part? Could you just rely on VirusTotal? I don't know. So to me, it sounds like for specific attribution, to an extent we can, we do need much deeper threat research. But could you just outline the relationship between threat research and hunting in the modern era?

Yeah, I think that only for a small number of victims, they are really interested and they are ready to pay what it costs to make the research in terms of human efforts dedicated to something like that. I think for most of the cases when it's crime war and things like that, Most of the people, they don't care. They just want to be protected somehow. But the thing is that these days there is this attribution hell. I feel we are in the worst moment in terms of attribution. And what I have seen is that as we were developing tools in the security industry, yes, we were making advances like using similarity tools and things like that. Of course, we could make the connections, but at some point the attackers, they said, well, you know what? Some of them, they don't care. Some of them, they make their malware publicly available or they use commodity malware, so they don't need anything else in order to get into their victims. The really sophisticated ones, I believe we don't really see much from them in the sense that they probably go through the ability to run in this ring-2. If they have the ability to run in this network equipment, Then it's really hard for us because we don't have eyes and we don't have telemetry in all of this, right? And some of them, they play with these false flags, like this Olympic Destroyer that we discussed. So in most of the cases, unless attackers, they make any OPSEC errors, which could happen obviously, or unless defenders, they get some extra, let's say, eyes into this, like, I don't know, someone can share like the NetFlow from the servers or they know something else. In many cases, I see that there is a lot of intuition, like, well, this attack, yeah, I think it belongs to this group. And you're asking, but why? I don't really see the connections. No, because somebody told me. So there is also this human factor here, right? And it depends on, well, everybody has partial visibility. Maybe someone has better visibility, which means maybe they have access to the network of the attacker or something like that. So this small circle with rumors, like sharing partial information, But let's say this is the last stage of this attribution where like true experts that know all these actors by heart and they are sharing this information in private circles. But for, let's say, any security analyst just getting this from scratch and trying to make this connection will be extremely hard. As I said, I think that for most of defenders, this information is not that valuable.

I think I agree with you on the value of that kind of information. It makes us feel good to know it, but I don't think it actually helps us in our mission as defenders. I want to roll back to an earlier fork in the conversation here and talk about YARA rules. Could you walk us through like the evolutionary journey of a YARA rule from a threat hunting artifact to a detection artifact? Is that even a lifecycle? How did YARA rules evolve in people's process from hunting to detection?

This is very important because some people, they don't really think about that much, and they think that the same YARA rule is valid for everything, which is not true, obviously. Like, you can use some much broader YARA rule just to start your investigation, and then you keep narrowing it down if you want to go for detection, obviously. I will say that most of the time you want YARA rules for hunting, but I wouldn't say detection, purely detection. I think that you need many more factors to say this is malicious for sure. But for hunting, they are excellent. Probably they are the best tool we have as researchers. Usually, I like to start with a bunch of samples, and I like to start with automatically generated YARA rules, which many people, they hate. I understand. because usually they are very broad and they are terrible.

Hmm.

But you start taking a look and you can see how prevalent are some of the things that you use for these rules. So from here, this will help you big time to find common code between the samples, to find like common strings. And from here you can start polishing. I like to pay attention to things like the metadata. I'm a big fan of metadata. Things like compilation times to make correlation. Obviously strings. If you find any typos in there, like some commands that they are receiving or things like that. So you can take a look to things like some hashes, imhash, and things like that. If there is any section name that is very strange, but these days this is very, very rare, I think. So with all this information, you are good enough to start hunting for more stuff. If the malware is really complicated, then yes, you need to go full reversing and find the parts of the malware that you want to use for detection. But obviously, this is very, very time-consuming. So it really depends how deep you want to go. For instance, in VirusTotal, there is one advantage that we have a VT module for YARA, and you can also use the metadata from VirusTotal just to do these YARA rules a bit more efficiently, like submission country from the malware and when it was first seen and things like that. You can incorporate into the YARA rule to make it a bit easier to deal with.

Okay. No, that makes sense. And to me, this is kind of the example where a good rule does not mean you can run it and then detect something and wake people up at 3:00 AM. We are not in IDS rules or even SIEM rules. We are in hunting and research rules, which to me means good here is a very different good compared to good for the reliable CRISP rules with low false positives that are ready-made detections. This to me is kind of a critical conclusion from some of the discussion. So what's your take on the future of threat hunting? Will we need to hunt in 2030, or will AI do it?

AI will help, of course. I think that still there is the huge effort to translate everything we know as humans into something that the machine can interpret. As I said before, like some things happen still at this human level, but I think that little by little we are getting there. Still, this is this part of art and still, you know, the number of complexity is growing exponentially in terms of more relationships, in terms of more things we can use to determine like these relationships and how they are. Working together different pieces of all this giant puzzle. Every day we get more and more samples, more and more telemetry. So at some point, of course, artificial intelligence will be needed. But I tell you, I think that still threat hunters will be valuable at the time and we'll still be there. So don't give up. Even if we are just there to solve the CAPTCHAs for our artificial intelligence, I think there will be a role for us.

I'll tell you what, you'll get no disagreement from your hosts And I suspect most of our listeners, if you say AI is not going to replace all of us in the near future. So with that, I want to move towards wrapping up and ask you our traditional closing questions. Do you have recommended reading for our listeners so we don't leave them empty-handed? And also, do you have a practical tip that they could follow to improve their threat hunting experience?

Yeah, I have a reading, but it's not related to any of this, but it was about Kasparov. He was discussing his experience with Deep Blue when he lost the world championship for the first time against an artificial intelligence, in this case playing chess. So I don't really remember, I think it's called Deep Thinking from Kasparov. So this is my recommendation for reading. I enjoyed this book. In terms of pro tips for hunting, my pro tip is to learn more about the mental biases and to read a lot and try to apply these mental biases in a critical reading every time we see a publication from a vendor or from whoever, and try to understand what kind of things the analyst was thinking. And the best way to do that, I found, is to read the YARA rules, because in a YARA rule, it's not like an IoC— here you go, this IP— but you see what the analyst was thinking when he was creating this rule. So I think it's a very interesting way to learn from the others.

Huh.

That is one of the more deep-thinking sorts of tips we've gotten so far on the show. Vicente, thank you so much for joining us today.

Thank you both. I really enjoyed it.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can also follow us on Twitter, twitter.com/cloudsecpodcast. And of course, your hosts are also on Twitter, @anton_chuvakin and @timpeacock. Tweet at us, email us, argue with us. And if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode. Security.

Security.