No One Expects the Malware Inquisition

Security. Hi there. Welcome to the Cloud Security Podcast. Thank you for joining us today. Your hosts here are myself, Tim Peacock, a product manager for threat detection here at Google Cloud, and Anton Chuvakin, a reformed and tamed analyst and member of the cloud security team here at Google. You can find this podcast wherever podcasts are distributed and at our website when we, I promise, I promise, I promise, finally launch it. Until then and after, follow us on Twitter as well, twitter.com/cloudsecpodcast. We've got an awesome guest today. His name is Brandon Levine, and nobody expects this, but his title is Malware Inquisitor here at Google.

Well, I'm sure he has some kind of a boring title in our official records. I don't know, with a number and all that. So we'll stick to Inquisitor. I like that.

Yeah, they don't really enforce it too heavily on LinkedIn, so I can put whatever I want, which is really a nice perk.

But why not a Chief Malware Inquisitor or like an Inquisiting Lead or something?

I don't wanna overstate my capabilities, so it works out a little better just to under-promise, over-deliver.

So recently you were seen saying that for many organizations, maybe crimeware matters more than the nation-state threats. Would you care to defend this argument? Would you say that for some organizations, they don't have to care about state-sponsored threats, or how do you think about it?

Well, you're giving me a soapbox early, but that's an easy one for me to mantle. So there are a few key factors which contribute to why I state that this is the case. You know, first, the average cost of a breach is estimated at just under $4 million, with the average ransom payment is about $233,000. Actually, the highest ransom amount was roughly $30 million according to Palo Alto Networks. I've seen myself $20 million. Financially motivated threat actors must extract payment from victims, and they will go to exceptional lengths to increase their odds. This means going after backups, sensitive data, and threatening leaks. This is in opposition to, say, a recent type of thing where the Nobellum, the SolarWinds types of attacks, where they prioritize stealth over anything. And really the biggest problem here is that modern businesses are dependent on their data. It's customer data, customer lists, R&Ds, timetables. Data is a core of an organization's ability to operate. So a threat actor will follow the money, and that has led to the targeting of data in 2 complementary ways. You have disruption of an organization's ability to access data, things like hospital records in the case of UHS, local governments, which have hit multiple municipalities in the US. And that has surfaced as a prime monetization opportunity in the form of the ransomware itself. But you also have this interesting side effect where you have interruption of data control in the form of data leaks. This is seen in numerous dedicated leak sites operated by ransomware crews themselves. Really, we tend to overemphasize these really exceptional cases in the media. You know, outliers and at extremes garner more attention and more views. More views equals more money. You know, and cybercrime is generally mundane. It rarely gets the same coverage as nation-state actors. And it is important to understand as a CTI professional or anybody that is consuming CTI, Cyber threat intelligence, for those who aren't familiar with the acronym. You need to understand who is capable of attacking you, why they would attack you, and what their goals are. So when you place that in the context of risk, data access or data control is a major problem, and thus crimeware becomes a major, major focus, or should be a major focus for many organizations. Not discounting the threat of nation-state threat actors, you know, obviously losing your customer lists or losing your IP is extremely, extremely harmful to the company. Having all your emails leaked, especially for an espionage purpose is extremely harmful. But at the end of the day, you are much more likely to fall victim to cybercrime and end up with much higher financial damages, at least in the short term. So the prioritization, I think, is a little bit off in many cases.

Okay, that makes sense. And I think that sounds like it's both the chance of it happening and damage are likely to be higher for most companies or most organizations, right?

That's absolutely correct. Like, just the probability of you experiencing a cybercriminal trying to hit your organization doesn't even have to be targeted. Is astronomically higher than getting hit by a nation nexus or nation state threat actor.

One of the interesting things about human beings, not just security professionals, and security professionals are human beings, to be clear—

Are we?

Some of us— is that we're particularly bad at assessing risk. We focus on the things which are loudest and scariest, not remembering that really risk is probability of bad outcome times the pain of the bad outcome. We forget that this probability factor really dominates the things that we ought to be worried about.

And I think that's anchored in a little bit of how cybercrime used to really accomplish its goals. Back in the day, this was 2 or 3 years ago, you were nickel and diming people for $200, $500 from their bank accounts, from every bank account. Now you end up with threat actors that are targeting organizations and hitting $3 million, $4 million, up to $20 million a pop. So the ROI has increased, but also damages to the actual victims has increased as well. So I think it's rooted in a very old threat model, but you're 100% right. Like, we are terrible at risk assessment in general.

Never thought I would miss man in the browser and feel a longing for the era of Zeus dominating things we worried about. So how do we approach cybercrime mitigation here at Google? Because it's fun to talk about Google things.

So at Google, cybercrime mitigation is a multi-team effort that involves representatives from more than a dozen teams at any given time, depending on the vectors observed. So BotZ is the anti-botnet effort and is part of Google's Threat Analysis Group, likely the premier CTI organization, you know, just to toot our own horn. This group works in constant collaboration by developing short and long-term mitigation strategies and monitoring emergent threats. My personal involvement is not as a member of TAG, just to be clear, but as a member of Uppercase under Google Cloud. So I have a bit of a different perspective when I am participating in that effort.

When we're working on that, what are the biggest obstacles you've encountered?

One of the biggest things that I've had a problem grappling with is quantifying the value of trust, and specifically trust in Google products. It's very difficult to come up with a metric that says, well, people are getting malicious attachments in Gmail, or they're getting malicious links on Google Drive, and thus are less likely to trust Google as a result. And it's a very difficult issue at a company such as Google which thrives on metrics and measurements. So this quantitative sort of concept has been really hard for me to wrap my head around.

I think that's true for many of us in various other jobs. You know, I'm not really trying to shift attention here, but I think quantifying some of the things like you're describing is a bit of a challenge in many areas, not just in this one. So to kind of further continue down that path, presumably you're involved with different larger security community efforts. And it's not just about like using Google's all-knowing eye or whatever. So how do you work with the broader InfoSec community, or maybe I should say cyber community? No, I cringe when I say that. Sorry.

Oh no, that hurts.

That's a big cringe there. Google has traditionally been very selective about its interactions with the larger information security community. That's mostly derivative from the types of threats that Google itself faces being relatively, not necessarily unique, but complex and involved. So when it comes to cybercrime, This doesn't really work in our favor, as the campaigns are typically much farther reaching. It's not just Google, it's not just Google customers, and it's usually by orders of magnitude in the cases of potential victims. This has pushed the Cybercrime Working Group, BotZ, to more actively engage with our peers to collaborate on investigations, especially when Google infrastructure is involved. And that's been one of the biggest gains that we've made over the last year and a half, is just getting the InfoSec community to tell us, hey, we see this bad thing on your network, we're seeing this bad thing hosted in Google, can you help us deal with this? You know, we really want to bring down the ability of threat actors to abuse our services. I'd say we've made enormous strides in working with trusted working groups to help identify threats, but there's obviously always more to be done.

That's an interesting thing there is you say we hear about what other people are seeing. Does that inform what we work on? How do you prioritize what you investigate and what you track?

So prioritization is actually really specific within Google, and we break things into 3 tiers based on attacker TTPs, so tools, techniques, and procedures. Threat actors, the first tier is Tier 1, threat actors that actively monetize using Google services. These are things like ad fraud, YouTube fraud, things like that, things that could impact Google's bottom line. The second tier is a much broader tier where these are threat actors that utilize Google services to facilitate malware in any way. Things like, say, RaccoonStealer in the past, or a specific threat actor using Gozi. Those are things we tend to categorize as Tier 2. The third, which is Tier 3, are threat actors that impact Google service customers but not the services directly. So think of things that steal passwords from Chrome, dump cookies from Chrome, although that is actually generally a Tier 2 at this point. So within each tier, we assign a priority, which is again 1 through 3, based on measurable impact. This dictates the amount of analysts and other resources devoted to that particular threat.

Hmm.

So it sounds like a fairly elaborate process to decide where to focus your attention because obviously there are many samples, many groups doing malware. And it sounds like prioritization is a big piece at the center of your efforts. It's like, where do you shine your light? Where do you go and invest more time?

That's absolutely the case. We really need to make sure that we maximize our resources and don't end up spending time in rabbit holes of, okay, this malware is extremely interesting. It's written in Rust, for example, but it doesn't really impact a whole lot of victims, or we don't see it very often. We don't have a great way to interdict it, given Google's position in the attack chain. So we do commonly reprioritize. based on, you know, interdictions that we've put in place, mitigations that we've done, and what has been going on. And we generally do that every 2 weeks.

Ah, okay. So there's a kind of ongoing rebalancing, reweighing process, right?

Nothing is static, right? And honestly, things change even day to day. But every 2 weeks, that cadence has seemed to work out quite well for us in assigning the appropriate resources to understand what's going on. And we actually sync on every other week as well, for all of our analysts share what everybody is seeing. So all 3 tiers, tiers 1, 2, and 3, we share with the entire team, hey, these are the updates we're seeing, these are the changes we're seeing. And that also helps to inform us, you know, hey, this person may need more help because this thing is a much bigger deal than we thought. And it helps us really to come together and understand the— I hate this term— the threat landscape, so to speak.

That is the worst term possible.

It really is. But it does actually do a good job of describing the quote-unquote big picture, and that's really what the purpose is.

Yeah, I can actually make a contribution and make it even worse. When you hear people talk about the changing threat landscape.

Yes.

Because what do they assume, a static threat landscape? They expect like a bluster and like I love you worms to be the threat of the day for 30 years or something? So of course it changes. So why add the word changing threat landscape? Okay, and anyway, it's a bit of a rant. So let's get to some of the machinery of this. Maybe you can reveal some of it. So how do we actually track malware? Don't we need this dreaded word attribution for it to connect malware to a group? So can you reveal a bit more of the kitchen of how do we actually track malware?

I can talk a little bit about that, but I will caveat this with saying this is a bit of a philosophical debate. And I will say that how we, Google, organize things is not necessarily the best for other organizations to emulate. So I'm not saying that this is the holy Bible of ways to do things, just what we do. That said, we organize them the following way. At the highest level, we call threat actors groups. And these are the people we believe are running a particular set of malware campaigns. These are both the developers, the controllers, and the people that generally benefit from that malware. This can and often is distinct in the cases of distributors who have their own groups and are linked back to the operators. So you can have multiple groups involved in the same sort of campaign. And this is actually extremely common in the world of cybercrime. And underneath the groups, we have tools linked to them. And these are the things that you would expect, like TrickBot, Emotet, QuackBot, IcedID/BockBot. The benefit of this hierarchy is that a tool can be linked to multiple groups at once. Take, for example, Gozi/UrSniff. That's used by dozens of different threat actors. And if we were to try to attribute based on the malware itself, you'd end up in this circumstance where, okay, I don't know which threat actor this actually is that's using this. So that's one of the reasons why we sort of organize in this way.

So presumably you would use various sources of telemetry. You would use obviously certain logs, captures from different sources. So what are some of the more useful telemetry sources to study modern malware?

So if I was a defender, I would probably prioritize 2 things. I would say EDR is number one because host-based visibility is essential at this point. It's obviously not infallible, but if you have at minimum command line arguments being logged and centralized, you're in a pretty decent spot as compared to most peers as far as being able to at least One thing to consider on that point is that the primary investments by cybercriminals are towards landing on a target. They know if they can land initially, generally they will never get caught. This is sort of an artifact of the way that networks are designed. Traditionally, you have very poor east-west or lateral visibility versus your north-south visibility. The second thing that I would prioritize is event-based NDRs, which you can derive NetFlow from. as a side effect, which is awesome. So you get 2-in-1 there. Basically everything is encrypted at this point. It's trivial and/or free to get an SSL certificate, but there's a ton of value for monitoring things like DNS, total data transfer, TLS certs, et cetera. That metadata and correlating that to endpoint activity is really, really important and should not be relegated to the dumpster. I know organizations that take all their SSL metadata and just, oh, it's encrypted, I don't care about it. And they don't do anything with it. That's a deadly, deadly thing to do.

Hmm.

Yeah.

other type of situation. You really want both. If you have gaps in network visibility, then you're going to have pain when you're trying to work on host stuff and vice versa. And please, if you can, log lateral traffic between your endpoints and domain controllers. At the very least, at the very least, please, please, please log every network connection that goes to your domain controllers.

That is maybe the most prescriptive advice we've had on the podcast yet.

Well, I'm happy to help. You'd be surprised at the lengths that attackers go to when they see an Active Directory network is theirs for the picking. It is off to the races.

A very quick side question. You mentioned event-driven NDR. What was the event-driven? Like, this means that you're not just recording all traffic? Could you clarify this point?

What I mean by an event-based NDR, the typical comparison is between, you know, Suricata or Snort, which are signature-based IDS detection or network visibility, where you're only looking at a packet or at best a stream of packets. Event-based NDR like Zeek are much different. They actually are looking at the entire sessions and all of the sessions combined, and that means okay, here's this entire SSL session, here's this entire HTTP session, this SMB session, and the like. And it's very very useful, I find, for investigations, especially because you can also derive NetFlow from that and get a much higher level of statistical anomaly analysis that you can build out of that. Not saying that signature-based like Suricata or Snort are useless, but they certainly have their place. But I would at this time.

I think with the secular shift to things being encrypted, that's a very reasonable piece of recommendation there. I see a lot of organizations that when they do the cloud migration really step away from any kind of network monitoring that involves anything below the summary of flow logs, essentially.

To me, that's pretty reasonable, and that's really what I mean. Flow log is sort of a step above what you'd get from like an NDR because a flow is pretty much Source, dest, how much was transferred, and ports, right? Whereas event-based is still application-aware at some level. So I do still encourage to go that one level deeper if it's possible.

I think that's really reasonable advice. So I want to shift gears a little bit and actually want to ask you to predict the future. Where's ransomware going? Do they have a bright future?

That's a trick question, right? So ransomware is only going to continue to grow as more enhanced extortion schemes surface. And there's a really interesting one. At the start of 2021, we saw an interesting development in the case of multiple alleged breaches of Accellion FTA appliances, which lead to the malwareless ransomware of multiple organizations.

Hang on, what kind of devices were breached?

These are Accellion FTAs, so they're file transfer appliances. So back before the cloud or like Dropbox or Box, sending files via email was a huge pain, right? You had like a 5 megabyte limit. So an FTA was basically a way to say, hey, here's my big file that I want to transfer to you, Here's a link, basically setting up a point-to-point web server or what have you. A particular one had, which is EOL to their credit, like they know that this thing is old, it's end of life. There was a set of zero days in this appliance, which basically allowed for arbitrary read access of all the files. And it was effectively at its core SQL injection, you know, simplistic as it sounds. And what happened in this case is a threat actor took advantage of this and said, okay, give me all your files that are on this FTA. appliance, and that is generally every file that is transferred. And this turned out to be a very bad day for a number of organizations. And so there's no ransomware involved here. There's no malware. What's happening is this file access was monetized under the onus of basically data control. Like, you don't want sensitive things being leaked. Victims did pay up on this. So that's one sort of interesting sort of scenario.

Ransomware without the malware.

Yeah, exactly. It's just the extortion. And we also saw a little bit of this happen with the Exchange vulnerabilities, although that did actually involve malware. predicted, and so far I have unfortunately been wrong, or fortunately been wrong, it depends. I think on this one I'll be okay being wrong, that organizations that were vulnerable or compromised with the Exchange vulnerability recently, the on-premise one, would have emails exfiltrated and then the organization would be extorted to not release those emails. Hopefully nobody's listening and gets that great idea because it hasn't really happened yet as far as I know, but I suspect things like that will happen.

Well, bad guys, if you're listening, please do write us in and thank us for the good idea.

As far as traditional access to Intruder ransomware is not dead. That's a highly lucrative market. Orgs rely on their data, both control of it and access to it. Ransomware denies both. It's a high ROI, no-brainer for any mildly competent criminal.

Your conclusion is, yeah, bright future ahead.

Yeah, unfortunately.

So what are the changes in cybercrime operations that you've seen? So where do you see the threat actors making the biggest investments based on the evidence you see? What are they working on, if it's possible to conclude that?

The biggest investments, as I've mentioned a little bit earlier, are in initial access, landing on target, establishing that first beachhead. This is evident in the constantly updated cryptors, loaders, packers, delivery vehicles, lures, et cetera, et cetera. Once a threat actor gets into a network, they're generally home free because of the way that networks are instrumented and the way that most organizations' telemetry is configured or set up. And they use that to their advantage. And intrusion activity is— and this is post-compromise intrusion activity, is predominantly from open source red team tools. I don't see that changing anytime soon. This unfortunately frees up more development time for initial access. So you take tools that are off the shelf that are freely available. Okay, these work really well. Now I can put all of my dev resources, all of my money, all of my time into landing successfully on the target.

So basically initial landing is the harder part because lateral is so easy at most companies, right?

Correct.

So they would be thinking a lot more about the landing because there is no real need to think about lateral. Whatever you do is fine.

That's generally the case, yeah.

But this is sad.

It is. It's a little bit embarrassing. I think anybody that you've talked to in the last 15 years, if you ask them what their biggest pain point was, it's east-west visibility. But it's expensive in the way that networks have been designed, unfortunately. And I don't really see that changing with the way enterprise or on-premise networks are. I do believe a shift to cloud does absolutely help that quite a bit, as you're able to instrument the visibility quite a bit better. Cloud obviously brings about some more interesting attack vectors as well.

So this is a cloud security podcast. I want to pick on that a little. What do you see as the change for how criminals have to tool up when they go to compromise cloud environments as opposed to on-prem or corp networks?

So I think we'll see an increasing focus on cloud as a target for financial purposes. Right now, really what we see is, other than espionage, resource abuse. So crypto miners, things like that, or DDoSers. It'll be really interesting to see how threat actors pivot within those environments. The biggest focus is going to be on hitting IAM, right? So hitting identity services is going to be an absolute goldmine for threat actors.

Mm-hmm.

Fortunately, it's actually pretty complex and it does differ between the major cloud providers. So you may end up seeing crews that actually specialize in specific services, the Azures versus the AWSs versus the GCPs. I think that will be a really interesting game in the next year or 2. I think that initial access will probably still be the same, right? I'm going to go after cloud engineers. I'm going to go after cloud architects.

I'm going to go after these people.

That get access to that.

Well, that makes sense. I'm just picturing the criminal job board now, looking for attacker with 5 years experience, Google Cloud IAM.

Oh no, they are generally a little more friendly to not necessarily entry-level, but trusted or vetted. Certain criminals are a lot better at recruiting and vetting than most organizations are at this point.

Yeah, this sounds like a decent podcast episode, but—

Breaking into cyber.

You say depression, I say job security. So—

Touché, my friend. Too sure.

So I want to hit you with one of my favorite questions. So sometimes I get asked about predicting something related to threats, and I would usually kind of avoid the question because I say, you know, I don't want to predict what the threat actors would do. Frankly, if you predict something and you commit your resources based on your prediction and they do something else, you lost. So I was kind of hesitant to really go and predict because it may commit your resources in direction based on your thinking. But can you predict something related to malware, like what to expect What fun things we would see in the coming couple of years? I don't know, like open your imagination wide.

And so I don't really want to spoil a future talk that I'm giving, but in short, cybercrime operations have become much more specialized by role, and that has really come to the fore in the last 12 to 18 months. And I'll go a little bit deeper into it just to tease. When I mentioned distributors versus malware authors versus intrusion crews versus ransomware operators, those are distinct parts of an operation. And you can imagine that the organization involved to coordinate all of that is It's quite tricky. And we've seen threat actors that have like Jira to manage tickets. We've seen things like Elasticsearch backends where there's notations against their potential victims to understand viability of those victims. So while a lot of these guys probably do hang out in the same Jabber channels, there is quite a bit of coordination that goes on beyond, hey, I have this available, who can deal with it? I do believe that that specialization by role will really, really be exacerbated in the next couple months.

That makes a ton of sense. I find it so interesting the degree to which they're willing to engage in cooperation. When I was at a previous employer, we were monitoring some of what they were saying. We actually saw a post of somebody who had reverse engineered one of our pieces of defense technology, and their reverse engineering of it and their documentation of how it works was actually better than our internal documentation.

It's Shockingly how effective money can be as a driver.

Yes. And the thing that they were monetizing against, they were very successful monetizing against it. I'll put it that way, at risk of spilling any beans. So with this fun discussion of how bad guys perhaps are better at working together than we are, we'll end on that note. So thank you for listening today. You can find this podcast at Google Podcasts or wherever podcasts are sold. You can follow us on Twitter. twitter.com/cloudsecpodcast. My co-host is @anton_chuvakin, and I'm @_timpeacock. Brandon, thank you so much for joining us. You've been a terrifically fun guest. Listeners, you don't know this, but in the chat that we have going on the side to coordinate what we'll be saying next, we're already thinking about how we'll invite Brandon back for a future episode. And so with that, want to thank you so much for joining us. If you liked what you heard, Tweet at us, email, argue with us. If we like or hate what we hear, might invite you to join us here on the Cloud Security Podcast. See you next time.