EP211 Decoding the Underground: Google's Dual-Lens Threat Intelligence Magic

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts, as ever, are myself, Tim Peacock, the Senior PM for Risk Analytics in Google SecOps, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever you get your podcasts, as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you enjoy our content and want it delivered to you piping hot every Monday, please do hit that subscribe button. You can follow the show, argue with us and the rest of our Cloud Security Podcast listeners all on the Google Cloud Security Community page. We're talking about threat intelligence today. Anton, what do you think about that?

I don't think we are talking just about threat intelligence.

No, it's better than that.

Yes, much better. I think we're talking about how Google does threat intelligence.

How Google does threat intelligence, yes.

And to me, that's the magic. The magic is, in my Gartner days, I recall that we tracked something like 53 threat intel vendors, and I think that we were lamenting that the list is just not complete at all. But it was like way back, so I think some of them may have died by now. But the point is that threat intel was a favorite pastime by a lot of people, but doing it really well is super hard.

Oh, yeah.

And I think we do it really well.

One of the things I found really interesting in this episode was how much of this was not yet a science. And maybe never will be a science. Maybe my phrasing, not yet a science, is itself incorrect because so much of it sounded like it was necessarily art.

Yes, but I think it's actually even more fascinating than that because to me it felt like it's a combination of many sciences and many arts combined together. So it's sciences and arts.

Yeah, it's like a small liberal arts school mashed together.

Yes, that's probably a metaphor way too crisp, but yes.

Way too crisp?

Oh boy. Way too crisp.

Well, before Anton and I get into a fight about metaphors, listeners, as much as y'all might enjoy that, Let's turn things over to today's guest. Today, we are joined by Kimberly Goody, the Head of Intel Analysis and Production here at Google. Kimberly, thank you so much for joining us today. Delighted to dive in on this. We're talking about Google's threat intelligence today and how we've got a pretty interesting lens on the world, looking at sort of the traditional— you think of monitoring the dark web, looking at forums, as well as all the incident response data you could possibly imagine. And so could you share a little bit about how this sort of 2-perspective model informs our ability to attribute campaigns and disrupt them?

Yeah. First, thank you for having me today. When I think about this question in particular, there's really 3 aspects that I think interrelate and really inform a process or cycle that we go through on a regular basis. And so the 3 concepts here are prioritization, contextualization, and proactiveness. And so from a prioritization perspective, as you alluded to, we are as a company responding to intrusions that are happening every day around the world. And so we are able to see the threats that are actually having an impact on organizations directly. So these are the attacks that, you know, we could say those are working, those threat actors are being successful. And so that's one input that we can take to help us prioritize the groups that we're researching and we're tracking over time. The second benefit is being able to contextualize. So when we're able to associate these attacks or certain components of those attacks, such as the malware used during an intrusion, back to underground forum data, that can help us make decisions around attribution. So one example would be when we identify a malware that is sold as a service used during an intrusion, and that service, we may know from our research, registers or provides the command-and-control infrastructure to their clients. So the domains they will register on behalf of their clients. So when we see domains that match a certain pattern, that's not actually indicative of the person who's using the malware. That represents the service. And so that might not seem very important to some people, but it is a really important distinction for us because the users of the malware will have different goals and the response to an attack by any of those users could look very different. So think about a ransomware attack versus somebody who's stealing data to maybe gain a competitive advantage in stock trading. Very, very different ways that you would respond to those intrusions. And then the last note that I'll make here is that when we're able to identify specific actors that are operating in forums associated with the attack or the tools, we can then proactively monitor them. And that will tell us when there's a change. Yeah.

What kind of change are you thinking of? What changes with these folks?

Yeah, of course. So some of the changes might be adding new configurations that their clients can make in the malware. Or we've seen that in some cases with ransom notes, for example, the actors who are selling ransomware will give the customers the ability to change the note, or maybe they'll give them the ability to target certain file extensions for encryption. But we could also see cases where threat actors are looking for new partners, right? So maybe historically we've known that they've always used this malware, But now we see that alias looking for additional partners. So that tells us that their future attacks might look a bit different than we're seeing today.

Hmm. But a lot of this kind of relies on you knowing, well, the who question, answering the who question, namely attributing cyberattacks. So I know it's kind of a sensitive topic, and people still argue on social media about does attribution matter. There are very well-known characters who make a strong case that it really does. But can you walk us through a process of how your team uses to connect maybe incidents to public threat actors, given how complex it all is and given that threat actors don't always want to be identified? Again, without revealing any deep, dark secrets, how can this even be done?

Yeah, there's— I mean, like you said, people definitely don't want to be associated with the attacks, right? That's why they use aliases when they're communicating with one another. And sometimes they'll even use like a throwaway alias for a single thing, right, so that you can't even you have a harder time trying to tie those together in and of themselves. But at a high level, we're looking at the technical characteristics to identify patterns that might be consistent with prior activity, and then also trying to understand, have a broad understanding of what tools and services are available in the underground ecosystem. And so really, when I think about attribution, to me, it's at a high level, a series of really good questions that we're asking, right? So we're saying, is this tool publicly available? Has it been used by multiple actors? Is it only used by one actor?

Hmm.

Do the patterns of the domains, the registration or the domain structure, is that tied to a known threat actor? Has the victim been targeted before? Is there geopolitical context that we might pull in that would help us understand why that particular victim is being targeted? Is there infrastructure characteristics that are in common? So for example, like custom fields, certificates, ports, an internet service provider that a threat actor might be using. And all of these are questions that we'll ask to help us make an attribution decision. Now, any of those things by themselves, that does not tell you attribution. And so, anybody who is looking at one of those in isolation, please don't do that.

You can use it—

You heard it here, folks. Don't do that.

That's a very nuanced advice. Just don't do that.

That's so actionable. I love that. So, so many times our guests are like, well, you really need to think about your people and your organization, but here you're just saying, don't do it. That's great.

Well, I mean, you're likely to be wrong, right? So, I think it's more like a fingerprint, right? Whenever you're looking at all of these characteristics together, that's what really makes it unique. There's a recent suspected FIN11 case that I can point to with this, right?

Hmm.

So there was the vulnerability in the Clio Secure File Transfer sharing products. And we know FIN11, they are a group that going back the last 4 or 5 years has been targeting vulnerabilities in secure file transfer sharing systems. And they've been using that to steal data, which they then post publicly on their data leak site. And so when we see this, we're like, okay, well, that's very similar to FIN11. And so our little alarm bells are going off like, oh, this is FIN7, but we can't— or FIN11 rather. Sorry, getting all my actors mixed up. So we're thinking like, okay, this is FIN11, but that's not enough for us to say this is FIN11 just because the targeting and the types of data that they're stealing is consistent. But over time, we were able to build out a better picture. So we were able to find commands that we saw run by the threat actors in incidents that we were responding to, similarities in malware configuration and infrastructure. And then obviously, ultimately at the end, right, we have—

Mm-hmm.

credit being taken for these attacks on what is known as the Clop data leak site. And so all of that information then allowed us to increase our confidence level in that attribution over time. Hmm.

So if somebody wakes you up at 3:00 AM— sorry, I'm down to my favorite question.

Oh, the 3:00 AM test.

If somebody wakes you up at 3:00 AM and says, Kimberly, attribution is more art or more science?

Well, I don't know. I think it's kind of both, right?

I did it.

I did it. Is that an acceptable answer? Is it both?

Yes, yes.

Yeah, yeah, yeah.

We've had a lot of guests say, on one hand, on the other hand.

Yeah, no, I think, you know, going back to, I think one of the questions that you're asking here is like, okay, well, if we see a tool used in an intrusion, like, how do you tie that back to something that's for sale in a forum, right? If we're looking at it from that perspective, how do you make these connections? And so I'll kind of talk through, you know, and then you can tell me if this is an art or a science. We have timing attributes. We'll look at when did something emerge, what are the earliest samples of this thing, distribution. So we want to get a sense, particularly if we think it could be sold, what is this user base? So you can look at volume over time. You can also look at malware delivered over time as well, and whether those payloads you know to be exclusive to a single group or if you know those to be sold in forums already. uniqueness of the malware itself. So sometimes there might be some characteristics or features that you can like really zoom in on, or a combination of those that's not typical. And then there's also, I would say, kind of an element of luck as well. So in the right place at the right time. So for example, when I'm like, sometimes I get like laser focused and I, I'm very intent on, I must figure out Who is behind this? And I will search the oddest things. Like, I will tell you, like, I've searched, like, phrases, I've searched portions of domains, I've searched, you know, like, wildcarded IPs, etc., just to, like, try and find anybody who might be talking about a particular thing, right? And so there is this element, I would say, a little bit of luck as well.

That's not surprising at all. I mean, to me, this is some of the attribution based on attacker mistakes. Is ultimately goes into the bucket, right? It goes into the luck bucket. They made a mistake, you got them.

You know, that reminds me of the thing that people always say so self-righteously, oh, attackers only have to get lucky once, defenders have to be lucky all the time.

Oh, hate this one.

I hate that one. And I think this points to where attackers have to be lucky all the time or not reuse their phrases or else Kimberly's going to find you.

Well, I can hope.

We can hope. So a phrase that Anton and I both despise aside, I want to talk about tooling and aliasing and tracking people. How do you do this at scale? How does the team keep track of all this? And how do you relate tool names and aliases with what the actors are talking about? How does all that come together?

Yeah, so I can give kind of an example that actually ties in some of those elements. There was recently some research that we did on a malware called what we call ShadowLadder. It's not actually called that by the threat actor who sells that. And that's what we would call a crypter. And so that's a tool that threat actors are using to basically make their payloads unique so that they're less likely to be discovered by, you know, defenders or security appliances, etc. And so that was a service that— or that we suspected it was a service, rather. And we looked at samples, we saw samples were appearing in mid to late 2023. And so that kind of gives us a timeframe to narrow down, like, where are we going to search in these underground communities? And one of our analysts actually was like going through, and this definitely doesn't go to your point about scaling, but they were scrolling through all of kind of the topics that were posted on this one underground forum called Exploit. And the reason that they were looking there is because a lot of the malware families that it was delivering we knew were sold on Exploit. So we knew that the people using it had a presence there. And so they started with that forum. And so they were scrolling for hours.

Hmm.

And they eventually came across a post that had a demo video. And demo videos are great, right? Like, we can watch them, we can see, like, what other things did they have up on their screen at the time. In this case, they were downloading the malware from a link, and that link was still live. And so, we were actually able to download that malware and compare that to what we called ShadowLadder and confirm that those were one and the same, right?

Huh.

So, this used some of those attributes that I mentioned. I think, you know, from a scale perspective, right, we do collection at scale. So, right, those— the communities that I was referencing, like, that is information that we are collecting Not manually, right? That would be not feasible. And so, we have an archive of all of that that we can effectively search through and map back whenever we want to dig into that.

Interesting. So, it sounds like a lot of this is a really skillful combination of kind of art, science, and luck, and a few other things. It's kind of probably one of the most cross-domain things in our land of cyber. That's like the impression I'm getting from you is kind of like, so many sciences and so many arts mixed together.

Yeah, I think it's certainly not an easy thing to do. And I think, you know, one of the really great things about working here is that we have so many different sources of data. And I find that to be a bit of a gift, although it does— it can make things more time-consuming and complicated, but it really can tell you a more complete picture of what's happening, how broad a particular threat is, And it can also, you know, when we're able to kind of connect these different pieces together, we can tell a better story and I think make better predictions and provide warnings about how threats might change in the future.

Okay. That actually makes sense. So let's switch gears to sort of uplevel this a little bit. And clearly, intelligence informs many things, from like IR to product security. So I'm going to try to ask you a long-winded question and then say, Actually, tell me this. How does this make Google more secure? So that's what I'm really trying to ask.

Yeah, so I mean, I guess to work backwards a little bit, Google's a huge company. There's so many teams at Google that do security or security-related things. And so that's, again, like, it's wonderful to be able to work with all of these people who are focused on trying to make attackers' lives harder every day. But it does require a lot of coordination across these different teams. And so I think, you know, we talk about the incident response component. We have a whole team that kind of supports those incident responders, and they're the ones intaking that information from those victims, and they're trying to connect it to things that we already know about to help them respond to those attacks. Right. So they're trying to identify who's behind it. That information from incident response is also stuff that we can then take and say there's a new malware or there's a new vulnerability that's being exploited. That then we can take to protect the rest of Google users and Google customers because we can write detections for that malware and we can deploy that in some cases, right? Like we can identify websites that threat actors are using, and we can actually block those so that you, as a visitor to one of those websites, you will never actually connect to it, or it will display a warning to you. And so we're able to take that information and kind of roll that out to everybody. And then not only that, but obviously from the research, right? We have hundreds of researchers and analysts who are looking at threats on a day-to-day basis. And that's information that we kind of push out to the rest of teams at Google. And then, we are also working with the product teams as well to identify the gaps that are being exploited by the threat actors. So that might not necessarily always result in, like, let's say, infrastructure being blocked, but in some cases, that might mean looking at a policy. Like, is there a policy that is allowing the threat actor to take this action? And if so, then maybe let's talk about how we might want to close that gap. And another team that we would work with is the legal teams. And so one public example that we can point to was related to a malware called Cryptbot, and that was targeting users of Google Chrome. And so one of the teams that is part of Google Threat Intelligence Group, they were able to investigate that in concert with our Cybercrimes Investigations Group, and then the legal litigation team was able to take civil action against the malware distributors of Cryptbot. And so again, this is very, I think, high level and only touches the surface. But it kind of gives you some sense of how vast and how many different teams there are that we might work with at Google in order to make both users and customers safer.

That makes a ton of sense. Anton, I think you want to ask question number 5.

Yes, indeed. And I think that it's one of those detailed and possibly fun questions about what we do differently. You've seen sort of intel operations elsewhere. So what do we do differently than other orgs that focus on collecting and analyzing threat intel? And I'm not trying to push you to say we have AI, but just give us your take on this.

Yeah, I think for me, the vast amount of data that we have is something that does make us unique. We have telemetry, we have the underground forum data, we do the technical research at scale.

scale.

We are responding to those incidents. But maybe, I think there are some other companies that are also quite large that have data that is similar, you know, right? But one of the things that I think that we do really well is that we actually have a data model for these different sources of data.

Mm-hmm.

And what sets us apart is that we've kind of been in the business of modeling, for lack of better words, that data out for like the last decade, right? And so, we have a decade of modeled out data from intrusion operations, just as one example. And a lot of companies really, I would say, only started using these data models for threat intelligence to kind of document and contextualize certain types of information, whether that's like attribution assessments that they're making or technical characteristics or behaviors of the malware. But I think that's more of a recent development. I would say a lot of companies can't say like, we've been doing this for the last decade. And so I think that's one way that we are also unique.

Hmm.

In terms of AI, we're certainly looking at ways that we can use that to streamline our operations as well, particularly when it relates to like manual tasks or mundane tasks that we have to do that, to be honest, like we don't need an expert to do some of the tasks that we have to do on a day-to-day basis, right? And so we're really looking for AI to help us there. One good example is that we have a product that we push out every single day. It's one of our most read products for the Google Threat Intelligence product, and it's called News Analysis. And it basically takes things that are happening in the news and it summarizes those. And then an analyst will go through and kind of respond to what the news is saying. We'll, you know, sometimes tell people that the news is completely off base.

Hmm.

Or this is actually consistent, and here's some additional context for that. And so when we look at AI, one of the things that's really helpful is like, do we need an analyst to summarize the article? Like, probably an analyst shouldn't be wasting their time with that particular aspect of it. And so this is a simple example, but an easy case where we can then potentially respond to more of these news events that people are asking us about by kind of eliminating that step and leveraging AI to help us.

That makes a ton of sense. Interesting that there are so many things that you are just better at when you've been doing something for a long time because you've learned what not to do. You've got organizational connective tissue, and it's a hard thing to catch up on. So as much as I think Anton and I are both enjoying this, I have to pivot to my least favorite part of every show, which is our closing questions. One, do you have a tip to help people get more out of their threat intelligence? And 2, Do you have recommended reading? And the recommended reading could be anything. We've had people go from The Art of War to Mary Oliver and everything in between.

One tip I think in general actually would just be curious. I think that that implies to your threat intelligence offering too, right? It's like asking the right questions, understanding what are the threats that would impact me, how can I kind of better prioritize? Because, I mean, frankly, it's impossible for a single person or a single team to look at everything that is happening out there. And so you really have to narrow that down. And so I think asking yourself and coming up with what are the important questions to tailor that scope is probably important as you're sifting through the vast amounts of information that are available to you. And then for reading, actually, I really love historical fiction, like reading that for fun, but I do like when it contains at least elements of truth. And so I've also been getting really into just kind of more journalistic representations of history. I think that that generally sticks with me better. So one book I read recently was called Wild Swans: The 3 Daughters of China.

Huh.

And that talks about 3 generations of one family started in like the 1900s, and it's kind of told from the viewpoint of the author, the mother, and the grandmother.

Really?

I thought it was really fascinating for me as somebody who hasn't really focused on Asia personally. Like, that wasn't an area of the world that I focused on, for example, in school, etc. So it was just really interesting to hear about their kind of their lives growing up.

That sounds like an awesome recommendation. One of the things that always strikes me about Chinese history is how recent So many of the things were.

No, for sure. I mean, a lot of— I think too, like another— well, I could give book recommendations all day, but another one that I also read was Nothing to Envy, and that was the ordinary lives of North Koreans. And that one, I think, kind of similar too. It's like whenever you're reading about this, it sounds like it should have been something happening so long ago, but it was actually really recent, right? When you're reading about like the famines, et cetera, that they've experienced.

Oh, yeah.

All right, Kimberly, with that, thank you so much for joining us today.

Yeah, thank you so much for having me.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton underscore Chuvakin and underscore Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security running.