Special: Coordinated Release of Detection Rules for CobaltStike Abuse

Security.

Hi there. Welcome to the Cloud Security Podcast by Google. Thanks for joining us today. Your hosts here, as ever, are myself, Timothy Peacock, the Senior Product Manager for Threat Detection here at Google Cloud, and Anton Chuvakin, a reformed analyst and senior staff in Google Cloud's Office of the CISO. You can find and subscribe to this podcast wherever your podcasts are available as well as at our website, cloud.withgoogle.com/cloudsecurity/podcast. If you like our content and want it delivered to you piping hot every Monday or off-cycle like this special release episode, please do hit that subscribe button. You can rate the podcast in your app of choice if you think highly of the show. And if you don't, send us a note and we'll try to get better. You can follow the show and argue with your hosts on Twitter as well, twitter.com/cloudsecpodcast. Anton, this is a special episode today. We're talking about Cobalt Strike.

Exactly. It would involve threats and, and fun stuff. So this is one of those where we kind of get excited about the threat reasons, so to say. And it's also a fascinating episode because it involves a legitimate tool that's used by bad actors.

This episode also involved one of the most clever ways we've ever heard of stealing cycles.

Oh, yes.

Oh, yeah.

Last one.

Listeners, Anton and I were just about dead when we wrapped our minds around what our guest was telling us. One of the fun things about being a podcast host, I'll say, is you get to realize when it takes a long time to catch up to what somebody has told you, because, I mean, sometimes it just does. And I think this was one of those cases. So with that, let's turn things over to today's guest. Today we are joined by Greg Sinclair, a security engineer here at Google Cloud. Greg, thank you so much for joining us today. Could you just give us a little bit about your background and how you ended up here at Google?

Sure. I've been taking things apart since I was a kid. My sister's still mad about several of the toys of hers I took apart to see how they work. That led into a career of being on a red team for a healthcare company in Chicago. From there, I moved into malware analysis and worked for iDefense for a while, and then From there, Novetta, and then Capital One, and then Google. So I've had quite a few years of taking malware apart. That's something I just particularly enjoy. It's— I'm one of those people whose hobby and career massively overlap.

You know, when I was in college, my father said to me, Tim, do what you love and you'll never work a day in your life. And I listened to that advice and I got a political science degree. And for a while there, it looked like I would never work a day in my life. But I don't think that's what people mean by that expression.

Well, in my case, it actually worked out pretty well because, again, I really enjoy what I do.

That's awesome. So tell us about the team you're on here at Google.

So I'm with the Uppercase research team. We are responsible for doing a bunch of things primarily related to the Chronicle product.

But you work with my product too. You also help out the Security Command Center folks.

Absolutely. Absolutely. We're kind of a far-reaching team in that sense. We do a lot of research, a lot of signature development, detection development, which is great. My particular little enclave on the team is responsible for taking pieces of code that are of interest and fully exploring what they are and hope to write better detections that we can then in turn feed into other teams and give to customers.

So to switch gears a little bit to our main subject, which is kind of, I guess, was sensitive for some time, but this is a special episode, so we are releasing detection code for certain types of Cobalt Strike, right? So we are all really excited about this, but could you tell us a bit of context and maybe share what this is and maybe why it's so valuable?

Okay, sure. So for those who don't know, Cobalt Strike is an off-the-shelf tool that can be used by— it's commonly used by red teams to simulate what an adversary would do: get a hold of your system and then lateral movement to try to compromise as much of the infrastructure as possible. That tool unfortunately has been leaked on the internet and is commonly used by a bunch of threat actors.

But it is still used by legitimate actors for sure.

Oh, absolutely. Absolutely. It's a legitimate tool. It's just that there are versions of it out there that are being used for nefarious purposes.

Mostly, predominantly, mostly malicious or Only for malicious purposes for these versions.

The leaked versions?

Yes.

I would say predominantly— well, I'll put it this way. The view that I have on it, I've only seen it being used with threat actors who are going after, you know, so, which definitely makes it a bit of a problem. But yes, there are plenty of red teams that use it for, you know, their jobs. You know, it's an important tool. So with that, having seen it so often, decided to create some signatures that would really kind of narrow in on the older versions of it, the ones, because typically the ones that are out there are the slightly, like, a version or 2 behind what's being sold. And we wanted to find a way to detect those. So, that's where this whole project started. I wanted to see what I could do to help detect those and, you know, in some sense, remediate the problem.

So, your outdated here is, as you pointed out, it's almost synonymous with Owned by malicious actors.

Or used by malicious actors.

Or derived from stolen version, or is the stolen version, and it's most likely to be just bad, right? That's the implication, is that the outdated version is probably used by a bad guy, not by the good guy who is testing.

Yes.

Yes.

So if you look in public repos like VirusTotal, for instance, you're going to see, generally speaking, the older versions for those who are familiar with the numbering scheme, for like version 4.4 and 4.5, which is a year or so old now, those are commonly out there. Version 4.7, which I believe is the latest version, you don't see a whole lot of that coming through. So that being the case, the assumption is, yes, that if it's older, it's probably being used for malicious purposes.

So I'm sorry for interrupting one second. I'm sorry for being a bit of an idiot here, but we do talk a lot about how threat actors are maybe in possession of hundreds of millions of dollars from successful ransomware attacks and other— attackers can just buy the last version too, right? That's not illegal. Well, it is illegal, but they can still buy the last version. So why don't they?

To their credit, HelpSystems, the company that currently produces Cobalt Strike, they do have a vetting process. So they will try not to sell to people that they can't verify are doing it for legitimate purposes.

Cool, what a creative idea. Can we apply it to pretty much every other tech?

Yeah, that'd be nice. But no, yeah, so that— yeah, again, to their credit, they, they do try to weed out people who are, you know, using it for purposes other than legitimate red team testing.

Well, that's pretty interesting. So that to me is surprising about this episode already. Like, there's actually some kind of sense of responsibility from the people making these tools. That's great.

Yeah, I mean, it's appreciated for sure. But as with anything, if it becomes popular enough, it gets leaked. Somebody will eventually upload it somewhere, or maybe somebody got compromised and it got stolen from them. So invariably, the new versions come out and get shared around. I don't believe I've seen— and this is just my personal view of the data— I don't believe I've seen version 4.7, the latest one released anywhere. So yeah, it's like I said, usually it's a version or 2 back from the latest is what you typically see being passed around.

So tell me this, why is this tool so popular?

Because it's easy. It's a kind of a point-and-shoot tool. It's written in Java. So you set up your server and it pops up a nice GUI interface. You say, I want to create A stager, what's called a stager. And that's the part that you drop on somebody's machine. And then that will in turn call back to your server to download what's called the beacon, which is the main, you know, it's the remote access tool, the RAT, the core component. And literally all you have to do is just say, put the IP address in of where you want to contact, and then it'll produce the binaries for you. So it's great. It doesn't require much. And then once you send it out, those things will, you know, all the— Active infections will call back into your server and you get a nice list of all the infected machines on there. You can right-click and go down and see attributes of the machines. Then you can issue commands all from, you know, like I said, all point and click. You know, there's no real skill. I don't mean to disparage people who are using it, but there's no real skill in terms of like, you know, you know, like you're not— it wasn't developed in-house, someone else, you know, by a state actor or anything. So yeah, it's really easy to use and anybody who can get their hands on it can. go and move across someone's system once they get a foothold.

But so here's the thing. You write a lot of rules, and you personally in Uppercase and other threat research teams. So what is surprising about this particular research? What is the fun part of this particular effort with detecting the outdated/malicious versions of Cobalt Strike?

So when you go into things like VirusTotal and type in Cobalt Strike, you'll see a bunch of samples. I mean, a ton of samples. When I was doing the research, I didn't want to have just kind of a I wanted to have very specific rules. So I can now tell you exactly what version of a beacon or a stager or various other components you're looking at. And one of the fun things that I did, and when I start going after a family, and I'm, in this case, I'm going to call Cobalt Strike a family because it's a family, you know, be a legitimate tool or not. I tend to try to find the earliest possible version and work my way back through time. With this, I was able to go all the way back to version 1.44 from like 2016-ish, I think it was. Before that, it was an open-source tool that was just strapped on top of the Metasploit framework, for those who are aware of what that is. So I like the fact that I was able to go that far back into time and see the way it's progressed. I've seen how they've changed their security internally. To try to combat some of the theft. So around version 4, for instance, the beacon component, which is like, again, that's like the core of the system on a victim's machine, that was encrypted beforehand. It wasn't encrypted. You could simply just extract it from the JAR file and bam, you had full access to it. Now you have to have a legitimate license key and it has to be decrypted and uses relatively impressive encryption for that. So they were actively trying to keep the theft down and keep it from being used. One thing that, speaking of fun things that I found, version 3.8 had a huge spike in the number of samples that I found. I mean, I'm talking about like millions and millions of samples.

Oh, wow.

Yeah. And we're like, what is this? Turns out there was another group called XM Rig that was taking a legitimate beacon, attaching it to their malware, which I believe was written in Go or had like a Go wrapper around it. And then they would change 1 or 2 bytes and produce millions of samples. And the reason they were doing this, it was a crypto miner. So they would drop it into— they uploaded to sandboxes and they were just stealing cycles because by having a Cobalt Strike beacon in there, it would trigger the AVs would say, oh wait, this is bad, we need to run this. And so it would run for a minute or 5. They were just stealing cycles by just changing 1 or 2 bits and using Cobalt Strike Wait a minute, wait a minute.

That's pretty cool. Wow. Actually, that is genuinely legit cool.

So you're, you're telling me that these guys figured out how to steal cycles for mining from the AV vendors by attaching malware to their miner and then getting the AV guys to detonate their sample?

Yeah, that's the current theory.

Yeah.

Oh, that's brilliant.

Yeah, it's, it's a—

and people complain about lack of innovation in our industry. Oh wait, wait a second.

Oh my goodness, that's brilliant.

Yeah, to the point I actually had to go through and modify the rule for 3.8 to not detect that very specific version of the beacon because the beacon also includes the configuration and everything inside of it. So I was looking at that exact configuration like, do not, you know, ignore this. So, and then once I did that, the number of samples dropped dramatically that I was finding for 3.8.

That's too funny. So one of the things that's fun about this episode, listeners, is that it's a special episode dropping off cadence. And we're dropping it today because this work is part of a broader effort. And so in security, we're not playing alone. It's a team sport now more than ever. So could you tell us about the coordinated disclosure aspects and how your work got tied up with all this?

Sure. So Google is not an AV company by itself.

Not yet, we're not.

Yeah. Well, currently we don't have an AV product and we wanted to have as much of an impact as possible. So the signatures by themselves aren't useful unless they're being actively deployed. Because at the end of the day, the goal here is to make this tool as useless as possible to the bad guys, not to the good guys, not to people doing red team stuff, but to the people who are passing around all the compromised or leaked versions. We want to raise the bar so that They have to find a new way to move laterally across someone's network, which for the threat actors who don't have that capability, who have been kind of boosted by the ease of Cobalt Strike, by removing that tool, we're making their life more difficult.

So you're not only getting rid of a tool though, when you think about it, you're also getting rid of all the expertise that's built up around that tool, all the playbooks to teach new bad guys how to use that tool. You're really taking away a pretty significant thing if you're able to take it away completely.

That's the goal. Yes, that would be the goal.

You're like burning the Library of Alexandria for these guys. That's— this is kind of mean.

Yeah, you— this is like a— it's probably a very, uh, imperfect analogy.

Well, you know, stealing millions of dollars from school systems with ransomware, it's also not nice too.

So— Also quite not nice.

That's true.

Yeah, there's that. Yeah, yeah. No, I mean, the analogy is that like ultimately you are destroying something really That the bad guys use, and it's a good thing, right?

Oh, yeah. They're bad guys. We want to do that. You're right. Sorry. This isn't mean. This is really great.

So to broaden this a little bit to other topics, which are kind of fun, in the past, I've heard you contributed research to Threat Horizons reports. Of course, we are now at number 4, and more are being cooked as we speak. So what's your take on this? What fun things you've contributed, if you can share? And just in general, anything you can tell us about this? I love the reports, and they've become better over time, for sure.

Hang on, Greg. Before you answer that, Anton, you said being cooked. Are they being cooked from, like, raw ingredients or from frozen foods? How are they being cooked? Is this, you know, cooked from scratch?

Uh, they're being cooked from using the secret recipe, so obviously I cannot disclose it, right?

Uh, listeners, that's a reference to a blog post that Anton and I wrote and argued about at lunch a long time ago. Greg, the question is about Threat Horizons reports, please.

Sure. So as part of my research, I generally write a lot of unnecessarily long reports. I tend to be— it's been politely referred to as verbose. But so as a result of that, some of that information has been funneled into the Threat Horizons report. I believe it was the SolarMarker work that I did, which is another tool for stealing data. So it's a great way for distilling an unnecessarily long report into something that is consumable by the masses. And we're looking forward to doing more of those as more of these reports come out. We now have a second reverse engineer on the team, a gentleman by the name of Josh, who is also producing reports. So we're hoping to see more and more of the information getting out to the general public.

You know, Greg, that's awesome. I really like those Threat Horizons reports. I think there have been a couple now that are quite interesting to me because, well, we have something in common, Greg, which is the Threat Horizons reports use your research They also rely a little bit on the research that some of my teams do, particularly when it comes to the impact of crypto mining malware in GCP. So it's an exciting piece of work, and I'm really excited to see those continue to be going out now. For a long time, I was very jealous of, you know, people who got to work on, say, the Verizon data breach report. And Verizon's reports aren't there yet, but, you know, that's like, it's a cool thing to aspire to. So we're getting up on time, and I want to ask our traditional closing questions. First, do you have a tip for people out there who might be interested in getting into reverse engineering? And then 2, do you have recommended reading for our audience?

Sure.

I get asked the question a lot, like, how do you get into reverse engineering? Well, the first thing you need more than anything is a curiosity about how things work. If you don't care about how things work, reverse engineering is not the job for you.

But you said interest rather than obsession, right? Interest is enough. You don't have to be obsessed.

You don't have to be obsessed.

But it helps, right?

It does help. It does help, yeah. But then it becomes all-consuming, and then your wife gets very mad at you for spending hours on your computer. That's the thing that's happened. But no, having a desire to understand how things work is important. And then when I started out doing this, there was no such thing as a class about reverse engineering. It was literally, you had to figure it out as you went along. There are now courses online, both free and paid, like you can go to college and get to take classes in reverse engineering now, which is great. There are also a ton of books that you can basically type in reverse engineering, look for a book by a guy named Michael Ligh, L-I-G-H. I worked with him in the past. He is a phenomenal reverse engineer and he wrote a book about how to analyze malware.

Huh, that's really cool.

Yeah, it's a great book. So I would recommend people, if you're interested in it, go out, just do some research. There are tons of blogs, there are tons of YouTube videos, courses and all that. Start small, take apart something that is relatively simple. Honestly, the best way to do it, if you really wanna know how something works, write a program in Visual Studio, compile it, and then take it apart. You already know what the source code looks like, so seeing how it looks on the backend is a great way of comparing. And there are also different languages you can write things in to reverse engineer. You can reverse engineer JavaScript, for instance. It's typically obfuscated, so that becomes more about reading code. Find what works best for you, and then just kind of, once you get your foot into the subject, then kind of expand out some.

Well, that's awesome and very approachable advice. I love the idea of just, well, do it. Just build a program and then undo it. That's awesome. I love that.

Yeah.

Our tips are not usually so possible to put into practice.

Hopefully that's a tip that people actually can use. And again, don't try to go over what you're comfortable with. I've seen a lot of beginning REs, reverse engineers, they want to take apart like Microsoft Word. Like, cool, but, you know, start small and work your way up because the whole, the whole, you know, idiom of you, you can't run before you can walk kind of thing.

That's fair. I certainly relate to that expression because until recently I couldn't walk very well. So Greg, thank you so much for joining us today. This has been a really fun look, and congrats again on the coordinated disclosure on this one. It's really exciting to, uh, take this tool out of the bad guys' hands so thoroughly.

Well, thanks for having me. I appreciate it.

And now we are at time. Thank you very much for listening, and of course, for subscribing. You can find this podcast at Google Podcasts, Apple Podcasts, Spotify, or wherever else you get your podcasts. Also, you can find us at our website, cloud.withgoogle.com/cloudsecurity/podcast. Please subscribe so that you don't miss episodes. You can follow us on Twitter, twitter.com/cloudsecpodcast. Your hosts are also on Twitter at Anton underscore Chuvakin and underscore Tim Pico. Tweet at us, email us, argue with us, and if you like or hate what we hear, we can invite you to the next episode. See you on the next Cloud Security Podcast episode.

Security.