Recapture of Glory: The Return of Persistent Threat Actors

All right, let's go ahead and get started. All right, good morning everybody. Thank you for joining us this early in the morning. We're incredibly humbled that you decided to join us first thing this morning. So Jibran and I today, we want to really talk about how threat actors attempt to regain access to organizations after they've previously compromised them. Jibran and I have had the opportunity to work with organizations about 1,000 or so times working on intrusion investigations as threat actors have maybe initially broken into organizations, but also as threat actors have broken into organizations multiple times. When you think about it, a lot of times when an organization deals with a data security breach for the first time, their security posture and their security maturity is usually X. But after the first intrusion, they tend to find religion, and they do a lot of things differently afterwards. And so we want to share what some of the experiences are with respect to organizations kind of changing their security maturity after the fact, after the first intrusion, but also talk about what do threat actors actually do to recompromise environments multiple times. And we'll share a lot of case studies with respect to what we've actually observed from the front lines of working with over 1,000 organizations responding to breaches. So a little bit about ourselves. I'm Charles Carmichael. I'm a Senior Vice President and CTO of the FireEye Mandiant organization. I've been with the company for the last 8 and a half years, and I've worked with over 1,000 organizations responding to breaches. I started my career, for the first decade or so, I was mostly focused on offensive security. So I spent my time trying to find ways to creatively break into organizations. The second decade of my career, I spent most of my time responding to intrusions, typically orchestrated by foreign governments and organized criminal groups. Jibran?

Sure, so my name is Jibran Ilyas. I'm the Director of Incident Response for FireEye Mandiant in the Central Region of USA. And my background, I've been with a company called Trustwave, where I was part of the SOC, did some penetration testing, and came into incident response, worked at a company called Strauss Friedberg, and I've been here at Mandiant and FireEye for a year. I've been fortunate to speak at DEF CON, Black Hat, ThoughtCon. I also teach digital forensics at Northwestern University in Chicago. Got a bachelor's from Nepal and master's from Northwestern. And just a couple years ago, Crain's Chicago Business Magazine, they honored me with Crain's 40 Under 40. So that's just a little bit about me. And just like Charles said, we've had the fortune of responding to incidents and learning from them, and that's what we aim to share with all of you today.

Excellent. So from an agenda perspective, there are really 5 things that we're going to try to cover today. The first thing is we want to talk a little bit about the state of the hack. So who are the threat actors that we're talking about and that we typically respond to? We want to talk about ways in which threat actors, again, attempt to regain access to environments after they've been eradicated from them. We want to spend a bit of time going through some demos of real-world tools and techniques and malware that we see used in the wild. We want to provide some recommendations to the group to apply the lessons that we've learned over the years, and then we'll leave it open for a few minutes for Q&A for anybody that has any questions.

Sure. So, um, there are a few goals. Like, if we are able to show you the new tactics and techniques that these targeted threat actors are using, we'll consider that a success. If we're able to show you through a demo how attackers interact with victim networks, that will be a success. And because you've decided to join our session bright and early, I consider you all a family. So towards the end, we're going to share some lessons learned, and hopefully you'll be able to apply those techniques right away. And going back to the family point, there's 10 minutes for Q&A as well. So one theme that you're going to see over and over is the theme of missed findings in first intrusions. So we're all humans. We learn and evolve. And what we have learned from the front lines, we're going to teach you. So when you come up on the aisle, Feel free to— when you ask a question, we'll all learn through that. But feel free to share something that you want to share with the audience because, you know, we're here as a community. We're here to learn together. And that's what we aim to do in this session.

Yeah. And I want to emphasize a point that Jibran just mentioned. You know, we've had the benefit of working in this space for about 2 decades or so. So we've made a lot of mistakes throughout the years and incident response engagements that we've done. And we've learned from it, and we wanna share some of those learnings with the group as well. So we'll definitely highlight some of the things that we've personally done wrong that in hindsight we recognize that we could have done better.

Yep, absolutely. So let's talk about the state of the hack. So just wanna set the base for what we're seeing these days. So a lot of times we get to see hands-on keyboard operators. So the point that I made earlier It's not always when attackers send a malware and the malware does all the work until the data exfiltration point. We see attackers interacting with the victim networks and then adjusting. That's a very important point. So when they see something, when they see a certain security software, they adjust, they adopt to that particular network. Living off the land attacks. So in a lot of the breaches, the malware doesn't fulfill the whole lifecycle of the attack. The attackers have some sort of remote access through the malware. They like to switch to what we call living off the land attacks. What that really means is that there are a lot of binaries within Windows operating systems or even Linux operating systems which can accomplish the tasks that attackers previously used to accomplish via malware. And when we do the demo, we're gonna demonstrate some of that. The utilities like certutil, PowerShell, a lot of the utilities that network administrators use, attackers like to use it because they wanna stay under the radar. And the next point is just leading up to that. They believe in slow and steady attacks. Because their mission is to stay in the organization for a number of years and forever if they can. They're not in a huge rush because they wanna do espionage, they wanna read emails, so they don't wanna actually rush anything to basically risk detection. Hiding in plain sight, that is another important characteristic. Rather than calling their files like, evil.exe or hacker.exe, you will often find these attackers naming their files after legitimate Windows binaries. So if you use Windows operating system, you know of a very common binary called svchost.exe, which runs out of System32 folder. So attackers make it a point to not only change the names of those binaries to like svchost and other common binaries that are in Windows, But they also do some timestomping. What that means is that rather than having the date of that file as like recent, they would actually change the time to the time that operating system was installed. What does that give them? 'Cause even if you're looking at it, you might say, hey, this is svchost, this was actually put on the system right when the operating system was installed.

Okay.

I may not wanna do anything with it. So they like to hide in plain sight. The social engineering aspect is built in throughout the attack lifecycle, not just in the phishing email or the initial intrusion phase. They use that throughout. Like I said, they watch the victim organizations, they watch how they respond. If they're reading emails, then they know what the administrators are gonna do for remediation.

Yeah, it's actually really common for us to see threat actors reading the emails of the IT security professionals that are actually working on the incident response or the containment. And it's pretty damning to our clients when the threat actor reads a remediation plan because they know exactly what systems have they identified malware on, and so they could do things to, you know, insert additional backdoors or find additional ways in which they could get access to the environment once the containment event or the eradication event occurs.

Right. And Charles, pretty recently, what I saw in a ransomware attack, the company, you know, the board, they were like talking about how much coverage do we have from insurance. And attackers were reading those mailboxes. And when they actually chose the ransom amount, you want to guess what the amount was? Exactly. They basically chose the amount that, you know, pretty close to what they had coverage for. They actually left a little room for forensic investigation and outside counsel fee. That was pretty interesting. Said, you know what, we're not gonna just try to do all the way. But yeah, those are the kind of things that we see attackers adapting to. And then last but not the least, attackers also use compromised networks for attacks. So they may compromise, one of the things that investigators do when they're looking at IP addresses, We're looking for common outliers like Tor IPs, VPN IPs, cloud services IPs. Attackers know that if you use something like that, their chances of getting detected are pretty high. So what they might do is they compromise a regular user system so that when they come to attack, the IP that shows up in the logs of the victims is like some DSL or cable modem IP, and investigators might think that this is some home user. So that is also something that we're seeing. So what do adversaries know? Why do they have so much confidence in not just the first intrusion but subsequent intrusions? They know that humans are always going to be vulnerable. There's going to be somebody who's likely to take the phishing attack, click on an attachment, click on a link. So that's one thing.

Yeah, and Jibran, just to interject, it's not just phishing emails that we see attackers do to exploit the human trust. A lot of us reuse similar passwords across different websites, and we see a lot of threat actors, what they'll do is they'll get access to common dumps of passwords from, say for example, LinkedIn or various other sources that have been previously hacked. Or they'll break into organizations, download all the password hashes for those, um, for the users of those services, and then attempt to crack those passwords and then leverage those passwords to attempt to log into systems or resources for organizations that they're targeting. And if you look at, you know, our intrusion base or our caseload, we're starting to see that, you know, the, the second most prevalent attack vector that threat actors are using to break into organizations is credential reuse. And it may not be the exact password and username that was previously used by somebody for a previous intrusion. It may be a variation of that. So for example, if their password was password1 a year ago, their password might be password2 this year, or some, some predictable sequence of passwords. And so that's becoming very quickly probably the second most common way that we see threat actors breaking into organizations.

Great point. Yep. Um, so the other point is that, um, Investigators and blue teams in general, we strive to get full visibility in the network, but as you know, when you deploy any IT tool or software, we don't generally get 100% success, right? So it's quite an iterative process where you do first round, second round. So attackers know that investigation teams also have constraints. Some folks could have budget constraints, bandwidth constraints. Like if you have a live attacker in the environment and then the internal team is responding, they've got to sleep, they got to spend time with family, they can't be up 24/7. So attackers know that that's a concern, I mean, that's a factor in investigations. The software point I made earlier. And then also when they adopt, so going back to my point about adjusting and adopting per visibility that attackers have in the environment. So when they change a little tactic, like for example, if a particular malware or a particular script that they wrote has a domain name, That's in there that it calls out to get its commands. And the victim organization is looking at the firewall logs and blocks that IP that's where the unauthorized traffic is going. What the attackers could do when they stop seeing data coming from that organization, what they can do is change the DNS record of their domain name. on where they purchased their domain name from. And then they could basically, they basically change the IP address where everything is gonna go. So that tactic is pretty common where attackers, when they stop seeing data coming, they just change the DNS record and then whatever IP they change it to, that's where the data starts coming. So I just mentioned that. Go ahead.

Yeah, so There's really maybe 2 broad categories of threat actors that we tend to deal with. Those that are incredibly opportunistic, and they don't really care who they compromise. They're just typically financially motivated, and they, they do things like, you know, maybe deploy ransomware at organizations, or they steal data and then they extort those organizations for money in exchange for not publicizing the information. So again, they don't necessarily— these opportunistic attackers don't necessarily care who they target, um, but if they can break into an organization, they'll do something there. As opposed to certain threat actors that are incredibly targeted, and there is a specific reason why they're trying to break into a particular organization. And typically those actors that we see that are very targeted generally tend to be state-sponsored actors. So a government entity is either tasking a company or a military unit or an intelligence agency to break into Company X for whatever reason. And a lot of times that reason is to steal specific data related to Maybe it's intellectual property, so the know-how to make aircraft engines or the know-how to produce enzymes or some specific reason. And typically, when we see targeted threat actors break into organizations, when they steal intellectual property related to whatever they're stealing, they don't just want the know-how or the IP from 2017 or 2018. They want to continue to steal the innovation year after year after year. And so you tend to see a lot of recompromises or at least attempted recompromises by, by state-sponsored threat actors and organizations that have data that's going to continue to evolve, that's going to continue to innovate, that governments want access to. We sometimes talk about attacks as malware, and Jibran touched on this before, but I want to emphasize a key point here. Malware is just typically a tool that's used by attackers to facilitate their goals or to facilitate their mission. A lot of times when people talk about ransomware, they talk about the software itself. And quite frankly, the software itself isn't that important. I want to emphasize that it's the human operators behind the tools that are actually deploying the encryptors, as an example. And, you know, typically when you hear about ransomware incidents nowadays, it's not a situation where somebody got a malicious email, they double-clicked on it, and then all of a sudden the company went offline. There's usually hands on keyboard by an attacker that's moving around laterally, escalating privileges, gets enough access to enough core systems within the environment, and then will manually push out an encryptor across the enterprise. And they could do that through GPOs, or PsExec with batch files, or, you know, a number of other software deployment mechanisms. But again, it's a who, it's not a what.

Right, and another important point. One thing that I've seen in recent investigations, in ransomware investigations, they come in, and, you know, one of the first things that they might try to do is get an inventory of all the security software, and they might wanna turn that off. So they create custom scripts to turn off the security software that's running on the box that they're gonna use as staging server. And one other thing, they do some internal recon to basically find out which are the key systems like a database server or the backup server because they wanna delete all the backups so that there are higher chances of the victim organization paying the ransom. And they do that to all important servers, not just like database server and the backup server, but think of like Veeam Backup Server or hypervisor where you have a lot of virtual machines. So basically want to encrypt the most critical assets.

So every year we release a report that we call mTrends, and what we do is we try to identify what are the themes and the trends that we observe in our caseload data for the prior year. So in 2019, what we found was that of the organizations that we did incident response work for, A lot of times when we do IR work for organizations, they engage us for our managed services so that we can continue to put eyes on glass and hopefully stop attackers from recompromising environments. What we found in those situations where we'd done both incident response work for and where the organizations were managed defense clients of ours, we saw that 31% of them were reattacked by the threat actor, usually the same threat actor that had previously had access to the environment. In the vast majority of the cases, we were able to stop the attacker right at the front. Some situations, maybe they compromised a system or 2, but we were able to pounce on that fairly quickly. But the goal is, for the threat actor, they want to get back into environments many times because they want to continue to steal data from those organizations.

Okay, so Charles talked about first attack and then the subsequent attacks. So this is just an example of like how attackers adjust from one attack to the other attack. So one of the things that I've noted in my incident response career is that when organizations suffer a targeted attack or data breach, the defenses that they invest in are based on the knowledge that was acquired from the attack from the previous attack. And that's pretty natural, right? 'Cause you basically know that this happened, that you wanna block that or you wanna have visibility there or you wanna have some remediation there. So attackers, obviously when they're coming in the second time, they know that the defense, the security posture is gonna be improved the second time around. So what do they do? So the first time around, like you might see You might see an attacker just using Mimikatz binary, because, and then they know if the defense is gonna be better, then they move to something like running Mimikatz via PowerShell. So PowerShell again is a utility that's built into Windows, and you can run it, you can invoke a Mimikatz PowerShell script like this. Then the third time around, they might realize that any type of Mimikatz that we're using in the network is getting caught. So what they might do is they take a proc dump of LSASS process. So LSASS process, you can take a process dump of that and then export that to the attacker server, and then you can run Mimikatz offline. So basically on the victim network, there's no trace of Mimikatz, All there is is a trace of ProcDump, which is a legit Sysinternals utility that's used to dump process memory by a lot of developers and the people who are troubleshooting the network. So even that, in our world in incident response and stock world, people know that if they see ProcDump, we have to at least do some follow-up. So what might the attackers do the 4th time? There's a tactic to dump LSASS memory through rundll32. So rundll32, again, is a Windows binary that's run for installations of a lot of things, and attackers could use that with this syntax, and they could still take the memory dump and export the memory dump for Mimikatz offline cracking. So just as you're thinking about your defenses, Obviously do everything that you know that's out there, but also be looking out for these red team techniques that are out there, the researchers are posting, because what the attackers do, they're also looking at the red team efforts, like what innovation red teamers are bringing, and they might use some of those techniques in an attack. in an actual attack? And we see quite a bit of that, right?

So I wanna actually dive into these. So just going back to the first use. So years ago, we saw a lot of attackers copying over Mimikatz onto systems. They would execute it, antivirus wouldn't detect it. And it was a pretty common thing for them to be able to bring in Mimikatz onto systems. Antivirus is getting better, EDR is getting better. A lot of people are now detecting that. So what a lot of red teamers, as well as attackers, started doing was they decided, hey, you know what, we're gonna use PowerShell. And years ago, most people were ignoring PowerShell from an IT administrator security perspective. And so PowerShell was kind of the new technique that people were using to try to bypass some of the antivirus software. But now we're getting a lot better at detecting malicious stuff in PowerShell. And actually, nowadays I feel like sometimes when we see big PowerShell blobs, we just kind of assume that it's probably evil by default. And so fast forward to what we've started seeing a lot lately, and we've seen it over the past few years, You know, the attackers leveraging tools like ProcDump, which is released by— it's a Sysinternals tool released by Microsoft. It's digitally signed. And in it, use in it by itself isn't actually malicious. But obviously when you see somebody dumping the LSASS process, it's obvious that it's malicious. And so again, we are seeing attackers leveraging new techniques to try to circumvent detections. And we tend to see these techniques start to evolve. sometimes in second and third intrusions of organizations.

Yep.

So, if you think about how are threat actors typically returning to environments after they get kicked out of networks, there's really 4 different ways that we see threat actors doing that. The first way is we see them leveraging knowledge that was obtained from the initial intrusion. It's pretty common for us to see threat actors writing tools like Mimikatz, or dumping hashes from domain controllers, And then essentially getting every username and password hash combination out there. And we see a— we know that they're doing a lot of password cracking offline because we see them leveraging the credentials that they, uh, that they were able to steal. From organizations' perspective, when they go through a containment effort, usually the hardest part of eradicating a threat actor from the environment is going through an enterprise password reset. It typically takes the longest part, the longest time if you look at the grand scheme of all the things that an organization needs to do to eradicate a threat actor from an organization. And a lot of times we see organizations choosing not to reset every single password across the enterprise. Sometimes they will focus on just domain administrator accounts. Sometimes they focus on domain admins and then domain user accounts. But what we commonly see is that organizations choose not to reset passwords for service accounts or other application accounts, because I promise you, When you do that, you will break something. It happens every single time. And a lot of organizations choose not to take that risk because they don't want to break things. And so it's really common for us to see threat actors log back into the environment using a service account password that wasn't changed. So it's, again, it's something that we emphasize to our clients. If you're going to go through an enterprise password reset process, you really do have to change all the passwords within Active Directory or the scope of the environment that you have— that they had access to.

Yeah.

We commonly see threat actors deploying backdoors in environments that are sometimes missed by the incident response organization that's, you know, doing the investigation. So we've missed backdoors as we scanned environments, and sometimes we see situations where we tell our clients that there's a backdoor on system X, and there are so many remediation steps that the organization has to take, and they forget to remove the malware from that particular system. Or sometimes we see organizations restoring virtual machine snapshots to versions of which had malware on it previously. We see supply chain attacks. So we see that from a first intrusion perspective. So sometimes you'll see threat actors break into big technology outsourcers and then leverage connectivity to other environments. We see that in recompromise situations quite often. And we also see attackers attempt to very aggressively recompromise environments through additional phishing campaigns, the identification and exploitation of new vulnerabilities, or sometimes they very quickly leverage new CVEs to gain access to organizations.

Sure, so what's some of the reusable things or data points that attackers can use for subsequent attacks? So Active Directory database, we see a use of a tool called BloodHound, So when attackers are able to get in the network, they run a tool called BloodHound, which basically maps the network. And it has a— you can call it everything that you want to know about Active Directory, like what are the users, what groups they're in, what kind of permissions they have, what trust relationships with other domains it has. So that basically becomes their intelligence data that they use for subsequent attacks. Like Charles mentioned earlier, if they take the Active Directory dump 1, 2, and 3 times, then they can crack the passwords for all those Active Directory databases and know which users typically choose weak passwords, so then they target those users over and over. And then also, what is the pattern of passwords for other users that they might want to target? If you have 3 passwords of a user, you might have a shot at guessing the 4th one.

Wow.

So that's one of the things that we see. Password repositories, so a lot of people use tools like KeePass, LastPass. If attackers get access to that, it doesn't just have the Active Directory password, it has passwords for other portals that are on the network. When attackers compromise emails, they like to forward email rules, so if So even if you reset the password of the victim, the emails with certain keywords that the rules that they create could be still going to the attacker. They could also set delegate access, so they could say that this X user can now read emails for this other mailbox, and then they're able to get access to multiple users in the network. So Organization charts, network diagrams, network configs, data from internal portals. Like, you know, we store a lot of data in our wiki portal, Jira portal, all that's important. When they do internal reconnaissance, they also get to know which third-party networks have VPN access to the network. So they know like in the back of their mind that, hey, this time let's say all of our access gets terminated, this organization that has VPN access into this environment, maybe we'll target that next time. And when they take keystroke logs, they also have a lot of intelligence on users and their passwords. So what are some of the planted backdoors, or how do they set up base for future attacks? So we're gonna talk about each one of them. RDP tunneling, we're gonna do a demo of it. So basically, if you wanted to access somebody's network via RDP, obviously you don't have inbound firewall rules to be able to do that. So how do you do it via reverse connection? So we're going to see a live demo of that.

Yeah, we see a lot of threat actors doing that. It's a favorite of FIN8 as they try to establish backdoors. And again, you're using legitimate software that's digitally signed. Essentially, it's the use of PuTTY Link. It's a pretty easy SSH tunneling backdoor, which is also used by— legitimately used by administrators as well.

Yep. And then a web shell. So when they exploit a vulnerability on a web server, one of the things that they want to place on that web server is a web shell so they could actually control that web server via port 80, which is always open. And basically this is an example of a very simple web shell. You type in a command and then it gives you the output at the bottom.

Scheduled task.

So if you wanted to give yourself reverse shell access every week, or, you know, in this case you could see that every, right around 1:55 AM every Saturday, the script runs and basically gives reverse shell access. Some of the missed findings, turn to Charles.

Yeah, so a lot of times when we do incident response work, we assume that the attack path is a very linear path. So we assume there's a patient zero and that the threat actor was able to essentially move from one system to the next, to the next, to the next. What we sometimes fail to realize is that sometimes there are multiple patient zeros. So sometimes a threat actor may send phishing emails to multiple users, and they may compromise multiple Windows endpoints and then have multiple different paths to ultimately get access to the data that they want. The other important thing to keep in mind is that sometimes there are multiple threat actors in an environment. So we, in about 15% of the cases that we worked on last year, we saw more than one distinct threat actor in an organization's environment. And what I mean by that is I'm not saying that we find malware on systems that, you know, were written by different authors. We're talking about there are 2 distinct groups, or at least 2 distinct groups, with hands on keyboard in victim environments. And so, again, when you think about, you know, conducting an incident response engagement, you have to you have to take a fairly broad look at the environment in the event that there are multiple attack paths or even multiple threat actors. And typically from a red teaming perspective, a lot of times red teamers really focus on the path of least resistance to get access to maybe a certain system or to meet certain objectives. And they typically follow a linear path. Sometimes they might branch off and try to find other ways to break in, but, but they're not trying to be exhaustive to find all different avenues or paths into an organization. Because they're constrained by time, and we all need to recognize that it's not necessarily a linear path. So just real quickly, it's a fairly just obvious visual. So typically, again, you could have multiple patient zeros, and a lot of times you may end up finding commodity malware in an organization's environment that has nothing to do with the targeted or hands-on keyboard intrusion that exists. It just may be malware that needs to be taken care of and needs to be remediated, but may not be directly related to the hands-on keyboard activity. We talked about investigations being iterative. Typically, when you identify evidence of compromise, you want to answer a few fundamental questions. How did the attacker get access to that particular system? What did they do when they were on that system? And then what other systems did the attacker access from that system? And as you start to build out the web, it's important to— and as you find new evidence of compromise, it's important to take a look at the entire enterprise to look for evidence of either the tools that the attackers were using or the techniques that they were using to try to scope out the total compromise in an environment.

Okay, so like Charles said, we're humans, we miss things, and then we learn. So one, when we are doing investigations, these are typically the things that we see internal organizations miss, or even instant response companies met. So I wanna highlight some of these for you. So when I mentioned that KeePass or some password repositories, more often than not, the network administrators are storing their credentials for firewalls, routers, and typically when organizations do remediation, they're changing or resetting the passwords for Active Directory account, but not necessarily for every single networking device that they have in their network. So if an attacker has access to firewall or border router, they could still make their way in, and that could facilitate the second or third attack. We know that some systems don't report to the central command, like some endpoint protection coverage might not be on 100% of the environment because of errors or just not coverage. Legacy remote access portals with single-factor authentication. We see that a lot where when companies get a multi-factor authentication as their security improvement, they might leave the single-factor one on for a time and they may forget about it. Legitimate remote access software. So we see the use of TeamViewer, Screen Connect, LogMeIn by the threat actors as well. 'Cause if they see LogMeIn or TeamViewer is being used legitimately by the organization, they might not even have a command and control malware. They might say, hey, TeamViewer is just gonna blend in, so how about I just install TeamViewer and access the network that way? And you can install TeamViewer in a persistent state, so when your system boots up, see that. On Linux servers, you can change SSH daemon settings to not log access from from a certain IP, or just not log remote access at all. Timestamp files, we're gonna show you an example. So just the key point here is that investigations, we do a lot of, in investigations we do a lot of timelining. So a single timestamping feature by the attackers could throw us off investigation. Sleeper backdoors, we're gonna show you an example of that. Malware with DLL virtual hijacking, we're gonna do a demo for that. And then last but not the least, the file transfers via legitimate cloud services like Google Drive or OneDrive. So this is an example of timestomping. Basically what this command is doing is saying this WBS, it's an attacker tool, make it so that the timestamps for this matches command.exe. But there is a way to forensically find out the real timestamp of that. It's called the filename attribute in master file table. So that there is a way for you to do it. This is an example of sleeper, backdoor malware. Basically what the concept is that there is a domain name here, evilsite.com. I mean, not to reveal the real domain name, but so if this is the attacker's domain, They could go to the DNS records and change that to localhost address, 127.0.0.1, so that when the victim machine is looking for that domain name, it's not resolving to an external IP, and if somebody's watching out for command and control, then it's not detected. This is again, if this is the malware, what you would see on the victim box is just connections to the localhost, which is not that suspicious. If it was going to an external IP that was not recognized, that would be something bad. So on the Linux servers, we get to see Linux servers as orphan servers most, and I don't know what the big reason is, but it's usually like a lot of blue teamers and defense organizations, when they see a lot of Windows malware, they have a bias of like, hey, this malware that we've seen is Windows-based, so we don't know if they're really going to the Linux servers. So there's a lot of that, and as a result, we see a lot of backdoors on Linux, reverse shells, and a lot of exfiltration done through Linux servers.

Yeah, sometimes we see threat actors waiting for new CVEs to get published so that they can quickly weaponize it. And we saw this in a few different situations. We saw this with Heartbleed several years ago. And I distinctly remember the first time I saw Heartbleed used was by a group we call APT18, which is a commercially contracted entity that the Chinese government uses to conduct offensive operations. And I remember when I first saw it, I was actually initially skeptical that Heartbleed was used to break into this organization. But after a few hours of investigating, we determined, or we learned, that APT18 was basically doing an attack against say Juniper SSL VPN and doing a session hijacking attack. And so again, they really wanted to get access to a nuclear environment and they were able to do that. We've seen a lot of exploitation recently of the Citrix NetScaler ADC zero-day vulnerability, and we see threat actors getting pretty broad access to environments because of that. So let's just go into the demos now so we can show you the use of some of the attack techniques that we see.

Okay. So, I don't know, have you ever been in a live demo situation? Sometimes technical difficulties could occur. So what I wanna do before I start the demo is kind of do a collective prayer to the demo gods. So when I ask the question, what time is it? Let's just say demo time really loud so the demo gods hear us and maybe all the demos work. So ready for that?

Yeah.

All right, what time is it?

Demo time!

Okay, well maybe a little louder so we're more effective. What time is it?

Demo time!

All right, let's do that. Okay, so let me kinda explain the setup here. So this is a virtual machine. What I've done is think of attacker web server running on localhost. So this is XAMPP control panel. This is our web server. It's running on port 80 and 443. This is where the victim organization is gonna download, the malware's gonna download the files from and upload the data to. Then what I have here is basically 2 files that are gonna do the unauthorized activity. So this file, I just wanted to show you, it is digitally signed binary, so if you, this is the process that you're like hunting for, you're checking, you might see, you might see digital signatures and you might ignore. It's the DLL that gets loaded because of a search order hijacking trick. So basically, if this was in a Symantec folder or this was in the software folder, it would have loaded the DLL that was legitimate DLL. But because it's outside of that folder, it looks for the DLL in the same folder, and that is why, you know, it is able to do— perform some tasks that are instructed by the DLL.

Yeah, and a few things to highlight here. So DLL search order hijacking is incredibly prevalent. It impacts a lot of different applications and technology, and we just happen to be using a Symantec binary. We're not necessarily poking fun at them, it's an older version of their software, and so we just— it happened to be something that used this particular library that we wanted to test with.

Right, and one other quick thing I'll do, I'll just do the MD5 sum and we'll just look at the hashes in VirusTotal just to kind of show you that a lot of investigators and blue teamers, when they find a suspicious binary, they put it in VirusTotal to figure out if that's something bad. One quick tip, don't ever upload the actual binary, 'cause VirusTotal database is accessible to researchers and attackers for that matter, so always submit the hash. And in this hash, if a blue teamer was looking at the hash, they would see all of this as clean. And for the other one, the DLL, we could do the same thing, take the MD5 hash and put that in VirusTotal to see if that actually shows up as something bad. And for this one, if memory serves right, because it's a custom file that attackers would create for a particular victim and not use it for any other victim, you don't have a pattern, so this file is not even found. Nobody has ever even submitted that file. So without further ado, let's just run the malware. But let me show you, so this is basically the the command.txt exists on the web server that attackers control. And the first command that attackers are gonna, after the malware installation, they're gonna say, hey, do the keylogging. Very common thing that attackers do to understand or gain some intel on that box. So if the command is keylog on the web server and I run this malware, it's gonna download certain files via certutil. I'm also gonna enable verbosity in this Process Hacker, so you can actually see what EXE files are being loaded up. So now that keylogger is running, I'm just gonna do a quick test. Type in Notepad, keylogger test. And let's say for a few hours this is running and whatever the user or the victim is doing is being keylogged. The attacker says, okay, well, I wanna see the results of that operation. In their malware, they have built-in keywords. Upload is one of the commands. When I change it to upload and I run this file again, attackers could run this via scheduled task or some other means. Basically now it has uploaded the keystroke logger output.

They may call the scheduled task something like Google Updater, which which is a pretty common scheduled task you'll see on Windows systems.

Absolutely. So this is the attacker's web server and this is the uploads directory. And they would name the output file based on the hostname of the victim box, so in this case RSA desktop. And you could see that I hit Control Control Notepad and Enter, so every keystroke that I'm typing in is being recorded and then sent to the attacker server. So what if the attackers wanted to do some recon on the box? So the command for that, just change it to recon, and it will basically, when it runs, it will keep an eye on the Process Hacker screen too, that it's running tasklist, it's running systeminfo. So this is just to quickly understand what's the profile of that host? And to show you the output of that recon data, again, the file's named after a hostname and then recon. So anybody could tell me what this is an output for? What command could it be?

Whoami.

Whoami, right? So they wanna see which context they're running in. What does this look like? Tasklist output, then you've got system info output, then you've got IP configuration, ipconfig. Then netstat. And then this is very interesting, 'cause if they see an admin account in C:\Users directory, they might be thinking that, hey, there's a hash that they might be able to grab from this box. So for example, if a domain administrator logged into this box, they could leave their password hash on the system, and they could run a tool to grab the password hash. So basically, this is a quick example of recon. If I change the command to say dump, this is to steal credentials. Well, not really credentials, but these dump of LSASS. And for this one, they do require some elevated privileges, so I'm gonna run it as an admin just to kinda show you the whole flow of this. So basically what it does is that it downloads a VBS script. Then it runs the VBS script through, uh, cscript.exe, which is again a binary that is built into Windows, uh, and when that binary is run, LSASS dump is created, but because LSASS dump files, uh, LSASS dump files could be big, uh, there's another utility that's run, 7z.exe, which is basically 7-Zip, and it's gonna compress the file, and then, uh, it's gonna upload the file. So, you know, if, uh, anybody is watching network traffic, they're not seeing seeing big dumps of data that are going out. Attackers always like to compress it, and also they like to password protect their archives so that if somebody found it on the disk, they can't tell what that file is. So that's that. There are other options like cleanup. I can actually show you the LSASS dump. So on the attacker server, basically this is how the file shows up. This is a 12-meg file rather than a really huge file. So there are other commands that we could run. I'll just run nuke if the attackers wanna become like a sleeper cell or they just wanna remove the malware. The cleanup command makes it a sleeper cell, the binaries exist. The nuke command could just totally clean the malware up. So you will see that my malware, and basically the process that's running behind the scenes is that SDelete is downloaded. which is a secure deletion utility. And SDelete basically removes the entire set when they wanna go dark. So that's the end of this demo.

Yeah, and you'll notice there are a lot of popups. We showed the popups just for demonstration purposes so you could see what was going on. But in a real attack situation, you'd have the— the attackers would absolutely suppress the popups.

Okay, so we talked about the RDP tunneling. So basically to show you the illustration there, I'm gonna connect to my attacker box first. So this is my very elite attacker box. So that's where I do all the red teaming efforts from. And basically what I'm gonna do is start my SSH server because the victim box is going to connect on the SSH port for this reverse RDP tunnel to establish. So now that I have connected to the attacker box, I'm gonna connect to my victim box. And when I do, so this is the victim box where a scheduled task has been created for a utility called PLink. So PLink is again a signed utility that comes with the PuTTY suite. And what I'm gonna do is, and if you look at the task, it says that, hey, this task is extremely important for the security of your system. Disabling this task may result in a security risk. So attackers kind of use these tactics to scare off folks. So if I run this and I go back to my attacker server, What all I have to do is connect on 127.0.0.1. And again, it's gonna ask me for the password. And once, when I do, let me take the password. You will see that I am gonna be connected. on that localhost port. And one thing to remember about RDP is that once you're on some system and you wanna just quickly do some file exfiltration, all you gotta do is just copy, do a Ctrl+C and Ctrl+V, and you can actually transfer files as well. So it's not just like a remote screen, it's actually, you know, it can do data exfiltration too. So, Pretty quickly, we'll go back to— by the way, thanks for the prayers to the demo gods. Seem to be in luck. Pretty quickly, we will go to the live slides and just kind of highlight 3 factors. So when we set up security, we usually set it up for— like, we set up the highest security for our gold mine, for our vaults. So one of the questions that you want to ask yourself is, Are you able to track unauthorized access? And if yes, how granular? Can you tell who did it at what time, from where, and how much data was transferred to basically be able to answer the key questions? You want to enable the command line logging for these processes because we see attackers using these a lot. Log monitoring should be a tiered model. Like a lot of people put their fresh graduates or interns to do log monitoring in a SOC. For at least the critical servers, it should be very experienced people with eyes on those servers, like for example, domain controllers and the servers that are providing 2-factor authentication so attackers can't take advantage of that. Incident response. When you get an incident response, really create a plan rather than playing whack-a-mole that, hey, wait, we got this system infected, let's just rebuild that. You want to create a plan, and you basically play chess with them when you have to contain them. You want to stay undetected too for a little bit so that they don't change all their tactics before you're able to scope out all the backdoors that they have and all the access they have in the network. Scoping the efforts is critical. Cloud access, sometimes the attackers come to the on-prem network and then they move to the cloud. to remember what credentials and what access could lead to what. Threat hunting, very, very important concept. If you are a targeted organization, you wanna look at MITRE ATT&CK framework for the hunting adversary tactics. And then do some outlier analysis, do some analysis on legit remote software. If they're really going to, if you use something like a TeamViewer, LogMeIn, you want to make sure that it's not being used in an unauthorized manner. 2-factor authentication. Don't 2-factor everything that's publicly facing because attackers, they see that VPN is likely 2-factor, but some of the portals that are outside or on the cloud may not really be 2-factored. Don't use soft certificates and review your policies for bypass codes. Sometimes when your 2-factor is not working, your network admin may give you permanent code to let you log in. So there should be some really good policies around that, like for example, expiring that after that one use. These are some very tactical advices to— number one is not to leave credentials on a box by network admins for password dump scares. And the last one, we see phishing a lot with macros. So this is the way to basically block macros running in Office documents.

So we apologize, we went a little bit over, and we do realize that you've got another session that's starting. We've got our contact information here in case you guys wanna ask us any questions. We'll be around for the rest of the event. And we wanna give a quick shout out, Matthew McWhirt in the back, he's gonna give you a talk on proactive directory tomorrow at 1:30. Outstanding talk, highly recommend attending that one.

Sure. Well, let's shout out to all the incident responders out there, our Manion team, and also the people who are sharing information in the community, Twitter, blogs. Special shout out to a blog for Hexacorn. We learned a lot from that. And again, thank you for making the 8 o'clock call. Hope you enjoyed it. And we'll be outside, so if you have any questions, we're there for a good hour. So please feel free to ask questions. Thank you.