TrustedSec Sysmon Community Guide with Carlos Perez

Welcome to the Test Guild Security Podcast where we all get together to learn more about security testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Security Podcast. Today we'll be talking with Carlos Perez, all about TrustSec, Sysmon Community Guide and other security topics. As you know, Carlos is an expert in this field, so who knows where we're going to go with this episode, but he is a research team lead at trustsec. He has been active in the information security security field for over 20 years. He's also contributed to multiple open source projects, ones you probably love use all the time like metasploit and Backdrop. So really honored and excited to have him on the show. You don't want to miss this. Check it out.

It's no secret that the number and complexity of applications are growing, but what is alarming is that 90% of security incidents result from exploits against defects in software. The need for better application security has never been higher and Micro Focus Fortify is here to help. Fortify is the recognized market leader in application security and it is the most comprehensive and scalable application security solution that works with your current development tools and processes. With Fortify, you can start security applications in a single day, including custom code, open source or commercial components and scale as your needs grow with an on premise as a service or hybrid implementation. Check them out. Head on over to micro focus.com app security.

Hey Carlos, welcome to the Guild.

Hey, happy to be here.

Awesome. Great to have you on the show. So Kalos, before we get into it, is there any missing your bio that you want the Guild to know more about?

Well, I live in Puerto Rico, which is kind of cool. Not many people here from the island are very active in the security field, at least from the island itself. I've been doing security now for 20 plus years. I actually started in 98 working for the government and I've worked for companies like Digital, Compaq, Hewlett Packard, Microsoft, Tenable Network Security, and now here at TrustaSec.

Awesome. So you've been in the game for a long time. Just curious to know as we enter the New year, as we recall this, it's the beginning of 2021. Are there any trends you see that are different from when you started or that you thought would be different? Or are things still the same? People creating secure enough software?

I would say 2020 was special because for the first time due to Covid, we had to deal with a mobile workforce that all of, well, a brand new mobile Workforce. Many companies out there actually simply had a simple VPN to, so they can have like 3, 4 users when they needed to connect. And all of a sudden now they were thrown into the middle with all of their workforce now having to work remotely. And they went like, how do we do this? And cloud kind of became this humongous thing where companies start moving to Azure ad and instead of receiving a laptop and prepping it and sending it to a user, all of a sudden they were having Best Buy or having B and H deliver the laptop straight to user. And now we had to connect it to the cloud and manage it from the cloud. So for anybody in the industry that was not used to that environment, which is a great deal of people managing clients in the cloud, it was a very daunting task to learn, adapt. So you started looking at organizations and knowing which ones actually had people with a kind of like a growth mindset instead of a static mindset to be able to adapt at this time?

Absolutely. Now with that switchover, were there any new security issues then that came up, like with people working remotely from the cloud that makes software more exploitable or networks more exploitable because people are working remotely because people didn't think about this type of work kind of conditions?

Yeah, especially when it comes to phishing attacks, social engineering types of attack. Many companies actually depend on having the control of controlling the traffic outside of the network. But now the users are at home. So how do you control that traffic? Oh, I have to force them in the vpn. But what if they don't bring up the vpn? How do we look at that traffic? How do we patch these machines? So right now it's been kind of like a bit of a struggle for people to figure stuff out. At the beginning of the year now through mid year, people were getting kind of like on track and now they're addressing a lot of that stuff. Some are still struggling, especially because as new vulnerabilities surface on software, they need to patch them. Now people don't have the filtering, don't have the protection of a corporate network and are more exposed. So companies have had to adopt what we call no trust model, which, which is something that Google does where it forces every asset needs to be fully authenticated, stuff is in the cloud and the client has certain software that connects to other cloud infrastructure to report stuff. Because now those machines cannot connect to the AV server. The AV server's on prem and if they don't VPN you cannot communicate to it, you cannot get new signatures.

Very cool. So I guess I don't know if this makes any sense. Like I said, I'm a newbie here. How about people using maybe not company sanctioned hardware to access software in the network? Can that cause security issues? I know I've been hearing a lot about maybe chips being in other countries being kind of exploited by shadow governments to kind of create havoc as well. Is that overblown? Is that a thing?

Well, it depends on your threat model.

If you're somebody that a government is going to go after, for example, you work with DoD or you work for a company that has ties to dod, that should fit in your threat model. All of those supply chain attacks, like who makes my software, who makes my hardware, who ships it through, which points does it come through? Because we've seen this kind of attack is not new. We've seen it. For example, remember CC cleaner got infected and a bunch of people got owned by it. Another example, when Snowden put out that the NSA intercepted Cisco and Dell shipments to put chips into those and backdoors into those same kind of behavior that many other countries actually do. For example, today is making the rounds a dissident in Russia. All of a sudden he had to take his iPhone in for repair. And they went like, what is this? And when they showed it to him, he had a GSM bug, physical bug, a small board with its own GSM antenna embedded into his iPhone that somebody installed at one point or another.

That's pretty crazy.

Yep.

We think of it as, oh no, that only happens in movies, right? Well, it does happen in real life.

Yeah. What's scary, I guess it was found by mistakes. How many people are intentionally not being found? I guess that brings me to how do you know? Another thing I've seen a lot is people usually don't know they're being hacked or they're being compromised until after the fact. Is that part of your threat model as well? Or being aware of your risk and how to hardened systems?

And the way you do it is.

You monitor your system. You build what is known like a standard behavior of the system. That is one of the things that, for example, you mentioned the Sysmon guide. That is one of the things that we cover for certain event types that you monitor in the computer. Our recommendation is, you know what, install the software, enable it to capture this type of events, look at it after a month, probably after a patch cycle so you can capture weird stuff that software does as it updates and build a known good list of behavior of the machine and then capture. But doesn't fit in it. That would be kind of like a good way of doing it. It's like when you go outside and you look out at your street for your walk, you've never seen a band with tinted glass, tinted windows, and all of a sudden now you're seeing a van with tinted windows every other day. That would make you suspicious. Same thing with your machine. If you see something that is not normal mod part of the normal behavior, you go like, huh, that's suspicious. And that's something that then you investigate.

I like that analogy. So talking about Sysmon before we dive into it, that's one of the main reasons why I wanted to set up this interview as well as I heard a lot about this community guide. I've heard that Sysmon is a great tool, but it's kind of difficult to learn because there was no documentation out there. So I guess first off, for the folks that don't know what is Sysmon?

Sysmon is a software that is written by Microsoft, part of the Sysinternals tool suite. It is one of the side projects of Markus Inovich. He's the CTO of Azure. And the tool itself is a freeware without support. But what it does is that enhances the event log. It allows you to create rules and tell it, hey, any program with a command line that matches this, log it. Any program that access this registry key log that. If I have a program accessing the memory of another program, log that. And it allows you to create a very fine tuned series of behaviors of tactic techniques and procedures and embed them into a config file that you feed to the tool so you're able to capture that. And then as another tool ingest all of those logs, they're able to take action. Go like, hey, Sysmon detected something that it says it's suspicious. Let's take a look or let's alert.

So what kind of tool would ingest this then? I heard a lot of things like elk stack or does it matter? Are you just feeding a database that then feeds a dashboard that alerts people proactively?

Yep, you can actually feed those logs into lstack gray log. Any siem, any, any platform that can ingest logs and you can set rules and alerts on. Sysmon will work because it actually adds or in fact it actually extends the already existing Windows event log. So anything that can consume the Windows event log can leverage Sysmon.

Nice. So I assume everyone that has a Windows environment should be using this. Would this be appropriate for anyone or should every team that's worried about security that has a Windows based environment, should they be using Sysmon?

Yes, I believe they should. But there's a caveat. If you're going to use it, you need to spend the time to tune it, to learn it. It is not one of those tools that you simply double click it next, next, next and then a blur is reboot at the end of the machine and now everything is super fine and it will detect everything. It needs to be tuned constantly because attacks change, behavior change and people need to know about it to be able to know how to configure it, to know what it's a miss when it reports stuff. So if you have a security team or a SOC that actually can expend the time to do threat hunting, it is very useful. If you don't have the resources to do it, it's just adds noise to the environment. For example, I had a hospital that decided to downscale and part of the downscaling sadly was the security team. The security team went down from six people down to two. Now that person for some reason also had to be in charge of the AV of updating it and also part of the change management process. And it had so many tasks. Adding Sysmon to the mix without a system that he can automate or have the knowledge or the training to build upon it would not be useful for them. It is one of those things that you do need to spend time to learn it and to play with it.

All right, so I guess the follow up question that would be then what do you recommend then? If someone's like, oh, this sounds great, but is there a rollout plan you usually recommend? Is this someone with a certain skill set you recommend? If you don't have them, then you probably don't want to start with this right away.

Typically you need to know what you want to log, what is the behavior you want to log, and then you build out. You should also feel comfortable XML because the configuration is an XML file. And typically what you do is you grab a version of Sysmon, you identify what is your install base and the most common systems, you grab a set of test systems and you deploy it to those. And you deploy the configuration, you capture data and across a schedule every three weeks, every month, you go back into the logs, you look what you're capturing and you start tuning that configuration either per section of your environment or user, and then you do a final deployment to everybody else once you feel comfortable with it.

Nice. So you did write a guide, the Community guide, as I mentioned. I think you wrote it because it was hardly any documentation on this. If someone read the Community guide, would that help them implement it? Or is the Community guide just for someone that already has the skills to kind of fine tune it the way you think needs to be done, it.

Will actually help them to implement it.

I even included scripts and commands of multiple tools, what to be careful with. You can shoot yourself very easily in the foot if you're not careful. For example, all of a sudden you set it to capture all drivers are getting logged and all communications out to the Internet. And all of a sudden when you open your browser, it floods with 60, 80 logs all at once, just by double clicking and opening Chrome or opening Firefox or Edge. Because when you open a page, you don't know that it's making so many outbound connections all at once. And you can very quickly overtax the CPU if you're careful. And that's why I wrote the guide, so I can tell you, hey, be careful here. This, this other place, configure it this way. On this one, you want to just grab this configuration from this page and you should be okay. On this other. No, let's take our time. And I give them kind of like an idea of the process so they can follow it, so they can adapt it to their own environment.

Very cool. So since you've written this, I believe you also, if I read it correctly, did you create some extra modules to Sysmon as well? And if so, could you tell us a little bit more about what those modules are and how they help folks that are currently using Sysmon?

Yes, in fact, I actually wrote the VS code extension, Visual Studio code extension that allows you to have snippets to help you write those rules a lot faster. And I also wrote as a personal project with a couple of friends of mine From Microsoft and FireEye and some other companies, I wrote a PowerShell module called PS Gumptu, which is for when you're doing insert response. But after doing several trainings for MAR, for cyber, Marine Corps cyber units that do specifically CPTs, which are known as Cyber Protection Teams, which are the guys that do all of the insert response. I gather all of their feedback on their use of Sysmon. And I created a bunch of functions or commands for the command line in PowerShell that you can use to query, filter and help you even build new configurations for Sysmon.

So Sysmon, I know there's a lot of different Areas in security. Where does this tool fall? Is it part of threat hunting? Is it part of Red team, Blue team? Where would this be used? It would be Blue team, Blue team.

Blue team, threat hunting.

Yep.

I've seen environments where, for example, they'll go in and they'll identify a series of ttps, tactic techniques and procedures from an actor in network and they'll quickly deploy a new configuration of Sysmon with rules for those ttps to find out on which machines they are currently in. So you can use it that way for threat hunting actively, also passively, because you can go into the logs and look for those ttps in the logs and say they must be here, here or there. So it's more of a blue team kind of tool. On the red team side, typically what they do is they work on ways on how to bypass those rules. Now, something I tell in all of my training classes and to customers is that the defenders control two important things for attackers. They control their tool set and their tempo, what tools can they use and how fast they can move in their environment. If all of the sudden you have a hardened, very well configured Sysmon rule set and they see that rule set and they know Sysmon is there, all of a sudden you've taken off the table a bunch of their tools and tactics and techniques because they're going like, I got to move slower now because they can see my actions. So all of a sudden you impacted their toolkit and how fast they can move.

So that's interesting. So I wonder, there must be, like I said, I'm new to security. Is there like a visual that has like this set of tools cover these type of exploits and you could see where maybe you have some weakness based on that type of visual?

Yeah.

In fact, there's a project called the Mitre ATT and CK framework that covers most of the techniques that you're seeing out there and you can actually grab a bunch of those techniques and you can start building your own configurations on top of it. There's also a project by a good friend of mine, his name is Olaf. He's going to kill me because I'm forgetting his last name. It's Olaf Hartung. He is in Austria. He has a Sysmon configuration project where he keeps a bunch of small snippets of configuration from Sysmon that you can copy paste and he has reference to the MITRE ATT&CK framework on many of them.

Oh, that's cool. So I guess. So that's. It sounds like you could share snippets almost like custom functions are between each other. So are there any functionalities then in Sysmon that you think folks don't know enough about or maybe don't use enough that you think would really benefit them?

For example, there's a new one which is for process tempering that will actually tell you if somebody's using process soloing. So one of the Red Team tools that sadly, many APT groups now have adopted is Cabal Strike. Cabal Strike. When you try to execute NET binaries or you try to run certain commands or actions in it, it will start a new project and do process hollowing on it. This new event type that was added on Monday will actually detect that behavior. So that is very useful. It is out there. I encourage people to start testing it and start deploying that one. One of the things that I like is that it makes the life of the threat hunter a lot easier, simpler and easier. Because typically before Sysmon, when you were doing hunting, you were going by process id. The thing is that many people don't know that every time you start a process, an ID gets added to it and those IDs get reused. So when I'm querying the logs and say, oh, I want everything that was created by this process, the ID is 1886. And I run a query for everything 1886. I'm also catching a bunch of other processes that are reusing that id. Sysmon introduced process guids and machine guids that actually are unique to that process across your entire network, which actually makes threat hunting a lot easier, especially when you're trying to do that initial scan across your network to find where they can be in actions.

Cool. So earlier you gave the great example of you're trying to look at your traffic or anything out of the ordinary, and you did an example of like a yellow van. If you saw a yellow van in your neighborhood with tinted windows you never saw before, how do you create that, though, in Sysmon, like, it seems like it's collecting all this data you put into dashboards. Like, how do you filter to find those type of anomalies that are real things that you need to investigate more?

Okay, so with Sysmon, one of the things you can do is you can create a rule that catches everything. Once you create that rule that caches everything and you run it for a period of time, then you can query your logs and build a known good.

And that known good you add into.

Another rule group that you call exclude. Like, hey, these are my Typical processes with their command line exclude this from being logged. Now everything that will be logged now is what doesn't fit that. No, normal, for example. And that's where it's going to pop up and you can look at it and see is it a change in the tool and the way it behaves, or is it something nefarious or something that actually triggers alerts in your head like, huh, shouldn't be doing that. That's that command line looks kind of funny.

You also mentioned in the beginning that you know, it's not just, you know, plug and play. You let go and it does its thing that people need to be able to adjust over time based on different attacks and I guess things that are happening in the field. How do you know when to change what you're doing already? I don't know if that makes sense. But do you have like insecurity? Is there like a monthly sprint or some sort of scrum meeting? Everyone gets together and says, okay, here's what we have and what do we need to improve it? Or is anything change that we need now to introduce to or change within our current type of strategy, typically initially.

Is kind of like meeting every other day to a week. And once it is stable, it is more something that you can actually do biweekly or monthly. And typically you are going to involve two different teams. One of them is an operations team that can tell you, hey, Sysmon is impacting performance over here, or we're collecting too much garbage, or we need to deploy a new version. When should we deploy it, how should we deploy it? And that's mostly on the operational side or your DevOps team. And then you have your SecOps or security operations team that will go like, hey, we've been gathering threat intelligence actions that actors have been doing either via Twitter, via recorded future Threat intel feed or any other free open source Threat intel or even Twitter is one of the best so far for me. I have kind of like a list of people that are constantly hunting and putting stuff out and I'll just simply go and look at it and go like, huh, I never thought of that technique. Oh, I didn't knew the Russians were doing this. I didn't knew that this crew from the US is doing this, or this crew from Ukraine is doing this other thing. I could track that. Let me write a quick rule. And they'll gather all this threat intel and then decide what applies to us, what doesn't apply to us, and let's update the rules.

Okay, nice. So yeah, I always Think of Twitter as just wasting time. But I guess there are people actually posting things are really are helpful like that. That's a good kind of hack, I.

Guess for sure you need to curate it.

Yeah.

So I guess this is probably off the deep end here. Talking about curating and knowing what's real. Especially nowadays how do you know if something's really been hacked? Especially when it comes to elections and people's confidence in elections. One side always says the election's been compromised by Russian hackers. And then four years later I don't think there's been any new technology or effort to try to avert this. And people say, oh, this has been compromised. How do you know? I guess what's real when it comes to security. I don't know if that makes sense.

Yeah, at the end it's show me the proof. Yeah, like I need that proof. Like for example, any system can be hacked from your iPhone, your Android phone, to our laptop, to our computer, Anything can be hacked. Now the thing is you need that proof that says yes, I was hacked. Especially when it comes to something like elections. Because one thing that people forget is that when you go in and you challenge it and you say I won all of the voting records so I can check them by hand is that votes are supposed to be confidential. People are not supposed to know who voted for who, only to tally the amount of votes. So too much info opens the door for in any government change that it becomes authoritarian. To go like this are the guys that voted against me or these are the guys that have this view, I'm going to go after them. So that's why the voting system has been set up where the only confidentiality interior that is important is that X amount of people voted and they voted. And this is the tallies of it. But there's no one to one like Carlos voted for this candidate, Joe voted for this other one. And that's why they're built the way that they're built. Now when people say it's been compromised and so far everything I've seen is like we think it is this way because it doesn't behave the way I think it should have been.

I go like, well that's an argument.

That if I try it with my wife, I'm going to get in trouble. If I go like hey honey, the fruit was supposed to be great and it, it, it ain't. Or this food is supposed to look to taste bad because it looks bad or stuff like that. It is not what we think could be. It is what can actually be done or has it been done? For example, Krebs, the guy that ran cisa, he actually did a bunch of good stuff that us in the, in the information security arena were very happy with. He started, he worked with DoD, so as soon as the Koreans, the Chinese or Russians were taking action, he was very quickly putting a lot of those malware samples out there and helping coordinate that stuff. Because now you burn that tool, all of a sudden you may be able to identify every other place that they have been going into. So that was unique to him. Started working there as soon as he started the campaign. He had a group of threat intel analysts that were going and analyzing all of the data and everything that was set. And he created a page and started saying, like, this is fake, this is real, this is fake, this is real. And started squashing that stuff so it wouldn't grow. Because a lot of this psyops campaigns, psychological operation campaigns, actually depend on putting out their uncertainty and doubt. So as soon as you address that stuff and this is how we know that it is that way, you kill that. Now that sadly, some politicians on both sides go like, oh, no, but. But my feelings say that it should be that way. Well, here I was actually asked, talking about actual factual proof. They monitor those systems. They check the versions of the systems. They, as soon as something came up and sounded suspicious, they had people go in and check stuff. So he knew that everything was secure and everything was not tampered with because they did the due diligence to look into that stuff.

Absolutely. And I know how my mind works. But as you were telling this story, reminded me of another story that's completely different that you were talking about on one of your YouTube videos about you were doing some sort of work for someone and they claimed you brought their system down because you were doing work on it and talking about proof. This is what came. Why I thought of this is you said, well, you track everything when you're doing type of engagement, so that you have hash tables and hash, you have timestamps for everything to say just to show proof that, look, you're saying this and it's false. Because here's proof that I did this, this and this in this certain time span. Could you just talk a little bit more about the importance of that?

I guess, yeah. If we look at it from the pen testing side or red team inside, every time I do an engagement, my different platforms, what they actually do is that they track every command that I issue and it has a timestamp and what is the output of each command. I remember on one of my first pen tests back in 98, 99, I remember I had this lady at a government agency say that I burned down her CRT monitor. It was that month ago. And it was like, no, I did not. For starters, there's no way of me issuing a command that will burn your CRT and bring it down. It's just not possible, period. And I had a log of all of my actions. I had configured in my CMD EXE a prompt that included the time of each one of my commands as I was issuing them back there in Windows 98. And it was keeping all of that data and going like, this is the command I issued. Look here, this hour, at this hour, when you're saying that you had this problem, I was not in your system. I was in this other system. So there's no way for me to have done that. In fact, I had no interaction with your system. Same thing when you're doing forensics because you have to take it into port. You need to know specifically each and every command that you took inside of the machine. Because in forensics, there's this principle called the Locard Exchange principle. Damon Locart is the father of modern forensics. He was a French cop. And the principle actually boils down to any interaction between two objects or two subjects is going to leave a trace. So with the absence of trace, there's an absence of proof that there was an action.

Perfect. Okay, Carlos, before we go, is there one piece of actionable advice you can give to someone to help them with their security? Security testing efforts. And what's the best way to find or contact you?

Yeah, and Twitter.

I'm Carlos Underscore Perez. Or you can contact also the company at trusted sec www.trustedsec.com. i'm the lead for research there, so I get involved in engagements. Kind of like a support to engagements when the red team needs a new implant or a new tool for an action, or the incident response team finds a new TTP that they need us to reverse engineer and help them work out stuff there.

Great. And Kyle, do you have any one piece of actionable advice you can give to someone that they could take away and implement right now to help them with their security efforts? Security testing efforts.

If it is somebody in an enterprise, I would say just like it will sound corny, but know yourself, know what are your systems. A lot of people, believe it or not, when do not know how many machines run in their environment, how many users do they have and even if you don't have that basic piece of information but clear inventory of what you have, you can now go to the next step, which is to know how do they interact with each other, because then you can group them and be able to take action. So knowing yourself by having a clear inventory of your environment, it's critical. And that's one place that I see so many fail.

Thanks again for your Security Testing awesomeness. The links to everything of value we covered in this episode. Head on over to test guild.coms4.5 and while you're there, make sure to click on the Try Today link under the Exclusive Sponsor section to learn all about Micro Focus Fortify, which is the recognized market leader in application security. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Security Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack security Testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Security Podcast. Head on over to testguild. Com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.