Professional Red Teaming with Joe Vest

Welcome to the Test Guild Security Podcast where we all get together to learn more about security testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Security Podcast. Today we'll be talking with Joe Vest all about his book Red Team Development and Operations, a practical guide driven by his curiosity, perseverance and passion for technology. Joe Vest's mantra for his work in teaching is the journey of gaining experience can be as valuable as the end result. Joe has over 17 years of experience in red teaming, penetration testing and application security. His experience ranges from being a technical lead for a DOD Red Team owning and operating a security consulting company, to a former director of Spectre Ops. This diverse experience has given him extensive knowledge of cyber threats tools, tactics, including threat emulations and threat detection. Joe is also the author of Red Team Development and Operations, which is his experience of information technology in cybersecurity field and as one reviewer on Amazon called it, the go to book for those who want to understand red teaming and build their own Red Team operations. You don't want to miss this episode. Check it out. It's no secret that the number and complexity of applications are growing, but what is alarming is that 90% of security incidents result from exploits and against defects in software. The need for better application security has never been higher and Micro Focus Fortify is here to help. Fortify is the recognized market leader in application security and it is the most comprehensive and scalable application security solution that works with your current development tools and processes. With Fortify, you could start security applications in a single day, including custom code, open source or commercial components, components and scale as your needs grow with an on premise as a service or hybrid implementation. Check them out. Head on over to micro focus.com app security.

Hey Joe, welcome to the Guild.

Hey Joe, thanks for having me.

Great to have you on the show. One thing I always ask authors before we get into it is why they felt the need to write a book. So why they need to write this book.

Write a book. So first of all, you said you called me an author, so I guess I am. Because I published something, I still don't consider myself an author. Which might lead to answer your question, when I was looking at this material, I've had this material in my head in different versions over many, many years and the latest version that we wanted to put out was this book. It was a way to get this information out to the masses. I actually expected maybe 100, 200 people tops to want to get a copy, but it's something that I've been teaching, I've been preaching about this topic. All my consulting revolved around this quite a bit. Helping people build up their red teams, create some sort of adversarial emulation or red team component to their operations. So it fit naturally. It was just something I was already writing and I just wanted to formalize it into a book. I created this with James Tuberville. So both of us wrote this, we've been working together, ran a business together for a long time. So oftentimes he's kind of the guy behind the scenes. He just does a lot of crazy smart things, but just stays in the background a little bit more. So yeah, he's definitely a huge part of this. He just is very quiet and just makes magic happen in the background.

As I mentioned, I'm kind of new to security and I'm sure a lot of people listening to this show are as well. I guess we should start off. What is the term red team? What does that mean?

So that's a really good question. If you go look, you can actually find all kinds of Twitter wars on defining this thing. The definition of this is one of my core precepts of teaching and trying to educate people on this topic. Not that what I'm going to say the term red team is correct, but it's really important and critical to define all of these terms that are used because terms typically imply some sort of prior knowledge or something that often taints what you're trying to achieve in goals. So when you're looking at red teaming, that's important because I actually I use a goal, I mean I. I use a definition that I did for myself to create. So I say red teaming is the process of using tactics, techniques and procedures known as ttps to emulate real world threats with the goals of training and measuring the effectiveness of people, processes and technology used to defend an environment. Big long definition, but what it boils down to is a goal driven exercise that emulates real world bad guy techniques with the goal of training and measuring security operations as a whole. So that's really what you're trying to do. You're trying to understand security operations. If you look into other security testing, oftentimes these are looking for bugs and flaws and those are great and they're really very important to find those, but they don't focus on security operations as a whole. So red teaming allows you to take and apply threat based tactics and understand what you've done with spending time on money, people, tools, all these processes you've put together to actually build a comprehensive security program. You put that to the test through red teaming.

Nice. So you kind of touched on the differences. But a lot of people may ask, how does red team really differ from other security testing engagements? And sounds like it's more holistic.

It's different. Holistic is a term. Oftentimes people would put categories in the three main security testing types. You have vulnerability assessments, penetration testing, and then red teaming. Sometimes that they're viewed as one is more valuable or more comprehensive than the other. They're just different, they have different goals and different focuses. When you look at vulnerability assessments, you're looking at measuring some security, whether it's application security, you're looking at code, whether it's network security, where you're looking at configurations, where you're looking for flaws that exist in that, that system with the goal of identifying those flaws. And I always like to put things in terms of risk and what value do you get from doing this? So if you perform vulnerability assessments and you come up with some findings and you find these flaws, when you apply those mitigations, you're effectively reducing attack surface. So I typically put vulnerability assessments in the category of it's an attack surface reduction effort. Because if you apply those mitigations, that's what you're doing, you're reducing the attack surface. That's great to do. But what we haven't done is dealt with a threat yet. Penetration testing takes that to the next level where you're actually going and exploiting, identifying attack paths, finding how the threat can move through a network. Many of the techniques, the attacks that are actually being done during a penetration test are similar to other tests. But the goal is to identify these flaws that allow a threat to move and attack and exploit systems. So again, if you apply the mitigations from a penetration test, you're going to reduce the attack surface. So again, I put this in the category of an effort for attack surface reduction. And that's from a business side. I always want to put things back in terms of business and risk because this is what's going to pay for these engagements and those that actually ultimately benefit from the results. Red teaming takes a scenario to exercise, some sort of threat based scenario. You can get the scenario from maybe a threat report. You read some new threat report. ABC threat actor is doing all these bad things. The question, and I look at it from a leadership side to say, a leadership rightfully would say would, how would we fare against some actor that's doing these things? Red team Engagement, you create a scenario that exercises these things. They can be highly technical in nature. They can be even contrived in some sense where you are going to allow some actions to be notionally occurred just so that you can fulfill the scenario. Once you move through the exercise the scenario, you are observing how the people, processes and technology actually respond. So there's different levels. You can be very open, you can be very close and have everything occur naturally. In other words, have a real threat, attack your network, move through the network, stimulate it. I call these operational impacts cause some sort of impact and force an organization to respond as if it was real. This is where you really put everything to test. And you don't just have notional yes, we've patched all the systems, so we're fine. You actually see and deal with how can you deal with a threat?

Nice. How does this typically work? You said it seems like it covers a lot of things. So you like to bring it back to the business and risk how big of engagement is it and when does this engagement occur? Is it after the software is delivered or like if this software development lifecycle is like at the end of before that you release something into production or is it in production already?

So that's a. There's a couple pieces of this and I'm glad you mentioned it because there's extra things you can do when you're looking at from a software development cycle and you're producing a product or you're building a network, or you're building the infrastructure to support this piece of software. Those are all the technical components that do benefit greatly from vulnerability assessments and penetration tests. Identify flaws, look for good practices, reduce the attack surface, reduce the ability a threat can use that thing against the organization. But oftentimes there's more than just a single application. You have an environment, you have a giant ecosystem. And this is what you're really looking at when you hire. If you're an organization that's hiring security defenders, you have security operations centers, you have threat hunters, you have teams of people designed to go and prevent and detect and respond. That's what we want. So from a time frame my normal go to is you're looking at if I just kind of walk backwards from right to left, I say an engagement is typically going to be about two weeks of execution. You're going to add or subtract some based on the scope. You're going to have at least two people on that engagement and that gets into some two person integrity process that I talk about more in depth. In the book. But you're going to have at least two people and that's two weeks of execution. This does not count planning or the culmination phase. So you can have at least a week or two, maybe more of planning, depending on the nature and the risk level that you have from an organization and honestly your experience at going through these processes. But I've often seen planning take two to three times longer than the actual execution because you spend so much time prepping and making sure everything's going to move the way you want so that you can actually engineer this engagement to trigger those goals that you're after. You really want to measure the ability for prevention, detection and response. And that response side is something that's very unique to red teaming or adversarial emulation or threat emulation, whatever terms you'd like to use. That response side is what we want to know. We want to know how well was an organization fare against a real threat.

So Joe, I did work for a huge company and I've never heard of a red team before. So is this typically a team within an organization? Is it usually better to have like a outsourced company that handles it, that's not biased, that can really test out your systems?

So both is fair and fine. And I look at organizations as they mature. Internal teams can be work really, really well because as red teams and this red teaming process grows, the bond between, I'll call it the red side, the attack side and the blue side, the defense side, they grow closer and closer. And as long as those are not adversarial nature and they work to support each other, you can work, you can start to have internal red teams really, really push the defensive capability to be much, much better. With that said, external.

Engagements can be really, really valuable to give a new insight and a new view. There are a lot of companies out there that claim and do red teaming. Not all of them are really what I would call doing red teaming. There's only a small number who actually put this together. Create real engagements that are geared towards engaging an organization to cause some sort of real world impact, to actually measure that people, processes and technology. So there's benefits to both and a lot of it comes down to your size. If you're a really, really large organization and you have the security budget to have internal staff, then that's going to be what you're going to do. And you may occasionally supplement with external teams just for some of that validation. But many organizations, like, I'll say a normal mid sized organization, will Pull in third parties just because that expertise is unique. And they spend more time and money and their security budget on internal teams for protection, their tools, their security operations center, their threat hunters, et cetera.

Great. So as I mentioned, I never heard red teaming, but we did have a security group and I had no idea what they did. No one knew what they did. So typically, is this like an activity that would happen unbeknownst to everyone in the organization or your developers? Or is there education that goes on when you have an on team, like a red team, part of your division or your team in general?

So first you say security don't know what they do. They're the group that always tells you no.

Yes, exactly Right.

That's the one that says, hey, you didn't do this, you didn't do that, and this is bad. And they tell you the sky's falling and. And they all run around with tinfoil hats on and tell you how bad everything is. Which is a whole different problem the security industry has in needing to bridge that gap. Still, I've been doing security for a long time and that has not changed. I feel like everything old is new again and we're repeating a lot of the same things. With that said, red teaming, if you want to go down this path, and this is how I define things, so that when I start off, start any sort of engagement, I want to have all of the definitions and I want to make sure understanding is shared across the board and then we all come to a common understanding, which means, what are you doing? Why are you doing this? What is the goal of this thing? And my argument for any red teaming is red teaming is often looked as an adversarial type role, defensive security role, But I actually argue that it's part of the defense because ultimately the red side is trying to make the defense better. That's their job, either by training or measuring capabilities and understanding. With that said, no matter if an engagement is announced or unannounced, there should be a really, really good exchange between the offensive things that were done and the defensive actions that may or may not have occurred. And try to understand why did we see this, why did we not see this? We have these controls in place. We thought they were going to work, but they did not. Why did they not work? Was it a people issue, a process issue? Was it technology to really come to the root cause of that? With that said, announced versus unannounced, the decision on announced versus unannounced really comes down to goals. So if you have an unannounced test, then you're really trying to understand how an organization actually respond. They're sometimes risky, riskier, or at least the perception is riskier. And when you don't have direct involvement from the defense, you may not hit all of the actual targets and goals as much. On the flip side, if you announce it, you can actually engage the defenders in this. It becomes more of a collaborative approach. Some people might put this into like a purple team category, but basically you have the red and blue working a little bit closer together. That's great to identify very specific technologies, defenses, processes and such. But your natural reaction does not typically occur. So it comes down to what are your goals? Are you trying to initiate a real response? If that's the case, you want to go unannounced.

Right. And Joe, you did mention it's. You like to define terms. Since I'm new to security as well, and maybe people listening, I like to define some things you've touched on. You've mentioned multiple times, offense and defense. So I guess what is offense and defense? And like, what are some activities of each within red teaming?

So I'm actually really glad you say that because this is where I find our community misses the mark quite a bit. So the offensive side, this is the side that is portraying the bad guy. They are exercising the ttps, the tactics, techniques and procedures of a bad guy. So they're running malicious tools, they are exploiting things, they are taking over systems. So that is what the offensive side is working on. And typically you'll say something in the offensive security world, this is where you're looking, looking at command and control frameworks, malware exploits, attacks, all of these things that you might learn in various security trainings and such. This is where you're playing the bad guy. Stuff that typically is often called the red side, AKA the red teaming side, versus the blue side, which is the blue side is going to be on the defensive side.

So also in your book, you have a methodology. You go over, get in, stay in and act. Could you just talk a little bit about that?

Sure, sure. This is a. A lot of what you're seeing. When I created this book was. And one thing to note, I created this with James Tuberville. So both of us wrote this. We've been working together, ran a business together for a long time. So oftentimes he's kind of the guy behind the scenes. He just does a lot of crazy smart things, but just stays in the background a little bit more. But just wanted to throw that out there. But this methodology, as I'm presenting this information and I go through a process I call educating up where I sit there in the middle. And I really want to educate up the chain of command. In other words, I want to make sure that the leadership understands this. We can come with goals because so often we have a technical barrage against leadership who doesn't understand this. So I want to keep this in a nice simple way. So get in, stay and act came out from doing a lot of briefings and a lot of meetings with leadership to try to keep this simple. And the other side is to try to teach new people how to take all these complicated technical tasks that they're about to undergo and keep it simple. So you're looking about basically, a threat has three things they're going to be doing. They've got to get into your network somehow. They've got, once they get in, they're going to have to stay in. They're going to be living in your network, moving around systems, learning about your network. But they have to exist and stay in there somehow. So they have to have that means. And finally they're not there just for fun. They have a goal. They are stealing information. They are wanting to live on your network and exist for long periods of time for information gathering. Maybe they want to gain access to your network to cause harm, you know, maybe some sort of political reasons, or they want to use malware to hijack in your systems and demand ransom, like ransomware. But either way, these three phases have very distinct components to them. And when you break it into these three, one, it's easy because it's fairly easy to do and plan and build out an engagement. And two, because these are scenario driven, you can pick any part combination of these and create a scenario. In other words, if I say get in, stay and enact, I don't have to breach a network every single time. We can say in our scenario, we assume the threat has gained access to X system and we begin the scenario from that point on. And that allows us to control the scenario, to be efficient with our tools, cost effective with our time and energy, and allow us to focus on those goals that we want.

As you're going through this as well, you did mention old is new. Again, just curious, no exploits. When people exploit a system, is it always the same type of exploits? Or there are certain exploits that used to be more common than they are now, or they used to be uncommon, now they're more common. What are the common exploits? I guess that you want to make sure as a Red Team to be.

Aware of exploitation is interesting. Old memory corruption exploits, they still exist, but they're not nearly as common. A lot of our development frameworks and the way we write code now has naturally reduced that effort, especially since we've had some more higher level languages that were not as close to the memory side. So we're not writing things in C as much anymore. We just have less ability to do traditional memory corruption exploits. I'm not saying they don't exist, but they have gone down. Now we're looking at higher level language stuff, serialization attacks to where you can deserialize.

Like Java or some other languages that allow you to execute that language's code behind the scene. And we really look today a lot of misconfiguration and misunderstanding of technology flaws, especially as we've grown so much in our ability and everything's linked. I'd argue we really don't have an internal and external network anymore. We just have systems that can talk to a lot of stuff. But now you're looking at as we have all this cloud technology which opens up to various APIs to control systems. Those systems have control not of just the application, but maybe the infrastructure on the back end.

We have flaws in implementation and understanding of say like a Windows network and understanding how authentication works, Kerberos attacks and understanding how to steal information for to enable yourself to gain access. With that said, all of these exploits, an exploit is just a means to an end. It is a single step. Our security industry focuses quite a bit on exploits. But I would challenge anyone to say exploit a system and you're done. Once you've exploited that gives you, gives you access to the next step. And now you start to begin your real threat activity because you're going to be living in the network. You know, this is where I get back into the stay in phase. This is where you're going to stay in. You've exploited the system, you've got in. That's a fairly, I'll say quick step in terms of time. It might take some time to build an exploit, but once you get in the next step is you're going to live there for a while. So exploitation phase. There's a lot of space out there and you do really need to understand it.

But it's just sometimes it's over emphasized. And we have plenty of other security tests that can focus on that with the vulnerability assessments and penetration tests which is really, really geared to identify and understand that attack space.

Yeah, I know it's because movies or books Usually focus in on the exploitation piece probably is why it's the most popular. I don't know.

It is. And it's the fun part, it's the cool part to write exploits and it's amazing to do that and it's a lot of fun to go through that and figure out how to do this. But exploits is really, it doesn't have to be normal like memory corruption type exploits. You're looking at flaws and understanding things. When you're looking at all the APIs we have out there now, application security is still huge. And understanding how to manipulate applications and know how to speak the their language, even if it's a simple REST API and knowing how to enumerate and work through that, there could be some flaws in there that are logical, that allow you to control and modify some backend system that gives you access to something else and you just slowly work through that process. Which is why I always make the assumption that and everyone I think should do this is the thread is already in my network, they've already breached. So the breach side needs to be taken care of from our vulnerability assessments and pen tests. Let's reduce our attack surface. We need to really think, okay, the threat's here. Now what? What do we do? Can we actually deal with a threat that has breached our network?

Absolutely. And usually when you hear big news about people being attacked, a lot of times it's like they didn't even know they were on the network or in there for like months after or even years after. So. So I guess it goes towards your view of team maturing. So as a team matures, I guess are there any best practices in running a professional red teaming team in house, like a program that'll help them mature and not just focus on the exploit, but all the other things that encompass a really fuller picture?

Well, I mean, not to say that this is. That was the main purpose of what we wrote in this book. This book existed prior to the book form. We myself and James submitted and wrote this as a sans course. So it was a sans red teaming course at first because. So that we could actually provide this in a training format. So this actually existed as a method to train and teach people how to build professional red teams and understand that we did not focus on the technical stuff as much. The technical is there, but we really wanted to focus and I still think today the operation side is where it's weak. So we really wanted to focus on that. So we wrote this as a sans course, taught that for about two years. We pulled away from Sans and then wrote the book. But in short, that was what the book was for, was to help give people an understanding of Red Team Inc. And provide that information on how to build and run one. So. And I've had so many people come up to me and say, hey, I was told to build a Red Team. What do I do? And that was the intent for all of this information.

Awesome. It's almost like a playbook. Here's like a roadmap or a framework you can use, I guess.

Yeah, that was the intent. And as teams mature, I would even argue that the material that we have is good foundational information. Once you mature, it's fairly easy to mature past what we've written in this book. The foundation information is there. It's important, as long as you can circle back to that and you maintain that. But, you know, if I had someone who wanted to learn how to do red teaming and they could grow past what we've been teaching, then I would consider that a success.

Okay, Joe, before we go, is there one piece of actual advice you can give someone to help them with their security testing, Red Team efforts, and what's the best way to find or contact?

So if you want to improve, learn technology.

Everything we talk about, if you look at what's going on in teaching in schools and such, we're teaching, you know, there's classes on hacking and attacking and all these things. I think there's not really such a thing as hacking. It's just creative IT administration. That's really all you're doing is you. You can't attack something if you don't understand it. So you really have to understand things from the fundamental level of what that technology does before you can attack it. Some of the best, most creative methods I was able to move through a network and attack and generate things was by recruiting the help of experts on various pieces of technology. They were not necessarily hackers, but I could say I could work with them and ask them how they would do certain things and then start to apply threat techniques to it and we could work together. So the big piece of advice is make sure you understand the technology that you're working on. And to get a hold of me. Twitter Joe Vest. I don't have any really cool hacker name, so I just keep it simple. But yeah, Ovest on Twitter is probably the best way to get a hold of me.

Thank you, Joe, for your security testing awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com S16. And while you're there, make sure to click on the Try It Today link under the Exclusive Sponsor section to learn all about Micro Focus Fortify, which is the recognized market leader in application security. So that's it for this episode of the Test Guild Security Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Security Podcast. Head on over to Test Guild for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.