AWS Penetration Testing with Jon Helmus

Welcome to the Test Guild Security Podcast where we all get together to learn more about security testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Security Podcast. Today we'll be talking with John all about his new book AWS Penetration Testing. John is a pen test leader, security engineer, educator, author and cloud hacker who has been working in engineering security information technology for 10 years. He's he specializes in penetration testing, threat and adversarial assessment, vulnerability management, cloud technology and he also has experience as a technical educator and university level professor. So as you can tell, he really knows his stuff. He's also known as the Granola Hacker due to his consistent attitude for giving back to those trying to get into pen testing. So he's the perfect guest for this episode and for the show in general. And so I'm really excited to have him on the show. You don't want to miss this. Check it out.

It's no secret that the number and complexity of applications are growing, but what is alarming is that 90% of security incidents result from exploits against defects in software. The need for better application security has never been higher and Micro Focus Fortify is here to help. Fortify is the recognized market leader in application security and it is the most comprehensive and scalable application security solution that works with your current development tools and processes. With Fortify, you could start security applications in a single day, including custom code, open source or commercial components and scale as your needs grow with an on premise as a service or hybrid implementation. Check them out. Head on over to micro focus.com app security.

Hey John, welcome to the Guild.

Hey Joe, thanks for having me on the show. Glad to be here.

Really excited to have you. Jeff. Before we get into it, I'm always curious to know first off is anything I missed in your bio that you want the Guild to know more about?

You know, you pretty much hit it on the head. That's my the long winded version of my concise bio is that I have a very vast background and different experiences which I've given lots of talks just based on like how to get into pen testing based off of my vast experience and how individuals can with that as with their vast experience. I really like the Granola Hacker bit where you mentioned, you know, my way of giving back is kind of being this, this hippie pen tester where I'm more about like keeping the peace and helping out those who want to get into pit testing rather than, you know, being a gatekeeper and making it you know, more difficult than it needs to be to get into pen testing.

Did you find that that is a barrier for some people that there are gatekeepers in this particular niche?

Oh, yeah, absolutely. Like if you go on to various different social media platforms, you'll see a lot of passive aggressive toxicity within the field where there's a lot of individuals poking at each other based on what they think is a valuable metric to I guess, qualify someone has a pen tester or not. And you see that all over the Internet.

Gotcha. So I'm always curious to know when people write books. I mean, I tried writing a book five years ago. It took me a year of my life and I published it. Why write a book? And your book is huge. I think it's over 300 pages.

Yeah, yeah, that's a good question because like you said, it's not. It took you a year. I think for me it took about a year from signing to actually writing, to getting the book, you know, on the, on the digital shelves is why write a book? This was something where I was actually hit up by the publisher to write a book based on pen testing and they wanted to get more focused in cloud. And I had done a lot of AWS pen testing at my current employer that I was with and my previous employer. But the thing I noticed was there was not a lot of material on, on pen testing the cloud and more directly with pen testing aws. And as we've seen, AWS is something that is really taking the market by storm. They're the market leader for cloud and you know, also like Biasly, I live in Seattle, so that's where AWS is. And I just saw it has this area of like, hey, this is a really good way to get my niche in there has a cloud pen tester but to also more importantly give back to the community and be one of those, the forefront runners or the founders of setting up the beginning of formal writing for cloud pen testing rather that be in a technical sense or in a theoretical like methodology kind of sense, which AWS pen testing really does a good. I think it rides the knife edge pretty well of doing both where it teaches you things at a theoretical level and a conceptual level, but also at, at a practical level as well. And yeah, I saw writing a book has a way to really help encompass all that and give back and then start the conversation around, hey, we need to actually start formulating the way how we do cloud pen testing.

Absolutely. So yeah, it seems like an area that there's not a lot of information in so you know we mentioned cloud security a lot but this is aws. If someone bought this book and Reddit would majority of it still apply to any base, any cloud based type of environment like Azure. Is it specifically AWS all the way through?

That's a really good question. So the beginning of the book talks more about aws. If I remember correctly. Looking at the book, the the first few chapters I believe they go over like this is AWS and pen testing aws. The second chapter goes more in towards like how to set up AWS and then it goes and dives into pin pen testing certain services such as Lambda S3 APIs and things like that. So that's all geared towards AWS. However, the last half of the book, the third portion where it's the good do's and don'ts of cloud pen testing, that is a portion of the book and that's a good hundred pages of the book that allow you to take those concepts and actually formulate those into new ideas that can be cloud agnostic.

Cool. For folks that are listening, I believe there are 12 chapters and there are three main sections like you mentioned. I think the first one is as you said again, building your AWS environment. Section two is pen testing the cloud exploiting AWS and section three is lessons learned. So I thought we'd dive into each section and try to pull some information out to really tease it out to get people to really buy the book. So first one is, the first chapter is just diving right in building your AWS environment. And I'm just curious to know from a security perspective, as people are building their environments, are there anything they could do from the beginning to make sure they're building a more secure application, rather secure infrastructure rather than wait until the end to do a pen test against it?

That's a great question. And in the book, this book doesn't actually go into it. There's ideas for future literature right. Based around that idea, but in the sense of like securing aw, like let's go with AWS in general because awesome AWS is one of those things where like comparing it to something that's also big in the market, like Azure. Azure tries really well to mimic on premise kind of assets as opposed to AWS learning. AWS is an entirely new skillset altogether, right? Cause it's an entirely new application. And so with in regards to, you know, ways to build better, let's go with AWS applications. I'm a big advocate for saying bring security in or early on rather than later on. And that doesn't mean just having someone, you know, like a security engineer there to do the tech review. I'm a firm believer that if you have an internal pen test team, a lot of bigger corporations now they, they do deploy internal teams, have someone there at the beginning of the conversation to start talking about, you know, security ideas and concepts based around whatever it is that you want to deploy. Because what that's going to do is, is that pen tester or you know, offensive security engineer, whatever title you want to give them, is going to bring in that adversarial mindset approach. And they, they think about things in that light as opposed to a security engineer. Like a typical one might just look at it as a hardening security control kind of basis. The pen tester is actually going to say like, hey, like if you don't incorporate, you know, I don't know, like detection or like detection on a certain part of the service or monitoring or if you don't have deletion protection, these are the impacts that can happen through misactions, misuse or even compromise. And so having a pen tester there is going to build that security awareness where it's going to have a trickle effect where, you know, a year down the road everyone at the table, at the pre deployment table, so to speak, is going to understand like, hey, these are things that we need to look out for and these are things that we need to share with other departments within the organization to ensure that they're following similar protocol to make sure, essentially ensure that we're deploying assets that have security built in rather than security being an aftermath.

Absolutely. And I guess because it's in the cloud also it seems like if you work for a large organization, you have multiple groups setting up environments unbeknownst to you. Could that expose bigger companies to issues if they have different, they don't have a full grasp on the magnitude of how many people are using AWS with services like that.

Oh yeah, absolutely. You know, when the cloud scales so fast that the other thing is, you know, I guess this is kind of more of like a, not a correction to the question but like more of a potential solution is when you're having multiple heads in on the cloud and it can become cumbersome really, really quick because the cloud is continuously scaling, continuously scaling internally with the environments. And it's one of those things where companies are just like executives and leadership loss often look at it as like a lift and shift, where the engineers are like, oh no, we, we don't even understand how to use AWS yet, Like, you gotta give us some time to, to catch up on that. I think one of the big things is that there needs to be before or during some kind of, you know, cloud development, there really needs to be company wide or at least department wide awareness training on the cloud. And what that means is like, how are we educating ourselves, how are we measuring the effectiveness of the security training that we're giving to the individuals that are gonna be, you know, know, utilizing and deploying the cloud? And then, you know, based on that, those two metrics, how are we going to deploy effectively and efficiently after we've educated and determined if, you know, our users or our environment is cloud ready, how do we deploy it securely and then how do we build off of that? And that was like a four step program or four step process that I had actually introduced at a talk at the Wild West Hacking Fest and they had a cloud pen testing summit that I talked at last week with a group of other cloud pen testers. And yeah, that's something that I think is going to have to be incorporated to ensure that we maintain this or the users maintain proficiency with the scale of cloud computing. Because like you said, it's, it's everywhere. You can bring up an entire infrastructure in less than a day. So we have to make sure that the engineers and the users and everyone who's using it is caught up to speed.

So on my podcast I also hear people talk a lot about excessive privileges. So if someone's building an AWS environment, is that when this comes into play that they need to be cognizant of that or is that a different step of the process?

Oh no. The number one reason for compromise in AWS is because of misconfigured services. And typically it's on the, the user level or the group or role level. And so like, cause if you look at aws, it's a billion dollar company, right? So they're spending that a lot of money into ensuring that their services are hardened and that they're secure by default. And when we're targeting it, we're not targeting the services themselves, we're targeting the things that the human element implements. And the biggest thing is the roles and the user level attributes that are being assigned to the AWS environment. So for example, you know, you might find a role that's overly permissive and you know, Engineer Timmy is actually able to execute role level privileges because for some reason he accidentally got Admin John's privileges as well. And now he's got the full Keys of the kingdom. And believe it or not, it sounds kind of, I guess silly is a good way to put it, but that's a huge, huge issue when deploying aws.

So if I have an AWS instance, it's in the cloud and I work for Company A and I have it so someone can easily get into my particular instance. Is there any way they can hop over into someone else's instance that may be on the same network or.

Oh, yeah, absolutely, absolutely. Sorry, I didn't mean to cut you off, but yeah, no, there's. So in the cloud they use what's called a virtual private cloud. It's the VPC network. And think about that as like a subnet in the cloud. And so if you have two instances, let's go with like two, I don't know, red hat hosts that got something hosted on Amura and, you know, someone gets access to that. Absolutely. If that host can talk to the other host inside that vpc. Again, remember, the VPC is like a subnet. Absolutely, you can definitely talk to that other one. There's potentials, you know, if, if that instance isn't kept up to date, the other one that the user's not on, that instance isn't kept up to date, or if it's, you know, got some kind of vulnerability on it because you still has the administrator, engineer, user, whatever you want to call it, it's your job to maintain the updates and patches for that host operating system. If you let those lapse, then yes, it becomes vulnerable. And if you have one instance become compromised, then the other one can be compromised. And even to further that attack you can, you know, there's such things called VPC hopping where essentially it's just hopping from one subnet to another subnet in the cloud. And that kind of goes in line with, with privileges getting exploited. If something's overly permissive. There's potentials that a compromised host in one subnet or VPC could actually hop into another VPC and then start exploiting things, you know, host services within that other vpc.

It's pretty scary. So I guess this gets us kind of into the second section of your book, which is exploiting AWS. And I think you have a chapter on S3 buckets. Is S3 buckets the main area that people will try to exploit to start with when they're dealing with aws?

Yeah, for the most part you see a lot of stuff with S3. If you go and you do like a Google search or a Google Scholar search, you'll find a lot of formal and peer reviewed articles on exploiting S3 buckets and it's very, it's company agnostic in a sense. And what I mean by that is that you would think, oh, it's probably going to be the smaller companies that are going to be getting their S3 bucket object storage exploited, but it's not. If you go and you look, you'll see companies, you know, small, medium, and then even the large, such as like Facebook, the NSA, DoD, contractors, getting exploited via their S3 bucket storage just because of, you know, weak policies or compromised keys. Which keys are you know, essentially the keys that allow you into the AWS environment? Yeah, so S3 buckets are the big thing and it was a big thing that I wanted to touch on in this book just because it's one of those things that we need to become very, very familiar with because it's such a common service and, and it's one of the most exploited services in aws.

Now is that good? Does that go back to when you talk about something being misconfigured? Is that what makes it so that that makes it easy to hack or sometimes people just, just misconfigure it to make it so it's easy to penetrate or to get into?

Yeah, that's a great question. So with S3 directly, there's two issues to the puzzle there. There's the user level issues where you might have a user that's overly permissive and able to get into a bucket, and then there's the buckets that are put in the public by accident. And what that means is essentially think about it has your Windows file server, right? You have Windows folders on your computer. Imagine being able to store all the data you want and you create that into a shared drive that you share with, you know, other individuals inside of your company. Well, Imag being able to give that folder an API and make it accessible by the public Internet, that is a big common issue that happens with S3 buckets is that they'll be given public access. And I can't tell you how many times I've done consultations with companies where they make them public and they think, oh, well, I thought it was public internally to, you know, on the Internet, not the Internet. And so these buckets essentially which are file storage, are put onto the open Internet. And what that means is that anyone on the Internet can go and query the URL to the API and grab whatever is in the bucket. So it's a user level issue and a service level Issue. That API issue is the service level issue.

So besides S3 buckets, I guess you also mentioned RDS. Is that something similar with the. So people can get through an API or to exploit it?

Yeah. So RDS is. Those are the relational database systems or servers essentially. Think about that as like a database. It's essentially a database within the cloud and you can give that a public API as well. And then same issue, if an RDS database is public, someone can potentially get into it. The other issue too is that if there's. Think about it with anything like if you have a vulnerable front end web application interacting with your RDS database, then you incorporate a whole new level of issues too. Just like you would with an on prem backend database server. It's the same thing. You can start exploiting that database via the vulnerable web interface.

How common is between S3 or the RDS that people attack? Is both of them just as common as one more familiar attackers than others?

Oh no, it's S3 all the way. If you Google AWS attacks or AWS compromise compromises or things like that, I would guarantee you I would put 10 free books on it that nine times out of 10, if not more, it's going to come up with some kind of S3 bucket breach.

Now how about Lambda? I've heard a lot about Lambda. Once again, I'll state my ignorance here. I just thought it had to do specifically with code. I didn't know you could design it in such a way that it could also open you up to be exploited as well. Can you talk a little bit about Lambda services and how maybe they could make you vulnerable to pen test?

Yeah, absolutely. So in my book we do some cool things where we actually exploit the service itself. There's another book out there that it's called, I think it's called Hands On Penetration Testing with KALI Linux or Hands on AWS Penetration Testing Testing with KALI Linux. And it's very Lambda heavy. So this, for this book, I didn't want to focus too much on that because the other book was such a great resource and there's a lot of good resources on the Internet for Lambda because like you said, it's essentially think of it as like a. An IDE in the cloud that can execute your code and you can execute it to do various things, interact with services, interact with APIs and do a lot of different things. It's also, you know, it's not directly related to one language.

Right.

You can use a bunch of different programming languages inside of it in the Book I focus on Python because I'm proficient in Python. So we use Python for that. So Lambda Services, essentially, like I said, it is a IDE within the cloud that allows you to execute code on a triggering basis. And what that means is it only executes when a certain triggering function is executed and then the lambda function responds. However, there's lots of issues with lambda where there's a lot of roles that can be overly permissive. One of the big issues is that integrating lambda can be kind of difficult and cumbersome. So one of the things that are often incorporated into it is that they add what's called a star attribute, where that essentially allows anyone and anything to execute the function. And for a developer, it's a lot easier to do that, then have to assign roles and make sure that whatever that lambda function is operating or working with is also on that same role. It's just much, much easier for them to say, okay, well, let's let the lambda function interact with what, whatever we need to, and we'll build security afterwards. Kind of like what we were talking about at the beginning of all this. And so what that does is that allows anyone to exploit the lambda service. And you could do anything with the lambda service from writing your own code into it to, you know, using it to escalate privileges to another user. And in my book, what we actually do is we use vulnerable lambda service to execute a reverse shell into the lambda function. And what that means is we essentially set up a persistent connection to the host operating system of the lambda function. And what you can do with that is you can actually use that has an undetected pivot channel in a pen test or a Red Team exercise that allows you to pivot off of, you know, say you have, you've exploited EC2 servers, you could use that lambda function has a pivot to go to, you know, somewhere else in the cloud and be undetected. And so we touch on that in the. In the book.

Awesome. So this book is, like I said, there's a lot of information. You cover all these points so far in detail in each chapter. So I just want to start wrapping up. I guess the next question I always ask everyone is, is tooling? Is any specific tooling you use for aws, is it just your standard metasploit that any pen tester would already be familiar with?

Yeah. So in this book, we focus on metasploit a lot because I wanted to make this book beginner friendly because for that simple fact of there's not a lot of information on here. So I'm a big believer in you got to start walking before you can run and people running all over the place with material on how to do this. But it's not concise and formatted. So I was like, okay, this is the perfect time to start formatting it at a entry to intermediate level and start building from there. And so, you know, metasploit is a great tool because it has some aw, some proprietary AWS tooling or tools in it. However, you know, in the book we use BURP suite, we use NMAP a lot. We use some open source tools that we find on the Internet. There's a lot of S3 tooling out there that you can find on GitHub. I don't have any one, any like specific tools that I can recommend, but you can definitely do a GitHub search for S3 bucket tools. There's PACU, which is a tool developed by some individuals up here in Seattle. You can download that. That's a good tool to use. We didn't talk anything about PACU in the book just for the simple fact of I didn't want to be redundant of the other book, the one other AWS pen testing book. And that book is very heavy in pacu and so that's a good tool. And then there's the. For like especially for S3, if you want to start doing your own tooling, you can use the Boto3 library that's built for Python and just start, you know, taking scripts down from the AWS documentation that you can find publicly online through. Through AWS and start actually creating your own S3 bucket scripts. So yeah, it Metasploit's a good one pack. Use a good one and then you know, just using some of the more traditional ones but you know, starting to get it and start, start learning to do scripting yourself and start scripting using various libraries that are more directly related to aws.

Nice. I think the three areas we touched upon basically specific to aws. So any best practice in general for cloud based penetration testing to maybe have people hardening their systems against attacks?

Yeah, so in the book we mentioned a term that I coined over the course of the past year, year and a half called functional testing. And I gave a talk again about this last week and it touches on it a little bit in the book and we're actually going to be expanding on the idea in future books. It's called functional testing. And what that is is where we as pen testers need to actually stop thinking about the pen testing the cloud in a traditional way, like you were mentioning, with tools. What pen testing tools can we use? A lot of the tools that we use are actually ineffective for assessing the cloud. That's why I mentioned, like, you can use Metasploit, you know, you can use NMAP and things like that, but really get into scripting because that's where you're going to start formulating your own tools that directly relate to the cloud. But the other thing is too, is that has a pen tester assessing the cloud. We actually need to start thinking outside of the box. And what that means is that we actually need to start looking at this as like a quality assurance engineer and we need to actually ensure that we're getting access to the consoles of the, of the cloud platforms that we're accessing. So in this instance for like aws, we need to actually ensure that we're getting access to the consoles of the organizations that we're assessing and actually start going through and start looking at, you know, the custom roles that are made, the custom groups that are made, the custom policies, see what they're tied to, see if there's any issues there. And then we also need to start looking at, not the individual services at a very, very granular level, but at least look at the services and how they have them configured through the console. And we'll discover a lot of issues there where, you know, like I said, you might see that monitoring isn't enabled on a, on a service, you might see that deletion protection is not enabled on a service. You might see that, I don't know, there's a sudden uptick in EC2 servers from a log and, you know, that might be cause of concern because you're saying, okay, why are you scaling this fast? Like, who, who's authorizing you to scale this fast? Who knows that you're scaling this fast? And what are you doing to secure yourself if you're scaling this fast? So we really have to take our offensive security mindset as pen testers, but really start to look at it at a more holistic cloud security engineering perspective and really meet those two at full peak to properly, effectively and efficiently assess cloud environments for our clients.

So would this apply to a company that's developing software? They're trying to shift activities left in the software development lifecycle. So should a QA person or a tester be somewhat aware of these to begin with? If they're in a Scrum team to have a checklist maybe of, okay, we're running in the cloud. So at a high level, before we actually have to get to a security team, let me make sure I have 1 through 10 at least checked off on my, my checklist here.

In theory and in a, in a pipe dream. Absolutely. Does it work like that? No, because that, I mean that takes up time, resources, and then also the thing is too, is that we want to kind of like always define a title to it and figure out like, okay, if we're going to have this person do this, we need to just make that a title and have them be that one person. And then, you know, that's a. But then you're like, how do you define that job tit. How do you define what that person needs to do? And how do you like, you know, define like, what do we pay this person and how much is too much for them to do? Do we need more than just one? Because that in and of itself is its own skill set and. But yeah, I mean, in a perfect world. Absolutely. Absolutely.

Okay, John, we just skimmed the surface of the book, so I'm really excited to get people to learn more, get their hands on the book. Before we go though, is there one piece of actual advice you can give to someone to help them with their security AWS testing efforts and what's the best way to find contact you or get our hands on your book? The AWS penetration testing book.

Yeah. So if you're looking to get into cloud pen testing or you're actively pen testing for a company and you're assessing organizations, cloud environments, the best thing that you can do for yourself, for your company and for those that you're working with is to ensure that you think outside the box and you look at especially AWS from a different perspective. You cannot just run your traditional five step, six step process, whatever you want to call it, on, on the cloud, you actually really need to start thinking about it in a holistic cloud security engineering kind of way with a pen testing mindset. So as we've talked out throughout the show, that means that you're going to really need to start looking at it in a different light.

So.

And if you want to find out more about how to do that, you can pick up my book, AWS Penetration Testing. It's available currently on pacpub.com and Amazon.com. i think it's about $45 for the paperback copy right now, which I highly recommend that you get the paperback copy instead of the E reader version just because you can have something tangible on you that you can make notes, things like that. And then if anybody listening would like to reach out to me, I'm on LinkedIn at my name Jon Helmus. That's J o n H e l m u s or I'm also on Twitter oosy underscore Moose that's M o o s one emoose awesome.

Thanks again John for your security testing awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com S42 and while you're there, make sure to click on the Try Today link under the Exclusive Sponsor section to learn all about Micro Focus Fortify, which is the recognized market leader in application security. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Security Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack security testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Security Podcast. Head on over to test guild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.