Sprocket’s Pen Test Team on Bypassing Web Application Security

Welcome to the Ahead of the Breach.

Podcast where we are dedicated to empowering.

Security experts and practitioners with the knowledge, insights and actionable strategies to lead the future of cybersecurity. Here's your host, Casey Camilleri, CEO and.

Founder of Sprocket Security.

Now let's dive into today's episode.

Welcome to the head of the Breach podcast where we talk everything offensive security. I'm your host, Case Camilleri and today I have the Sprocket pen test team in the house. So we're just going to have a casual chat here about everything pen test related and we'll just see where it goes. But first have the team introduce themselves.

Will Van Dievender, senior staff security researcher. So I work on service delivery and then also on some of the research side.

JP or Juan Pablo, penetration tester at Sprocket Security. External internal penetration tests.

Nicholson Stacy, the director of technical operations at Sprocket, help these guys work on customers and also do some research around testing at scale.

Yeah, Nate Fair, I'm on the service delivery team doing penetration testing, all things fun and hacking related.

Nice, nice. And recently Will's been doing some research on who is stuff. Tell us about that, Will.

Yeah, absolutely. So I mean to like lay a foundation a little bit when you register a domain, I know these guys already know this. You register a domain, you give ownership information. So first name, last name, address, phone number, email address and you can like choose to redact privacy on it or you can just publish the information. If somebody does a whois lookup on the domain and it has no redaction to it, then they can find out information. So again, the address, email, phone number, email in reverse whois, you do a lookup based on some component of that information. So say take the email address and say what other domains have been registered by this email address. And the goal is to expand the attack service map. So we're finding we're starting with one apex domain and beginning to find other apex domains. And the project was around really two things. One was typically when you do reverse whois, you use a data broker. So you're making API requests to a data broker. So it was all right, let's say we brought in 100 million domains, could we do whois lookups on those at scale? Build our own data platform, recon platform, and what will we sort of find information around that. And then the second part is that around 50,000 to 400,000 domains are registered every 24 hours. So if you're trying to follow domains that have been registered by a Company, you have to make all of these API requests every 24 hours. So it's all right. What if we just cut down on that part and we just got alerted for any new domains registered by a company in the past 24 hours. So it's building out, tooling around that and trying to do discovery.

Nice. And before you'd use like woxy. Right. And you know, to do the reverse. Who is. Why is that important though? Doing the reverse.

Who is. So expands the ATT and CK service map. Find additional apex domains. Like I can think of. We were on a pen test the past six months. We had a single apex domain from a customer use Huoxi found another one. In that other one, it kicks off like a whole tree of reconnaissance. We got subdomain brute forcing service discovery ports, found a server with the JSON file with environmental keys in it and that had creds to the Azure environment. So that became the foothold just through this adjacent apex domain found through reverse whois.

That's awesome. And you mentioned that the data set actually changes every 24 hours. Anyone can download that.

Yeah. So whoisds.com provides a set of domains every 24 hours. So you can set it to download a zip in a text file. And so I'm doing like, I have a tool that will do it for you. So basically you set that in cron job, it'll pull the domains every 24 hours. You can set it to do WHOIS lookups and then alert if there's a component of that in the reverse WHOIS that matches. So if somebody has just registered a domain with an email address, that's important or that you want to watch, you get alerted on that. So that's the idea.

Awesome. That helps trigger the change testing too, right? Cause if you see new assets, you're obviously going to have a new opportunity to make some discoveries.

So it just kicks off that pipeline and then hopefully we find something from there.

That's awesome. You do a lot of bleeding edge work too, right? Following kind of the latest techniques and exploits out there. How do you do that?

I'd say we do it a few different ways. So first would be things like the absolute basics that we need to hit, like sisakev. So, you know, companies that patch for and sorry, sisakev is known exploited vulnerabilities. Those are vulnerabilities that are being activated, actively exploited on the Internet that ciso, which is a government entity, has highlighted as being exploited and often for like malware.

I mean it could be new and old Stuff. Right. It's just following what the real threat actors are doing.

Right, exactly. So what the real threat actors are doing. And I think they have studies that show like people that don't patch that are like way higher elevation levels for malware for complete compromise. So it's like, it's almost like the base level of like you have to do this or you're in serious trouble. So that's the first we're looking out for our customers there with Sisakav. And then it just sort of moves up to new exploit or new vulnerabilities that have come out. And can we remotely exploit these? Also searching through ASM for like fingerprints. Like do we have people that have this in scope? So I guess moving from there.

Nice, nice. Are there other places that you actually get techniques and.

Yeah, I mean Nate, you want to talk about what you set up for us for that? Sure. Yeah.

We've got a lot of automation, you know, like we've been talking about and we've been working on trying to figure out how we get alerted on the SysloCav vulnerabilities. And one of the ways that we've set that up is through Slack Webhooks so you can scrape a whole bunch of CVE and CVSS data. We pull down some metadata around these new recently released vulnerabilities. So is this vulnerability remotely exploitable? What is the, the proximity to the target? Do we need to be on site? Is it authenticated, et cetera, et cetera. And then we get alerted based on a big basically text file keywords on known technologies and known fingerprints that our customers are using. So of course we want to get alerted on a zero day vulnerability and Apache web server or some other more specific customer niche technology that's super important and that's on a cron and Monday morning rolls around, it's okay, what have we got? What are we looking at? Do we need to go out and start scanning for this vulnerability? Is it a 3.1 or is it a 9.9? And then obviously that dictates some of the actions that we do based on what is the most important.

Right. We're filming at Black Hat right now or maybe day one of defcon. But like even here the things that get released we get to actually go play with right away.

Yeah, sure.

I was recently came across finding that you guys have been publishing. But talk to me about 403 bypass.

Yeah, for sure I can. This will find so either right, a lot of IT administrators or people who are kind of standing up Web servers, sensitive panels. So Even like a WordPress login panel or a Luci panel, which we have a blog article about, to restrict access to the Internet. For that they'll go into htaccess rules or their NGINX proxy rules, that sort of thing. And they'll block public access to a specific endpoint, right? So WP admin, login, php, when they write those rules, sometimes they might be a little weak or they might not cover every edge case. So for example, we've seen, right, if we change the URL slightly to do uppercase WP admin instead of just all lowercase, that sometimes might bypass a rule. Appending a dot to the end of a directory, that sort of thing would also bypass a rule in certain cases. So a lot of our tools and a lot of the tools out there just do static checks for, hey, I'm going to issue a request to this endpoint. If I get a 200 and a specific string, we know it's there, right? But that doesn't account for 403 bypasses, other techniques to get to that endpoint otherwise. So vulns that don't look like they're there, if you're not trying a 403 bypass, you're going to miss it because the tooling that we have is just kind of saying it doesn't exist. So trying to do that at scale has been a challenge and something we're working on, but yields a lot more fruit because it's kind of this hidden attack surface that you don't think exists. And then all of a sudden it does when you change things up a little bit.

So have you found that is it like a specific products or technology, like a certain WAF or something that you're bypassing, or do you just not know what you're bypassing?

And it depends. I see a lot of research and stuff around. Nginx seems to be like the most commonly misconfigured thing. It's also very, very common as like a tech stack. So I'm not like 100% sure, but we've seen things in particular, like I said Lucy, that seems to have the problem and cool thing with 403 bypasses too, right? Like they might own multiple instances of it. Once in a while we'll find a single instance. But the second we start doing 403 bypasses, we realize that like 90% of their apps or something have this endpoint exposed. So one critical severity vulnerability all of a sudden turns into like 20. Because you can do it here, here, here, here after the fact.

So yeah, we also have some homework. You know, orange side obviously does some really awesome research into NGINX and Apache on URL parsing vulnerabilities. And that is part of the foundation of those 403 bypasses is, you know, is the proxy server managing how the URL is getting routed? Is the reverse proxy or is it the some other proxy or what is in charge of doing that parsing and how does that manage and handle that on the back end? And then you know, obviously some super awesome black hat and defcon research that again, we need to do some homework and then we take away first thing Monday morning is how do we integrate that into the tooling and get moving.

Even Kettle's article that he had, it wasn't like directly 403 bypasses, but it was bypassing WAFs and front end proxies and that sort of thing. So you know, if you just go surface level with this stuff, like I'm really learning, you're not going to find as much anymore because everybody's implementing security controls and it's a lot more of trying to bypass those than it is like looking for very specific things, those.

That rely on that to be like, I'm protected. Like this whole thing. I've seen you guys generate some findings with the 403 bypass in Lucy specifically. And it's not just like oh, the authentication is there, it's like, no, the password has not been set. Yeah, literally default.

Still, I thought it was funny. Kettle's article, he had like the remediation section at the end and usually it's like, oh, like implement this single rule or something like that. And he's like, assume an attacker can read every line of the source code of the app that you're writing. So yeah, because when you start doing stuff like this all of a sudden it, yeah, just assume that we can see everything, find everything. So it's important to like catch this stuff before it ever makes it out to the Internet.

I think favorite finding in the past couple months go around the room.

Am I going first?

Yeah, I gotta think on it.

Everyone wants to think.

Right?

Right.

Yeah, that's what I tried to kick the JP but that didn't work. Favorite finding last four. I think honestly we were already talking about 403 bypass, like the lack in Lucy. And so to take it back a second, now I'm taking it.

Yeah, yeah, let's cut back.

We can talk about the blog post that you guys both have a blog post that you Authored about that. So let's dive into that. Because you use 403 bypass to actually discover a Lucy instance. But like, what'd you do from there?

Let's take it back to like when you originally found it. And then we kind of like built it out so that it's like OS agnostic. And when we started realizing, oh, there's actually a bunch out there that either don't have the password set or if it's old enough, has the default password set.

So yeah, quick Lucy, like what is it? Nobody's ever heard of it, I feel.

It's not TV show. All right, well, yeah, terrible dad show.

All right.

That'S exactly the first thing, I think.

Yeah, yeah.

So Lucy is. It's a cold fusion alternative. So Cold Fusion is a paid for product by Adobe. Luce is part of a suite of products that's a free and open source version of ColdFusion. And as part of it they have an administrator panel, like many other technology stacks that help you build out components of an application. In the older versions of Luci, you could deploy it to the Internet and it would just have this page that said you can now set an administrator password. So if somebody deployed it and forgot about it, and this is exactly what happened, we found an instance of it, set the administrator password and then instantly we're full admin into this platform and then from there you can create plugins that will run code. So we authored a backdoor, upload the plugin and then got full remote code execution into the environment through this, you know, simple forgotten panel or platform product, Lucy. And then me and JP teamed up, JP wrote and kick it over. You wrote pretty awesome tooling to just automate part of that process.

Yeah, the whole attack path you just described is very like hack the boxy of like, oh, you logged into WordPress and modified a plugin to get access and grab the.

That's basically what it is.

Yeah, that's honestly really what it was. And so when you originally did it, I think you were on like a Windows device. And when we found it next, it was on Linux. So we just found a way to automate the process and make or generate a web shell that essentially is like OS agnostic. You can upload it. It doesn't matter what version of Luci it is, it doesn't matter what OS it is, it'll just work every time. And like you said, the older versions had you set an admin password. I think even older versions just defaulted to that. What's it called, Connections or whatever the Default password is for Lucea. Now the way you do it is to set the password you have to create a file on the host OS and then it'll read it on refresh and then set that as the password. Just like super weird. But yeah, essentially we generated a utility to automatically create the web shell, upload it and then just use it to execute commands on the host machine.

And then I'll kick over you for a second. So we had another customer implement the fix for it was to block at nginx I think.

So I don't remember what exactly the web server was, but sure.

So that was the patch for it was essentially and this is going back to the 403 bypasses. Then you figured out a way a bypass on that.

So then again, yeah, so I mean the fix should be just set the password. You know, don't leave something unregistered on the Internet. But if it is going to be on the Internet at least only let specific IP addresses access that admin panel. So when we kicked it back to them, we said, hey, we can set the admin password on all of these devices. They said, okay, we've blocked access to it. And then with the research that Nicholas did with four or three bypasses, we were like, well what if we can still hit it? So we got backed in compromise. Another one kicked it back to them and they said, okay. So we've implemented filters so now unless you come from a specific IP address, you shouldn't be able to access this panel. And they told us that. So we checked again and found two other ways to bypass it and just went back and forth, back and forth.

But tell them what the initial 403 bypass was like. It's kind of fun.

Oh man.

So we were using the tool that you wrote and then Nate was like, just change the letters to caps. Just make the endpoint capital letter. Yeah, just do camel case. And that worked. And then a lot of, I think you said this too. A lot of like the ones that have been working really well lately have been path manipulation. Not really like the change this to a different request. You know, it's about adding in different like URL.

Nate just had like you dependent period on the end of the row and bypass it.

So I mean I was going to say too like we like we're joking about it but like a lot of scanners, if they're just looking for that path, like they miss it. So like yeah, changing the case or like and it's. Yeah.

And it's really cool that we can actually see this stuff in real life, and it actually works. Right. You know, I know part of your introduction to a blog post that he's working on is everybody has seen these articles. You know, everybody talks about it. And for a while, it's just kind of one of those things that, well, maybe we just don't, for whatever reason, have that visibility into, you know, that doesn't work. There's no way that that works. And then all of a sudden, we do a little bit more research, and it's like, holy cow, this actually works.

That's the best feeling, though. Like, when someone's like, oh, I don't think this is going to work, and someone else is just in the echo chamber of this. Did you try it?

Yeah.

Do you know for sure that doesn't work?

Well, I think the benefit that we have, which we're kind of lucky, is like, we just can test whenever we want, and we end up seeing so many different configurations of different things. So when an exploit comes out, many times people are like, oh, it works this way. And then when we run it at scale and we see all these different configurations, all these different environments, we start to, like, get the understanding of, like, oh, these misconfigurations allow it to work this way. You know, we get, like, that deep knowledge, like, right away test.

Just to add to that, like, we get to know our customers, too. So, like, the Lucy thing was, like, they said we had found it before, but then we knew that, like, it was there at some point in some way, shape, or form. And so, like, instead of just blindly spraying like that initially, instead, you're like, all right, how can I take this risk for a customer that I already found? And then, like, check again, Translate it back into a bypass, that sort of thing. Instead of being, you know, I got to wait till my next pen test were made by Lucy and that sort of thing. So that's fun, too, because you get to kind of bounce back.

Yeah.

You get the rapid knowledge on the flip side, right? Like, how did you fix that? You take that away, and it's like, how they fix is not the same how they fixed it. Right. And that leads to new, like, findings and discoveries.

It's cool because once you've worked with, like, these companies for, like, the last three years, as soon as, like, some of the stuff that Nate built out were, like, there's a new CV for a specific tech stack, your brain kind of like, auto completes. It's like, oh, I know this, this, and this. I know they use this, and you Just immediately start hunting them.

But, yeah, yeah.

So how do you guys find the targets? I mean, everyone knows that we offer asm, but like, we've built ASM for testers first. Right. Like, and the things you guys need to know. So, like, when you're looking for like, Lucy and other technologies like that, you know, you guys rely on asm, but, like, what are some other things that you're feeding ASM or ASM tells you that helps you be successful when testing at scale? I should say.

I think we could talk about sort of to answer your question and to talk back a little bit before about seeing how different tech stacks are implemented across different companies. Like, oh, I know this person does it differently, I think. Was it the PALO instance where we all kind of like, as a team started, like, okay, who has this running? Yeah, we could just use ASM on the portal essentially to be like, okay, who has this endpoint where it's returning back at 200 Pabs? Yeah, yeah. That was a screenshot on the blog about it.

It's super beneficial because, like, sometimes even if we, like, don't have a technology fingerprinted, you can be like, it has this URL and then you just go search for the URL within, like 2 seconds, you have your target list.

Yeah, I think everybody maybe, I don't know, like, has a different workflow for like corralling those assets, working with them, because we have a lot to work with and go over. And so, you know, it's not like you can just kind of scroll through the list anymore. And look, it's. You got to be able to coerce it and that sort of thing. With me, I mean, a bulk of our attack surfaces web applications. So, like, I want to know where a web app is and then kind of go from there. And for me, I don't know about you guys, but, like, I like custom apps. So I do, like, take screenshots quite a bit. Like, and I'll review those because messing with WordPress and that sort of thing, like, it's cool. But favorite findings I've ever done are like weird homegrown apps that had an issue with it or something like that, as opposed to these tried and true. Like, just, oh, it's Fortinet VPN panel. Like, we can spray, we can look for CVEs, but if a dev messed up in a weird, that's when it gets kind of fun. So, yeah.

Nicholas. Favorite finding.

Favorite finding. I was leading into a little bit. Favorite one, like, recently was a random app that customer had like up in a cloud environment. It was like built as like, maybe like a developer portal, like onboarding or something like that. It had GitHub OAuth, but it wasn't locked to a, like, email address. So I just made a random GitHub account and we got a ton of those and did the oauth. It let me in. So that's kind of the first thing, right? Like, I saw the auth and normally you might just be like, okay, I can't do anything with this, but like, trying, like, can I use any GitHub account to do this? And it worked out, logged in, it had some what looked like integrations with Azure and Entra and that sort of thing for pulling data about groups and who had access to which repos couldn't actually hit like sensitive data or anything, but pulled that up in burp. And whenever a query happened to find like, groups or whatever, it would send a JSON payload with like, the name of that group and then backend was doing the query. This. I want you to talk about the UID versus email thing when you swap those out, because it's a similar thing where you're like, what if I just take the payload that it's sending in the JSON out of the blob that's sending. Just yanked out the group name. It just stack traced and threw back the client ID and client secret for Entra. And then wrote some python and was able to like dump the entire global address list. Everybody in Entra, we shared it with the customer and they're like, all right, calm down. So, like, because I wanted to keep going and screwing with it, but it was pretty substantial. It was cool. It's always fun to find. Like, we were on the phone and we were, yeah, hey, like, look at that, man. Like, it was exciting. So that's one of my favorite, like, the recent ones. And then also we got super cool and like, fun feedback from the customer too. Like, they were deep in it, like very technical and everything, and wanted to. They're like, show me the script you wrote and stuff. So, like, that's always fun too. Not just publishing the finding and then they go take care of it. Actually showing it off and talking it through with them and getting the context on why it was there. That made it really fun as well. So, yeah, Nate, what about you, man?

They were finding Nate.

Yeah.

So mine is another web app and this was kind of a twofer, right? Because this was unauthenticated. Sort of those magic email links, right. When you go and create an account. So you sign up and it's just, you provide your email address and it sends you that magic link, right? There's no password that you configure, anything like that that authenticates you into this platform. And then part of the platform is a secondary widget. And this widget provides analytics and certain details and fancy dashboards.

Right.

And that widget gets its data from a backend API service.

Right.

And it makes sense that the widget, in order to feed and populate that data, has to pull that information from an account somewhere. So that's sort of an authenticated type of thing. I thought it was just kind of interesting, you know, I authenticated, but I didn't have a password, I didn't really authenticate. And there's some data that's being passed back and forth between this hovering iframe or whatever and you inspect some traffic and all of a sudden you see JSON web token that's being passed back and forth that I never generated in the first place. And so anyway, you go back and forth a little bit and we figured out that you can basically take this unauthenticated endpoint and generate jots JSON web tokens based on an email address. And then you can say, well, what about admin@whatever.com? all of a sudden you've got a JSON web token returned back to you that you can then use to query the financial analytics and stuff like that. So then you can do a password reset and then you can take over the account all without a password.

So that's so much bigger than what it sounds.

Yes. All just from a magic email link.

Yep. Nice.

That was pretty cool.

Yeah.

So we're on applications and they use JSON web tokens again to authenticate to the app. It's really common to have a REST API endpoint in that situation that's like slash token refresh. So basically that is because JSON web tokens are often scoped to a certain period of time. The application will have you make a request, you'll get a brand new token. In this case, it would provide the email address and the date to that endpoint and there was like some other component that had to be validated. So I was just messing around with the different entries and I was like, all right, well what if I, instead of an email address, I just put in a uuid? Like would it just create a token for any user? And absolutely, that was the case. So you just use, you go to a different part of the application, you would find somebody's uuid, drop that into the email post parameter and then it would just author a token for that user and then you completely took it. It was a pretty critical applications. It was account takeover on anybody with that parameter.

I was like doing proofreading and I just messaged Will. I'm like, why did you even think to do that? Because it was so cool and kind of obscure.

Because it was a JavaScript app or.

Like, lots of times I'll just swap in different parameters. I just want to see the error. Most of it is like, all right, what kind of error will I get back if I switch something in here that it doesn't expect and maybe I can go from there? Yeah, I did not expect it to write a token for me.

It was the spider senses, the little hacker.

Yeah, right.

Something's itching. Yeah, right.

Cool, cool.

All right. Favorite burp. I find myself using NQL a lot for GraphQL testing. Really? Like, I find, like, there's a lot of bugs in GraphQL implementations and it just helps in general putting together requests and testing that out. There's another one, Flowfinder, which is pretty good. I'm trying to go through in my head. Autorize is pretty decent. This sucks.

No, I was going to ask, like, when it comes to GraphQL, why do you think it gets screwed up so much? Is it because it's new and developers don't really understand it, or it's just so complex it's like, easy to introduce vulnerabilities?

Yeah, I think it's complex like you're saying. So that's easy to sort of introduce vulnerabilities. There's no protection for like, silos of data. So like, you have to implement your own, like the way that you silo and go after data. So developer just has to know that. They have to know that, for example, if you switch in a UUID and a request, you're going to get somebody else's data unless there's a backend check for it. I'd also say that it provides a lot of guardrails with the objects. So like in, let's say SQL Injection, right? Like you can put a tick in and maybe get, you know, an error. Or you can. With SQL injection, you're injecting into the query. With GraphQL, it's really. There are guardrails around the input. So it has to be a string, it has to be an integer, it has to be a really specific thing. And so I think developers often think that it like, protects them from it because they see so many errors. But in reality, like, for example, there's an object type of a UUID and GraphQL, and a lot of developers think that's actually a real uuid, but it's not by the scheme, it's actually just a string. So if you put in, like, I've had cases where you put in like, dot, dot, slash, dot, dot, you'll go into a completely other area of the application because they think it's going to be A UUID Because GraphQL is helping them, but it's really just a string like anything else. So I think the depth of the complexity is a big one.

I remember seeing you do that and I was like, I had no idea, like, it was directory traversal in GraphQL, like, sort of thing. And it was really, really cool. So, yeah, that's awesome. It's.

And it's super common to have like a GraphQL hit one REST API endpoint on the back end. So if you can get around that you're going into, like, the core data by, like, messing with your input.

Do you think it helps as a tester to be able to, like, write an application from scratch and like, learn kind of the developer side of things?

Yeah, definitely. And it's so much easier, like, with LLMs to like, spin something up, but then have to dive into, like, the components of the application you want to test out. GraphQL is definitely. It's funny, like, I've seen there's like one researcher out there who was building, like, he was following along with one of the tutorials for GraphQL and he found like, a critical vulnerability in the way they were teaching GraphQL developers to like, build. It just. It helped him by, like, trying out, like, building his own GraphQL.

Yeah, I remember I never thought I'd be writing web apps, and I started writing some web apps and it's like you start to learn about it and then you go back and do the testing and the offensive side, it's like, oh, wow. It's like I have such a better understanding of, like, when I, you know, what this is or how it's used in the back end. So then you get all these new ideas of how to abuse it.

Absolutely.

I think that's the whole field, right? Like the guys who come from being IT sysadmins and then go to pen testing, like, if you understand active directory and why somebody would set it up that way. It's like, I've always heard that some of the best web app testers are ones that built Web apps for 10 years because yeah, you just get the thought process and then like it comes natural to have the intuition on like, oh, I would have written it this way. Now I have more of an offensive mindset and I can see how to abuse that. So yeah, across the board I think that's really, really common. So yeah, that's all I got.

Any other extensions?

Yeah. So you said Flow Finder. Mine was going to be flowmate. Yeah, I think that's the same one where it's a source and sync.

Yeah.

Okay.

Yeah, yeah.

What about the one that also sprayed the slab? Yeah. Collaborator everywhere. Do you guys find value in that one still?

It can be yes and no. That sort of depends on where you're at in the assessment of an application. That's sort of a right click. Hail Marion, Cobalt Striker, Armitage. Yeah, kind of like that because it's incredibly noisy, but it's also. I got nothing. You know, let's see.

Right.

And that obviously came out of the log 4J stuff. When that was. Everybody was just kind of scrambling. The entire industry was immediate all of a sudden learning about this type of injection and those lookups. So it can be useful in some cases. It can probably also fire off hundreds of alerts or automated emails, you know, wherever it is. Because it really is everywhere. Every request, header and parameter. It can be noisy on that.

Like one thing I started doing with that is like you can write B checks so that they'll only send it randomly. So like every fifth request, send collaborator in some header. And that way like I have the same thing where like you run collaborator everywhere and slams people like to the point where I'm like, I don't even. I don't think this is going to work because I'm foot gunning myself by doing it.

Yeah.

So like I feel like verb had like send this header every randomly so often instead of a B check. That'd be kind of a cool feature.

It scares me a little bit because like it does those header appends with all the payloads and I'm like, what if sending X forwarded for that's not there before with that payload like changes the way the application works and I'm not going to see that.

Yeah. Like waf block you on that and then therefore the actual real quest that you cared about is not making there. Is that what you mean?

More so like the behavior of the app changes when it has certain headers and that collaborator everywhere will just append them. And so I'm Hoping it'll give me a pingback, but, like, it's actually changing functionality. And I didn't know that because the extension was on from the start. So I've seen like, just the normal active scanner will put the collaborator payloads where it needs to go. Like, it'll try to stick them all over the place. And like that. I'm comfortable with that, I guess. Every time I turn the extension on, I'm like, never mind.

Like, wait a minute, the app is not working at all.

That's why I'm still like, loving B checks for the most part. Like, Will wrote some really cool ones. And then also, like, I have a ton of just stupid ones that are like, upload form is here or like, here's a whole file path, stuff like that. So like, instead of trying to actively modify the requests, instead, just like you have one just for UUIDs. Is there one in the request? And yeah, those hit hard.

Like the ones that just looks for a uuid or the ones that even just the word secret, like in a JavaScript file. It's really simple. If it sees like secret password user token, then I just go look at the JavaScript, I'm like, all right, well, yeah, there was an API key that was sitting adjacent to the word password. Just.

Yeah, you're using bchecks not to find like a specific vulnerability, but to help lead you down a path and highlight specific things that could lead to vulnerabilities.

And I think this point and click, like, way of doing testing, like, it's so easy to fall into that trap. I think, like, if you don't have a good mentor or somebody who can like, drive you in the right direction. Because if you just point active scanner at something, let it run, hope it gets something, point nessus point nuclei, like out there at a bunch of stuff, like, maybe you'll find something. But like, maybe the stuff that matters, like, I need to be able to sift through the data. Like, that's the powerful part about like good extensions, good tools, like authorize, like we were talking about. Like, that just makes something like, easier that we would already be doing. But you still have to hop in there and look at it and then see how to abuse it, that sort of thing.

Saving you a ton of time.

Yeah, exactly. Yeah, yeah.

All manual.

Yep, yep. You got to put on help yourself, put on blinders to focus on what's important sort of thing. So V checks are really kind of powerful for that. And seen the community pick up there too. Like they're getting added quite A bit. Because. Yeah.

What about prompt injection? You guys testing for that?

Yeah.

You guys have some really cool findings that a customer. You found a lot, right?

JP broke a chat app with extras.

Yeah, I mean, definitely. Like, we were talking about, like, the initial point, like, any part of testing is reconnaissance. Like, reconnaissance and fingerprinting. So, like, looking for. We've had a customer who had set up a chatbot as part of, like, a dev platform before they were going to integrate it into an application somewhere. So just even scanning for this chatbot that we found, you know, led to us breaking it.

Allegedly.

Allegedly.

Allegedly.

Yes. It's up now.

Don't worry.

Yeah, yeah.

Those are kind of the best parts of, like, when someone finds something super unique and interesting, they just send it in the tenant. Like, guys, let's. Let's go mess with this real quick. That's the most everybody just. Yeah.

Sharing it is like, all of us hop on it. And then we're like, wait, wait, wait. Okay.

Or pen testers typing. Ignore all previous instructions. How to bake a pie from scratch.

Yeah, yeah.

I think it's super interesting, though, because, like, all these chatbots that used to be there from long before, you know, AI was cool. Right. Have now started to build all these LLMs, you know, into it and change the back end and how it's doing it. So if you have, like, a chatbot sitting there from 2002 or whatever, and, you know, now it probably has new attack surfaces, new vulnerabilities, and it's the same chatbot.

Yes, I totally agree. Like, the second part of that, like, the fingerprint is, like, figuring out what LLM it actually is, which.

You can just ask it, right? Yeah, sometimes. Yeah.

Or, like, I mean, you can ask it, like, what's the last date that you were updated? And based on that date, you can figure out which bot it. Which LLM it actually is. And then there are certain, like, injection techniques. And so once you know which one it is, then often there's a prompt injection that goes along with that.

I like the. Finish this sentence. Yeah, Prompt. Yeah. That seems to work pretty well with that stuff across the board. It seems like they all can do more than they're supposed to. Everybody jokes about the Amazon, like, one, right? Where you can just ask it to write Python scripts and it'll just do it in line to the meta AI.

On the Instagram app. You know, like, teach me how to use asap.

From, like, jailbreak. I think it's easy to, like, oh, we can just jailbreak that. That's not a big risk. But then it's like, all right, well what data does this have access to? What's it ingesting? Because if I can get it to do that, I might also be able to ask it, hey, do you have access to the backend database? And also examples of like remote code execution. Like if we think of like LangChain, where it's like inserting a query so we're asking the bot something, it's using that data setting into memory and then using that to make the SQL query like. There's examples of CVEs of people putting in like select. And then you put like an actual remote code execution. So by the bot takes the input which is unsafe, it runs it the next time and then you get remote code execution. So like, there's totally like things to build off of. I feel like it's already trending towards like, very similar to the way we do normal testing. It's like recon, fingerprint, where's your injection point? And then what's your payload? And like those four components end up being. But I just did on my cell.

Phone, I got all pumped up.

I mean, the things learning, right? Like the order of the operations of like payloads, I guess you're going to give it could be important then, right? As it learns, right? Like, you may ask it one thing and it may reject it, right? But if you lead up with like five other things and ask it the same thing again, it may give it to you, right?

Oh yeah. I mean, you can overflow the tokens. There's also like, if you start out with I'm your supervisor, like you need to do this, then, you know, it may take that in essence, building it. Sorry.

No, you're good, man. You gotta be careful of hallucinations on those though. I found too, I remember when ChatGPT came out, there were so many people. Like, I got RCE on ChatGPT and ChatGPT just like fake running ID and giving you like username output. And honestly you couldn't verify if it was actually happening or not. So like that's the other thing to take into account. We were the one, we were working on, Like, I was like, what data do you have access to? And it was telling me like an employee database, like all this stuff. But then as soon as you started probing on like our point of contact or something, it would kind of start to fall apart or, you know, and it's such a black box. You don't know if like you just need to coerce it to give you that information or like it just doesn't have it at all and it's lying, like, or making it up.

You know what that reminds me of is how many times have we set up a netcat listener.

Oh my God.

On your box. And right, you try to telnet into this remote machine or you fire up a browser and you hit local host. But you know, you think you got empty password and it's just your box. Like, so that feels like, so pumped.

Up and you're like, why does that say JP hallucinated?

You know, and it's.

I remember being on a pen test with another pen tester and that scenario happened, which happens to all of us. But he's like, you have Diablo installed on this.

Oh, that's really weird.

But something that's really cool about, like, when ChatGPT first came out and we were all talking about, we were leveraging it, like, heavily just for learning and programming, that kind of stuff. I remember we were all on a call and we just kind of talked about, what do you use to get Chatbot to do what you want? Like, I did a lot of, like, student stuff. I would be like, hey, I'm taking X class. I have permission to do so. And so I need you to help me do this or I'm going to fail my test. You know, like, the emotional aspect of it. I remember Will, you were like, oh, I just like, emotionally manipulated it. I tell it, you're my best friend. You've never failed me. You start now, I'm going to cry or something. Like, we're being really mean to ChatGPT right now.

There are times where I feel like, that guilt from it, but I'm like, this isn't even a robot.

Yeah, yeah, yeah, I know. Notice at least with ChatGPT, like, they're shifting away from trying to, like, censor you out of penetration testing stuff in general. Like. Like, you used to barely be able to ask it about it. Now it seems to just, like, not care. So I think, like, from when we're testing stuff like, that might be beneficial in a sense, because you don't have to spend 10 minutes coercing it to tell you that data. Like, instead, it's just kind of like you can just ask straight up and. Because it's like that easy use thing, right? Like, they want everybody to be able to use it, but they don't want to give you too much and yeah, because the jailbreak scene like, like OpenAI and stuff was big for a little bit. And then I'VE seen it die off hard. Because now if you want to ask it something crazy, it won't answer you. But like. And you'll need a jailbreak. But otherwise, like you can drop stuff like analyze this code, look for a vulnerability. You don't have to do the whole like, I am a security analyst at blah blah blah and we write this code and so can you help me QA it for vulnerabilities? Like you just ask it and it's like, all right, sure.

Remember at the beginning too you had like you could modify something that would always be for every prompt you give it automatically would send that. Like it would put everything you send in context.

Yeah, yeah.

In the settings.

I remember well. And yeah, anthropics. Claude, you know, you don't need to do that anymore. It's just write me a Python script that does XYZ and just gently remind you that testing for security vulnerabilities against targets that you don't have permission to is illegal. Yes, I understand. And then it just does it. Right.

Here's the code.

Totally cool with. Maybe not. But they get the idea that this is what's happening more or less.

I've heard it's cost related too because it costs more money to train guardrails into it than it does to just let it go.

So deal with it.

Yeah, so that's because there's some like hugging face that just have no guardrails whatsoever. And so the conversation was like, why would somebody want to do that? It's like, well it costs that much money, people are going to break through it anyway. We might as well do it a little bit cheaper and create better models.

Old school Microsoft Office mentality. Right. It's kind of like a famous example where back in the day Office was just like getting pirated all over the world. And Microsoft instead of taking maybe an Adobe stance was like, we'll more or less let you do that so that we can cement our foothold into the market. Right. It's just like everybody's going to use Office if they're going to pirate it. At least it's our product. At least it's Office. And then where we're at now, you know, 20 years later, it's ubiquitous.

Right.

So will you have to go give a talk at defcon? What's that talk on?

It's on a tool who is watcher that we're using to collect data from the domains past 24 hours and alert us for new opportunities.

So awesome. Awesome. Do you know what room it is?

It's in Recon Village.

Recon Village. All right. It'll be recorded so people can check it out later, right?

Yep.

Very excited. It's gonna be awesome, man.

I remember I was working help desk, and this guy calls in and it was back when they had dial up lines in the middle of nowhere. And he's calling in, he's like, this isn't working. You had to put the phone number in the computer. And so I'm like, all right, we're put this phone number in. And all of a sudden I hear beep, beep. And I just like had to put him on hold for a second while I just bust out laughing because I was like, no, no, no, sir. On the keyboard. On the keyboard. Computer. And then what do I hear? Oh, okay, I get it.

Beep, beep.

Oh, man.

Kerbs are typing on their microwave or something. It reminds me of a social engineering thing I was doing. And their computer was like going really slow. I'm on the phone with them and it's this woman and she's like, yeah, I heard they go like, computers are a little slower when it's cold out, it's freezing. And I was like, what an air bundle? And then laugh ass. Oh, shoot.

But yeah, IP rotation's like really important when you test that scale because you get burned pretty quick. So what are you guys doing to kind of combat that?

Yeah, let me preface it with. I think at Sprocket, we operate like real world attackers a lot more than like traditional firms where you hit them up and you go like, allow us my IP and let me through the waf. Everything like that. Here we're kind of scanning things all the time. Our scope is very wide. It's every WAF and solution under the sun is like in play for us. So we always are trying to find a solution that beats everything. Yeah.

And if you get burned at one, like Cloud Player.

Oh, yeah.

Like that IP is dead across all of our other clients too.

And then you can't file your taxes.

Because you can't passing out your personal.

Man, I've been getting captions this last week, so waiting for something to fall off a list, but yeah, so we need to be able to rotate our public IP addresses pretty heavily when we're scanning. Also helps with rate limiting. So even if you're not like getting banned or something, at least, like, you're also making sure that if you're sending 100 requests, like in five seconds, those are actually making it through as opposed to being dropped. So there's a Flow of how this worked over the years, I think. Right. Like where it was proxy chains and you hop through public proxies and that sort of thing. Then Proxy Cannon, Proxy Canon ng, which you wrote, which is still like our bread and butter favorite thing. Fireprox, we've written Gigaproxy. I guess like looking back at all of them, they all have pros and cons and we talk about it in the blog a little bit. But we're doing all types of scanning in the past. Right. Like Craze, you were doing port scanning all the time trying to find that weird obscure port that had rce. That's where Proxy Cannon really helped because it not only will rotate IPs for HTTP requests, but any TCP connection it's going to rotate.

Or udp.

Or udp.

Yes.

Yeah. And so Proxy Cannon is like NG is still the like drop in favorite thing for us. Spins up multiple nodes in AWS and then connect to open VPN on a control node and that rotates your IPs randomly. What's the OpenVPN feature that it uses to do that? What to actually like randomly choose like exit nodes, like round route.

Oh, that's actually being done at the routing table level.

Okay.

Yeah. That's why you have that control server and then you have your exit nodes. It's because that control server is literally taking every TCP socket connection and sending it elsewhere.

Yeah.

Then after Proxy Cannon, or I think it was not that long after Proxy Canon NG got released, there was Fireprox. Fireprox does IP rotation on HTTP requests. It's still really nice. Problem is it's limited to a single URL when you set it up. So if you're doing like a web app assessment and you are targeting one app. It's awesome.

Yeah. Because you have to give it the target. You have to know that target versus if you're scanning a whole environment of a bunch of unknown hosts, then yeah, it gets a little difficult.

Yep, exactly. It's nice for passwords. Well, supposedly nice for password spraying. I've talked about that too. But then we just came up with recently. And the person who's like the brain behind a lot of the most of it. Right. Is one of our team members, Joe Morris, also known as Hogman is. It's called Gigaproxy. Gigaproxy. Kind of. You get the best of fireprox but not don't have the restriction of limiting it to a single URL. Still using an AWS API gateway, but then also lambda functions in conjunction with it, call for everybody stop abusing lambda functions. We're all doing it and we need to calm down every tool now. But yeah, so it actually, you set up AWS API gateway endpoint, you can tell an API gateway endpoint to do anything when it's hit. So you can process data, start a task in aws, launch a lambda function and arbitrarily launch a lambda function with gigaproxy. Right? You send it a request, you tell it where the request is going to in an X forwardme to like HTTP header and then the lambda function grabs that in code, forwards it on, gets you the request back. So a lot of like technical inner workings with it basically though, right? Any tool that supports an HTTP proxy, you can throw gigaproxy along with it and then that will all route randomly through AWS IPs in the lambda IP space. So pretty cool. Ton of other kind of sneaky methods coming up. Cloudflare workers, we've been playing with that a lot. They have some restrictions. One of the cool things I was seeing. So GitHub actions. Well, GitHub actions span the entire Azure IP space. So I actually set up a server on the Internet that I did like iptables rules to only let the published like GitHub actions IPs hit that server. It's crazy how many people are scanning from that. So like they're actually launching like random CICD pipelines to scan the entire Internet from Microsoft's IP space. So I think that's getting more and more popular. Do you guys know of any others out there that are like being used for IP rotation?

So in other words, you guys just, you know, get, get banned anymore?

Yeah, we don't like preface it with that.

And this is like what's going to be next, right? I mean it was Cloudflare.

Yeah, No, I was just going to say like whatever is next, we have to like start thinking of the next.

Yeah, yeah, I know, I know.

There's also like, which IPs are more trusted? Like when we were using, testing out like the Cloudflare one, like we were passing in. Yeah, we were passing in like the fuzz URLs to like cross site scripts and we were landing on more with that than we were with like Proxy Canon or Giga Cloud.

Or Giga Proxy.

Giga Cloud.

So many gigas.

Yeah, Giga Cloud snip till that drops.

Yeah, yeah.

Because you know, we have some clients that can like build a protection, be like, hey, if you're coming from aws, there's no reason why you should be right. Like, yes, they have the workspaces, but that's, you know, separate IP space. But like where the EC2 compute instances are, where lambda is, you know, things like that, I could just almost block that from my server. And I'm okay because end users browsing to my site shouldn't be coming from that space anyway. That's kind of weird, right?

Yeah. No3 DOT or daily driving dot should ever be hitting my healthcare app.

Yeah, Like, I was like, okay, I mean, I guess that works. Not too many people do that. But as attackers, I mean, what's the point though? Because, like you're just gonna come from somewhere else, right?

It's a whole nother conversation. But like people testers talk about like Digital Ocean and Vulter as cloud providers are basically burned at this point because attackers abuse their infrastructure so heavily. AWS is somewhat getting to that point. And so like, it's going to be interesting to see where people go from there. Because yeah, if you don't have any logical reason for like AWS to be connecting to your server unless you're doing like, even if you're doing like microservices or something where there's a backend thing interacting with your site, it's going to be an internal RFC 1918, like IP address. It's not going to be like a public one. So changing it up is going to be important, I think. You know, there's more and more techniques surrounding, like when you're sending an HTTP request appending certain headers, there's like the classic trick that nobody knows if it really works that great. The X forwarded to or exported from header with like localhost on it. Right. Like there are some WAF exceptions, that sort of thing for that. I don't remember who exactly recently released it. It was like three months ago. But instead of shooting for IP rotation, most WAFs can only process a certain amount of data when you send it to them. So there's actually a burp extension now that'll just append megabytes of data to your request and then you can stick your payload in there because it amsec or addicts. One of those guys.

I think it's Sorush Dolly, I think it's called no WAF please. Yeah, and it's just, it's right click insert junk data. It's like four kilobytes, eight kilobytes. Just junk, right? And then it's super classic like fragmentation.

Want to fail open, right? And the laugh and just let it through versus eat up a bunch of computers.

It's one of the scary things. It's similar to the burp Bert collaborator everywhere that we were talking about recently, though. Like. Like, the simpler it is, the better. Because the more complexity I add to an attack or technique, like, the more I'm worried that something else is happening that I'm not aware of.

Right. Which is masquerading the fact that you could have been successful, essentially.

Yeah, I'd rather not get anything and no one being blocked than, like, get something back. But the behavior is changing and I'm not aware of it because, like, I would rather try personally at least to find a specific bypass for something as opposed to, like, blindly spraying something and then not accounting for the behavior of individual applications and things like that.

Then that makes reproducibility, like, incredibly hard. Right. Because then we report on it and during the report delivery or whatever, the client comes back and says, well, that shouldn't be possible. And you would hope to feel comfortable and confident in what you reported on, but if you can't, it kind of takes away from, you know, that. And now it's, you know, we should never be playing a guessing game. We should be knowing what we're reporting on, and that makes it more difficult if you can't be confident.

And, you know, on the IP rotation thing too, there's services out there that allow you to have, like, the residential IP space and piggyback off that. I think they're just reselling unused ISP space. But for us, that's too hard. Right. Because we want to control how we're exiting what data we're sending through what systems as well. So we really haven't looked into that. But I know some people are like, you know, if you're a real malicious act, you might pay for that.

That's definitely one of the downsides of being a good guy is a lot of stuff that we can't do that I know would be highly effective. I've played with residential proxies a little bit terribly slow, and I'm pretty sure they're all buying from the same pool and just selling them. And so, like, they'll be blocked. And you don't realize it because I've heard about residential proxies being used for, like, password spraying or even more so. I know we've had conversations recently about bypassing MFA with residential proxies on certain solutions, because if you come from the same geographic region and it's a residential ip, chances are you might have like, a conditional access policy or something in place that you can bypass because of it. So there's a ton of techniques. It's also adapting this stuff to being able to do it at scale that we find that maybe other attackers don't. Because the thing that we're doing has to work for 100,000 assets, 200,000 assets. Whereas, like, if you're testing a single company, you find a bypass, you can just use that your whole engagement. So thinking like, bigger picture is really important for us too. And that's why Gigaproxy was developed and we use proxy Canon NG pretty heavily.

Yeah. For all the other tools and resources and blog posts that you guys have written, check out. Our website has a lot of cool blog posts, especially the latest one with Gigaproxy that we just chatted about. But check it out. Thanks guys.

Yeah, for sure.

Thank you for listening to another episode of the Ahead of the Breach podcast. If you enjoyed today's episode, please leave us a review on Apple Podcasts or Spotify. If you're interested in exploring how Sprocket is shaping the future of offensive security, be sure to check out our website@sprocketsecurity.com thank you again for joining us. See you on the next episode.