API Pen Testing with Mateusz Olejarka

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. And if you've been listening to me for any amount of time, you know that I include security testing as part of the testing practices that we need to know as testers in general. So I'm really excited about this interview today. Today we'll be talking with Mateusz all about REST API from a pen tester's perspective. How to get developers more involved in security, Much, much more. Mateusz is a senior IT Security consultant at Securing. His key responsibilities are web and mobile application penetration testing, source code reviews. He works as a consultant helping software development teams cope with application security related topics. He's performed more than 50 application security trainings dedicated to software developers. Previously a software developer building software for the financial sector. Also is a well known speaker. I think he's spoken at like a ton of different events. He's a casual bug bounty hunter listed in the hall. Famed companies like Adobe, gm, Jet, Netflix, Tesla, Twitter, Uber, Yahoo. So he really knows his stuff. You don't want to miss this episode. Check it out. The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Mateusz, welcome to the Guild.

Hi Joe, thank you for having me here.

Awesome to have you.

And thanks for the introduction as well.

Oh, no problem. Yeah, is anything I missed in your bio that you want the Guild to know more about?

No, you said everything. I don't have anything else to add.

Cool. So I first stumbled upon you when I was looking on YouTube and I saw an older video from 2017 about Rest API security and it piqued my interest. So I thought we'd dive in a little bit into that and probably all other types of security since you are an expert. So I guess the first thing is do you still see an issue with the rest APIs not being secure five years after you did your or four years after you did that original session on rest APIs from a security perspective, yes, definitely.

I see my predictions from like 2017 are more or less working right now. We see more REST API because this is standard when we are talking about modern applications. Modern web applications or mobile applications. Behind that there is a REST API, probably old stuff like soap, REST services, so web services or previous ways of doing this either in. Net, in Java or just fading away when we talk about modern app. In the backend there is a REST API probably absolutely. As a REST API question is right now, almost like backend security question because this is the way you have a front end which is either the mobile app or the static HTML of a single page application. And in the backend there is an API which is 99% rest API.

Gotcha. So I would assume with more and more people going towards microservices type architecture that it's only going to increase. So I guess why do you find that maybe developers don't think about it from my perspective, I would think because you don't see it like a UI that they probably think that no one else is going to be able to actually see the communication because it's under the covers. So is that a true concern? Why maybe developers don't think about API testing or security when they're developing?

Yeah, sometimes when I show for example a web developer toolbar in Firefox or Chrome that I can actually see those requests from the web browser for some people there's still a surprise. How did you actually do that? So some of them are like concerned why you can do that or I can modify that calls. This is like mind blowing for them. But most of them are aware that okay, user can manipulate data which is entered into the user interface and I can manipulate it directly calling REST API endpoints.

So how does someone find the API interfaces? Maybe that that because they're not public if they wanted to try to see if an API is vulnerable or exploitable.

Yeah, in case of. If you have a web application you can clearly see the backend API calls. Right. Just app sends a bunch of HTTP requests, you can start with that and you can also discover additional endpoints. For example, most of the endpoints are connected with words or some like if you have API connected to user you probably will see something API version slash user. And here we have some English words. So if you see several, maybe there are more. You can enumerate that using for example English dictionary and to find those in a way sometimes you can find or on GitHub or just googling some additional endpoints because some developer dropped somewhere some documents for example for the internal API which is also sometimes the case. So you can discover additional stuff that way. Sometimes I'm faced with the API I don't know Anything about and I have to like in a black box perspective, I have to discover everything. I mean the endpoints, the path, the parameters. It's hard, but most of the time I know at least bunch of the endpoints and the way to properly call them.

How do you do that though? Is it just like you said, using dictionaries to try to find the methods and parameters of an API? You know the endpoint, but how do you know other than guesswork, I guess. Are there tools that do this?

Sometimes it's I, of course I automate the process. I don't build requests myself, but still I can get for example documentation. Most popular one for example is I think swagger and some companies just happily provide you with swagger slash. Sorry, I mean the URL to documentation you can get is either swagger or application waddle file which can provide you full documentation. And sometimes it's you can find it on websites which actually do not specifically provide an API documentation. Of course. So first thing I try to do when I don't know much about the API, I try to discover documentation. For example, I have a web application, I see like four different API calls. When I get the documentation, probably will be somewhere there I can learn about additional, for example two endpoints with how to call them. If not I can again enumerate that. But most of the time I start with finding, trying to finding existing documentation either in the app itself. Some service, of course provide documentation for end users so I can quickly learn about it. But if not, I try to discover then I go the hard way, which is manually or in an automated way. I try to detect parameters. And then of course if I get endpoint correctly, I can make wrong parameters or I can send any or I missing something important which is required. And then another helpful thing is detailed error messages. So if I'm asking, let's say API order one and I get in response missing order id. So I know that order ID I have to add for example inquiry string order ID equal one and again, again I repeat the process if I get something meaningful in the response.

So what would be the main reason to exploit a REST API? Is it just like you said, to query and try to get sensitive information? Are you able to like what kind of damage can you do, what kind of attacks? I guess once you know a REST API endpoint and the parameters, it depends.

On the function, the business function of the API if it's getting, for example, if I'm authorized user, but I can get another user data just by manipulating for example user ID parameter that would be an information leakage. In other case, I mean it really depends from the business use case of the API. If it's querying some data, you can get another data. If this makes an operation. For example, you edit user normally. For example, if you are a casual user you can only edit yourself. But if you have for example user ID somewhere there you can change it and maybe you can edit. If there is an error you can edit another user or you can change your roles, which you shouldn't do. But again I've seen errors like I can edit my profile and I have role user. So first thing I do is role equal admin and save it and I log out login and I'm an admin. So it's really depends on the API. Of course I'm missing all classes of vulnerabilities which for example I have SQL injection in the parameter, right? So I do that as well. First thing I try to learn how to call the API properly. Because if I don't know, right? If I know how it works, okay, I can try to manipulate that. But if I don't know how given endpoint behaves, what it returns, I try to guess the proper way, then I start to manipulate. Of course if I won't be able to guess the proper way, I can again still manipulate the parameter values. Maybe there's a bug. For example, I will have a SQL query and it will return for me in the error message data I want.

So you know we talked about detecting methods and parameters and endpoints. How common is it for people to do something really, really not dumb, but really like have clear text so that you just, you open it up, you're looking at the traffic and you're like. You're seeing passwords being passed back and forth. Does that become more and more rare? I would assume, yeah.

Authentication in the API is different. I mean right now we see several standards. For example OpenID Connect for authentication of users. There is a API endpoint for authentication you get in the response, for example JSON web token to use to authenticate API calls. Again it's like in a classical web applications, for example some endpoints require authentications, some others might not require authentication. So for example you have like orders reports and user, right? And user and orders require authentication, but reports somehow someone forget to require for example adding an attribute on this API to require authentication on it. I remember speaking of parameters, I've seen several examples when application actually returned more data that user saw in the response, right? For example those orders, right? For Example you get a price and number, address and something else. But in the response they're really, really interesting. Data was returned more, I mean it was in the JSON but not in the user interface presented. For example, I edited user and in the end my password was returned to me but not displayed. So again, you have to be really, really careful. What do you return in case of data from REST API.

Nice. So I think in this video, it was a while ago, I don't know if you remember, you actually had an example you talked about, I think how you were able to get credentials just making one API call. Do you remember that? How did you do that?

Yes, yes, that's another case I wanted to discuss. It was let's say misconfiguration because it was a case of spring boot actuator, which is a library which helps to build a REST API interface. And it has, let's say developer method for developers enabled, which was heap dump, which is basically returning you heap dump from the application for debugging purposes only. And I found one application which had that in the production. So having application heap dump allows you to basically get everything the app had in the memory stored. So all keys were used in that case were there. So this is another issue when you have, for example, it was heap dump also popular. One I saw is API slash trace which returned basically last HTTP calls in a JSON for the app, including for example another user calls with credentials or cookies or JSON web tokens or whatever the authentication was was used. You got all request headers sent to the app recent of course, so that could also allow to get access to another user's data. So this is another issue in case of API I saw also a trend, let's say for in most of the case it was a legacy application which was the backend was let's say a deep backend and you had only front end components which were making those queries. But someone decided that we will create a REST API and for example imagine a list and you had in the previous version you had all the filtering done in the ui, let's say and the backend component just got all the filters filled up. And for example you could query data from last year only and it was forced on the user interface. But here you got let's say a static user interface and the same query was sent in the REST API, which means that now I can manipulate those filters as for example I can query data from all years, not only the last one. I mean in the previous version the user couldn't do that because of the validation on the UI layer was forcing him to put proper filter values which I mean the validation was done in the back end but right now it's done only not in the front end, but not in the rest API because of this change because.

Some code was pushed a bit to front end. So also cases like this funny errors I saw were related to also integrations. I mean applications are more interconnected. For example, one application can use several other APIs from different software providers. For example, I remember a few cases. I think one of the funniest one it was a E Commerce. It was integrated with a courier provider. You could select if you were buying something, you could select several different couriers to deliver your stuff, right? And I was back then I entered one thing that I shouldn't. I just put a test for SQL injection, I put or 1 =1 and comment and I broke. But I broke not E Commerce, but I broke the API of the provider for a while because it tried to make a time consuming query and it was unavailable for a while. So sometimes that's the issue right now I always doing the if I know that there is some internal API integrations which use external providers, I also try to learn where exactly those integrations are done because in case of the user, I think developers know that you do not trust user input, period. But in case of API integration, sometimes it's not the case. For example, if you expose some API, you should do the same. You have external party which will put whatever it wants in the API call. So you should do the same for the API calls as well.

How do you prevent for that? Like say as a developer you're like, you know your stuff, you know your code, you can handle it. But like you said, a lot of these SaaS solutions, there's probably a lot of companies that rely on third party APIs. Before you consume that API, is there anything you could do to lessen the likelihood of consuming an API that could be exploited that's not yours but a third party one?

In this case there are two places when you could work on it. I mean from the perspective of a consumer, you know what you expect to get to make that API call, right? If you pass some values from the user to be securely convalidated, for example types you get or the length, maximum length is a very good validator. Let's say if you have a license plate number, you shouldn't be able to input their HTML tags with scripts alert one end tag, right? So yeah, you can validate it and again, from the perspective of the API provider, you should do the same. You should assume that the end users will happily send you everything without validation and you should do the same again to be secure. If you expose API for public use for your customers, for end users, you should assume that you will see the abuses as well. In that calls, any calls should be validated very, very strictly.

So you started your career as a developer, you've done over 50 trainings with developers on application security training. Do you see developers getting more involved in security as like they want to or are they being forced to like how do you get developers I guess my question more into security so as they're coding that you know they can handle the things they can as a developing rat and then wait till after the fact to have a like a third party security team try to find all these issues.

It really depends from the company they work for. I've met developers that were quite aware of security, but they actually didn't have time to dig into it to implement the features they wanted because the business comes first and they have such and such tasks and have to implement the functionality because functionality is what the business wants. I've met developers which actually did have time to fiddle with security, to think about possible threats and make some countermeasures to implement some countermeasures in their code. And for me the most important aspect here is to make them aware of security pitfalls because as I understand they are smart people, they know their stuff, they what they can miss, they can miss the knowledge that it can be abuses. And for me what which works best talking with developers is showing them real examples. Not only like hypothetical application, but the real world on the Facebook. Like the back I showed on my presentation it was missing API rate limiting on the crucial password reset code. Examples work. When they saw examples they say okay, and now I understand that I didn't thought about that. And this is the perfect place when they are starting thinking about is it my code secure? I have similar function, is it okay, so to talk with developers is important to show them examples of security problems issues and talk to them in language they understand. So we have like any other profession, our own mysterious three letter shortcuts, right. So I try to avoid that much and use with developers words the from their their sandbox, their concepts they have. So that also helps. Yeah, some I met developers which had security teams in their companies but they were like separated by thick walls and they didn't know them personally, which is bad I think then if they have like good Relation. They can talk about their issues and you can learn a lot of just by talking with developers about security just during the coffee break.

Great advice. So are there any misconceptions you see developers always have? Like maybe one was like they didn't know APIs could be hacked because it's not a front end. Are there any other misconceptions that you see common brought up by developers that they don't know about security in case of misconceptions?

I don't think so. I mean they know the APIs. The rest API is available for the end user. They know basic stuff. As a developer, I remember my approach was to put parameters properly and make it work the first so it should work. So that's the mindset. They create software like it should work. So if you set proper parameters, it will work. Right now I'm trying my best to make it work in a way that I want, not that it is designed to work. So sometimes also the mindset is important. I'm not saying that it's misconception, but the mindset is that they implement things that should work. And right now there's different mindset. I try to break it in a way that it will work as I want. For example, it will return me data I shouldn't be able to get. I believe that showing examples and talking to them what could go wrong, they can get more aware and they can think about what could go wrong. For example, in my code when I see this and I won't see that parameter, for example. So the mindset is important to show them how we as penetration testers work, how we look at the REST API or any other component of the app and what do we do if they know that they can do a little bit of that themselves.

Yeah, so I love the mindset kind of idea to change the developer from making it work to making it work. But checking for edge cases or things of ways that it couldn't work correctly. Besides that, is there any tips you have to get security added more into say the software development lifecycle like in the CICD pipelines or anything to help just get security started actually added to SDLC in general?

Oh, that's a very hard question. Actually I should say that recently I wrote article about that, but a few hints here. I would say that first thing is that if you want to build in a security into SDLC or in the organization, you have to have a support from the top management. It's hard to make it from bottom up. Right. It has to be organizational change if you want to build secure applications, you have to put it as a company into your core values. Without that you will fail. And of course security is a process. You won't do it properly, you won't be, let's say, 100% secure. If you only put some automated tools into your CI CD pipeline, right? You can scout, you can have code scanners, you can have some third party scans libraries, you can have some dynamic scans, you can have penetration testing done by external company, but still you can make mistakes. You have several different activities, but again you have to think. I believe the most important part is to have to think when you're trying to implement either you are drawing diagrams on the whiteboard. You have to think about security here. So the mindset, the activities like threat modeling are very, very important to start with. So you can do threat models small, you can do them complex, but you can start with thinking about what can go wrong in the application, on the future you're planning. And you can start with that because you can of course quickly add tools, various tools called scanners. But still you need someone who take out the outcome from those scans, from those tools to try to take a look at it. And does it make sense what I have to change? Is it false positive, for example, or is it really an issue? I believe the first step to making the application or your organization secure, your platform, is to make it from the top your priority. Security is important for us and build on that. I've seen different organizations with different concepts, but what works I think is if you have like small organizations, one person, you find an insider, let's say, which is interested in security, a developer, and you let him dive deeper into the security, for example, because he knows, for example the business of the application, he knows the code, he's the best person to if have time and you invest in his or her skills, is the best person to help you with that. So the first thing is the organizational change, that security is important. And if you want to make security not an obligation, not something which, oh my God, another requirement. This like crap. I have such and such tasks, so many I have Jira, I have some reports to fill. And then you have to like I have to make some security stuff more so. So it's not the way it won't work. They have to know developers or all people involved in software creations. They have to know that security is important. Why are we doing this? And the mindset and the attitude towards security is very important. You can mess this up. Really, really. I've. I'VE seen it spectacularly. I remember a few companies, when we enter, someone said that we are pen testers and developers were actually walking away. So you don't want to do that. The attitude is crucial.

Great advice. All right, so I heard you've found a lot of bugs over your career. So is there one bug that really stands out as maybe the funnest or the craziest one that you've located?

I think I have the funnest one. Most of the API I tested or I seen have a half endpoint, which is basically if you call, for example, some site.comhealth, it will say, okay, or healthy or whatever or hello, stuff like that. Sometimes it's not health, it's ping. You get pong in response, of course. So that means that the services or API is healthy. And that's not funny yet. But you know, one such endpoint, I found out that not only a get HTTP verb was accessible, but also a put. And put means that probably you can change something. And I was super curious, why do you allow put on that endpoint? And I basically went to check and I did the HTTP request and I placed put instead of get. And I got an error which says the message is empty. So it's like, okay. So I just added to my request message equals whatever, and I sent it to the server. Server said, okay. I was like, what just happened? And I returned to like query the get verb and I get my whatever in response. So I was like, why someone did that? I was super, super. It was like super funny without any real impact. I just reported it for fun. For me. It was like, why do you allow anyone? Because there was anonymous call, of course, to set the message you return in half endpoint so I could place not healthy or whatever.

It's like, why do they get back to you? Do they ever get back to you and say, oh, we did this because of XYZ or something?

I think they said, okay, okay, this old API, we will remove it. But yeah, it's like.

That'S funny. Actually, they didn't answer my question. Why?

Right, right, right, right. That's crazy. Awesome. Okay, before we go, is there one piece of actual advice you can give to someone to help them with their security, Security testing efforts? And what's the best way to find or contact you?

In case of contact me? You can look me up on Twitter or LinkedIn. I'm happy to answer questions. You can get my publications on my blog personal or on my company's blog. I think the best would be LinkedIn if you want to drop me a question, feel free. I happily answer.

Thanks again for your security testing awesomeness. The links to everything I value we covered in this episode. Head on over to testguild.com a361 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Labs awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.