Performance Testing APIs with Patrick Poulin

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance Podcast. Today we'll be talking all about API Shift left performance Testing when with Patrick Pullen, the CEO and co founder of API Fortress. You don't want to miss this episode. Listen up. Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Patrick. Welcome to the Guild.

Hey, thanks for having me. Love a good trumpet intro.

Awesome. So, Patrick, it's been a long time since you've been on the show. What have you been up to? For the folks that don't know, just tell us a little bit more about yourself.

Sure. Patrick Poulin, the CEO of API Fortress. We've been in the API testing space since I'd say 2015. We were mostly stealth initially, but the last two years we've been public and doing really well, getting a lot of large and medium sized customers.

So Patrick, I think a lot of times when people think of API testing, they just automatically go to functional testing. So I thought we'd do a little bit different thought process here and focus more on performance testing. So where does API testing come into the performance testing picture?

Well, I mean foundationally, APIs run a lot of these web platforms. I know in my background I actually came from, I was building my first job in tech. I was building the world's first mobile apps and mobile websites. And one of the bigger issues we would have was this was back in the iPhone 3Gs days when the networks were not the 5G they are today. Well, allegedly 5G, but everything was just slower. So if your mobile app had a one megabyte download or a payload that was a megabyte, the app might crash. The device didn't have the memory to work with that. There was a lot of the weight of things was significant. And now the more statistics we have, the more we prove that you'll lose a conversion if the performance of your application isn't up to snuff for customers simply because they have other options.

So, Patrick, I know you've been going to a lot of conferences as a vendor. Just curious to know what your pulse is on the industry and more folks using API testing for performance testing. Or are they still even neglecting it in the functional testing part of their test strategy?

I would say in the last year, which is why API Fortress sponsored close to 20 conferences. This year it's really changed in that API testing has become a much larger conversation in the testing community.

Everything is build first, test later. APIs where been around forever, but really the inflection point happened about two years ago. And so early last year we'd be talking to people, they wouldn't even really know what an API is. They wouldn't be testing anything. Like your sponsor, Smart Bear. Smart Bear has some pretty interesting white papers out about the statistics of how many large, medium and small organizations that don't have a formal API testing process. And in the last year that's really sort of exploded. We do presentations sometimes on how to go from manual API testing to automated API testing. And it's, you know, two years ago when we were doing these presentations, it would just be 10 people and now it's standing room only because it's, it's, it's, it's a thing now. Like now that we've sort of have so many solutions and been so matured in terms of UI testing and mobile app testing, it's time for that next phase, which is the actual foundation of those things. It seems crazy to say that people aren't testing them, but I could tell you for a fact that it's, it's pretty rare we speak to an organization that has perfect functional testing and load testing of their APIs.

Already.

So I would think it would be easier to do performance testing or functional testing of an API, because you usually have that done before the UI, and usually APIs are driving the UI. So what seems to be the resistance? Just education.

Education and lack of tooling. The tooling now exists, but you know, I mean, Load Ninja came out this year and API Fortress only had load testing earlier this year. Before that, people were trying to use like JMETER to do load testing and then they'd have you really had to program it by hand. That's one of the reasons that API testing has had some stagnation. You really have to be a lot more developer centric. And so it's, you know, you require people with a more specific skill set and those can be more difficult to Find and they already have a lot of stuff to work on, a lot of other stuff. So with the load testing, like the tooling has really come out as of late, that's been significant. And I know in our platform like the big focus for us is to not have completely disparate products that do different things. So in our platform, like we generate an API test and you tweak it and you can create this really smart end to end test. But the benefit is why not just use that same test as your load test? Previously people might use JMETER for load testing, but then they have to write a completely different sort of test. If they're going to do functional testing, they'd use a completely different tool. So unifying those has been one of our main focal points. And I think that's why things have become a lot more prevalent. And it's just the beginning this year, it's going to be even bigger next year, I'm sure of it.

You seem to always be adding functionality and I'm sure you're not just doing willy nilly. So obviously there had to have been demand for performance testing. So I like how you said you're reusing your assets almost all in the same ecosystem. So how hard is it to take an existing functionality API test and then make it a performance test within your platform?

It's the exact same thing. When you go to the load testing option, you just pick one of your tests and you say, all right, run this using these load generators for 300 milliseconds with a duration of 300 milliseconds ramping up over 30 seconds. We very specifically built it, so it's all built off the same thing. The thing that's been really interesting for us is that was obviously well received and it's now a major part of the packages we sell today. But what's been really interesting is seeing how people are now leveraging the APIs. Because we're a platform first, we have tooling and command line tooling and all that stuff. But ultimately everything in the platform can be hit by an API, whether it's execution, test generation, whether your data or notifications. On an API level, you can execute your functional tests, but you can also execute your API tests. That's been really interesting to see. We just built that just because we build everything on API first, but to see how well received that's been that people wanted to run these load tests as part of their CI pipeline. Because it should be noted when we're talking about these load tests, we're Using our existing functional test and actually running the functional testing along with the load testing. So you get all the statistics on the load testing, but we're also telling you like, hey, if there's one object that can't keep up, if there's 5,000 concurrent users, that's good to know. And customers report back to us, that's what they find. They'll find a memory leak or they'll find one database that's feeding one specific object that just can't keep up under a certain amount of load while everything else responds 200. Okay. That specificity is significant in this day and age where we expect more of ourselves.

So what differentiates a functional API test from performance API test? Is it just you're running more of them?

No, no. So a functional API test, the core feature of that, I mean this is all goes into everyone has a different definition of functional test, end to end test, integration test. But tested API for just like when we talk about a functional test, it quite literally means it's almost like proofreading a book. The entire payload being validated, not just that every object is there, but also like the data that's associated with that object is there. And then on top of that, like our platform, one of the big things we push is that the business logic testing, which you could only do with an API. So let's say you have a retailer and you search for the color red. We might find red shoes or a red dress. They're both going to have a size object, but that size object has a very different range, whether it's a pair of shoes or a dress. Pair of shoes would be like 2 to 22, although if it's in Europe, its shoe size would be like 32 to 52. So just those ranges can be validated on an API level and you can't necessarily validate that in a UI Selenium test. And there's always a way, but that's just not how people are doing it. And that's the benefit of focusing on API tests. You can catch a lot more. And to us, like that's how we're defining functional tests. A load test to us is running a reproduction of a normal user behavior at a much higher level. So for us, like search add to cart checkout, that would be one solid end to end test you create in our platform, then running that exact thing 5,000 times over four minutes. For example, like to us, that's, that's the difference. So it's the proofreading of a book versus like what happens when you run 5,000 of those at once.

Patrick, Another thing that usually comes into play is monitoring. So does the platform offer any sort of monitoring or do you integrate with any sort of monitoring systems so as you're doing that load that you're able to see different metrics?

Yeah, that's actually a funny. Literally, in the last 12 hours, we just closed two new deals. We're one of the major reasons why we won those deals against, you know, we went head to head against other people and we won them because monitoring is built into the platform. That actually was initially, that was our initial intent with API Fortress. It was a functional uptime monitoring platform. But with the shift to Agile and cicd, like, we had to sort of talk a lot about that other stuff. But as of late, it's shifting, it's shifting back to where we originally started, whereby, you know, monitoring your internal APIs that are the basis of your internal platforms or the basis of your customer facing mobile apps, you need to know those APIs are up, and not just up or down, but functionally up. That's been interesting for us. We get people interested by saying, yeah, we could integrate automated testing into your CI pipeline. And so that's how we started the conversation with the last two people we've just closed with. But then the next piece is like, all right, but then can I use this test as a load test? Yes. All right, cool. Can I also use this test as a functional uptime monitor? Yes, you can. And so it's been really powerful for us because we have. The one thing that people don't realize is that if they're not specifically in the space is that APIs can be very sensitive. They are often not available. Most of them are not available outside of an intranet or outside of their firewall. So you need to be able to either get your IPs whitelisted by the customer, which is always sort of pushback from the Security Ops team, or. Or just deploy the whole platform internally. And that's a unique differentiator for API Fortress. Like, we're container experts. Our whole platform runs out of a Kubernetes or Docker container. You deploy it on prem and you have complete control and ownership of all the data and the results. And everything can hit your internal APIs and microservices that are behind your firewall. Because it's all internalized, nothing goes back outwardly, so Security Ops doesn't struggle to, to approve it.

So I actually didn't know you had that functionality. I love the concept of Shifting right almost where you wrote your test. You're all the way through the lifecycle, you release it in the wild and now you're monitoring it, gathering that feedback to inform you how to develop again. So it's like iterative, so it sounds like you have the ship right in place as well. Now how many companies are using that feature? Is that common?

It's becoming more and more common. I would say in like the last two months, all the deals we've closed, about 70% are really into the, they really came in hot because of the monitoring aspect capabilities. So for us it's, I'll give you one example of one customer. Like we actually, we don't just talk to QA team. Our platform is very received by like architects. So people that are making a big push to APIs, they might be buying like a big API, like an EBS, like an Apogee or Mulesoft or Carm or Tibco mastery, what have you. They're buying one of those because they're making a big API shift. And so it's like a chief architect that's looking at that. But as they're purchasing that platform, they realize that those platforms don't have testing. They've never had testing. Like they just, they have pieces of it, but they don't truly have proper API testing. And so we get bought as part of that same purchase. So it's a budget of like, hey, we got this thing to help us build, but we also need another piece to help us test. So we come in as part of that purchase because the architects say like, oh cool, I can write an endpoint test using my own IDE execute from the command line. But then they can hand it off to the QA team who can pick up your endpoint test that the developers created or the engineers created, use those as the foundation when they create end to end tests. So a developer might test like he built this endpoint, he'll test that endpoint. QA is looking to see how does that whole API program work throughout a normal user flow. So a single test that does again search, add to cart checkout, that whole flow should be validated in one intelligent test and then it gets handed off to that same test. DevOps team and the ops team can come in and say like, hey, I'm just going to schedule this as a monitor. And it's not just monitoring production. Like we're seeing more and more the interest in monitoring different tiers of their staging environments, like staging one pre prod gold standard. Like they want to monitor all those pieces because everyone is working off of the staging environment, they're not working off production. So a lot of this monitoring comes into monitoring internal environments.

Awesome. So the full name of this podcast is the Performance and Site Reliability Podcast. So I think this type of testing is critical for your site being reliable. And like you said, you're able to monitor all your different environments. So what happens? How does that work? Does it say like there's a threshold that if you don't get a response back in a certain amount of time, three times in a row, that they get alerted? Like, how does that piece work?

Well, with API Fortress, for the monitoring, we're using a functional test. So if there's any functional issue which can include just like the size object not being in the correct range, if you choose that to be one of the options, you'll be notified of that. And if there's a critical outage, like we have throttling there so you don't get inundated with emails or slack notifications or pager duty pings or JIRA tickets, like all those are built out into the platform as integrations. But there's throttling for major catastrophic events. But you really choose whether you're just validating that it's up or down, whether you just want to validate that the fetch of the latencies below a certain level or and it's what we always suggest, like proper functional test. That same test you would use to validate your latest deployment as part of your CI pipeline, that same test should be your monitor. And that's really easy in API Fortress because we have a built in schedule. Take the test, click on the schedule button and you're done.

So, Patrick, I'm just trying to remember our previous conversation. I thought you brought something up where your tool had some sort of functionality, where it did a lot of things for you automatically that other tools may not have built into them. Was it the creation of Don? I forgot what it was.

Yeah, we had a test creation so we could create a test from either a payload or from a spec file. And so it gives you a draft of the tests and which is really beneficial for customers that are dealing with. I don't know if you ever looked at like Instagram or Spotify's.

Payloads, but they're very long, very nested. That's incredibly annoying to set up on like programmatically, if you're just looking in your ide, our platform actually set that all up for you and then it gives you the ability just you tweak it from there. So for one analogy I like to make is people ask, like, how do you write a test? I'm like, it's sort of the same way that you. We all learned how to create a website. You looked at a website you liked, oh, this thing has frames. That's amazing. Then you right click, and I probably just dated myself there. But you just right click and go, view source. And suddenly you understand how the structure of the HTML works. That's how we like to do in our platform. We create the foundation, then you edit it and make it smarter from there.

So, Patrick, I think a lot of people sometimes are worried about APIs because they don't necessarily know about all the other dependencies or other microservices that may be needed in order to make a full transaction. Do you usually just focus on one API, or do you create like a flow that automatically touches all these other APIs?

I mean, we always suggest that people create a test for every endpoint as well as creating entire user flows. So what, like, you know, your 80% of your customers do creating that flow as a test on an API level. But the thing that's been really interesting, we've been building a lot of stuff since we last chatted, and we have this micro gateway we call AFTUM. It's in our GitHub. It's open source. What it can do is it can record API calls. It makes it very easy for you to record API transactions. And we actually run into a lot of organizations that maybe they've worked there for a year, but that company's been around for 20 years building APIs. So they'll have a mobile app and they'll be doing mobile. This actually just happened with an airline I've been talking to lately. They have a mobile app, and the person that does that mobile app testing has been told, all right, cool, we need you also start testing the APIs. So he's starting this journey and he doesn't even know how to make an API. He knows how to make API calls, but not all of the API calls that are involved in the mobile app. He was never given a proper set of documentation that says, hey, the mobile app uses these 19 endpoints. And so what we're helping them with is like, well, you know what, you could just set up our micro gateway so you do your normal mobile app testing, run your automated suite, or just do it manually. We will capture the API calls behind that and give it to you in the screen we call the logger. Then you could See all of the API transactions that are involved in your normal usage of the mobile app. Then you can create tests right from that logger screen.

That's really cool. So you talked about educating people, going from manual to functional testing. At what point do you recommend performance testing? When you're talking to customers now, are you trying to shift it left or is it still after the fact once everything's built?

No, I mean we show them as part of the initial demo because I mean our load testing isn't meant to replace like HP Load Runner. It's meant to be fairly lightweight but very informative as part of like a CI pipeline for like testers. And so that's why we show them like just so you know, this exact same thing could all run as part of your CI pipeline. We try to get them hooked onto it right away because you catch a lot of stuff you didn't expect. Like there's a lot of memory leaks and just, you know, older services that aren't keeping up. That should be, it should be seen before it has to go to the loadrunner team. A lot of large organizations, their load testing team is completely disparate and different from their, from their actual testing team. So we're trying to give some of that power back to the testing team.

No, I completely love this approach. Absolutely. The sooner you could test something, you might as well test it. And like you said, it's not an enterprise full blown performance test. But that's good because as they're developing they can do this confidence canary type testing. Fix it before it gets to the point where it's like taking months to fix. So did you see a lot of customers finding value that way?

Yeah, everyone that started you? We actually have a couple customers that started talking to us just for this capability. They weren't even as interested in the functional testing. We've gotten them won over on that. But for some reason they viewed performance as more important than functionality, which is kind of confusing, but you know, that's the world they live in. Well, different companies with different goals and different types of APIs and with different business intentions.

Very cool. So Patrick, as I said, I know you're always constantly iterating on your product. You're always making it better. And I think I didn't see the study, but I heard a study came out by Foster, I believe that did a comparison of different tools. Like where did your tool rank compared to other maybe options out there? Not to say one's better than the other, but different ones have different advantages. So how would you say yours has an advantage over maybe another tool?

Sure. The thing with Forrester is that it's interesting. It takes a long time just to be recognized as a player in the space. And they actually haven't created an API testing wave in like three years. And they said they're creating a new one this year. We're very motivated to get them to create a new wave. We would love to see our tool compared with all of the players out there because we know we built a better thing from scratch. Their tools have 12 years of iteration, a lot of great features, but what we have is what people need for today's agile DevOps workflows. That's what we're built for. We are a platform that's API enabled. We are not a downloadable application. We have downloadable applications. That's not what we are. So we're very confident in the head to head comparison of all that stuff. Now, I don't know when Forrester is going to do their new wave, but we were not on the original wave they created three years ago because we were still a bit in our infancy.

Cool. When you say API testing platform.

Again, explain what that means to folks that may not know exactly what that means.

API Fortress is entirely focused on the testing and quality of your API program. That's our entire focus. There are other tools out there that offer the option. They'll have a tab that says, why don't you write some JavaScript in here? And you could do some testing as well. And it's a very good start. You have the ability to actually upload Postman collections in our platform because Postman collections are awesome. Postman is awesome. Postman is one of the main reasons why we've had such a good year, because they've helped educate the market on what an API is. With that said, we think due to our complete, unfettered focus on testing, we have a little more. We have more capabilities for those people that are really looking to create strong automation suites dedicated to APIs.

So you built basically were built from the ground up focused on API testing. That's your bread and butter. But I mean, it's a full stack kind of solution as well. From beginning all the way to the end. It covers you.

Exactly. We have a lot of our customers that, I mean, we don't have a customer that doesn't use Postman. Everyone uses Postman. It's great. It's great for what it is. But when customers are ready to take their testing to the next level, we've had a great success in being part of those conversations and having those opportunities. But again, without Postman, I'm not sure we would have had the year that we had, because they have truly helped us educate the market about what an API is and the importance and how foundational they are to everyone's applications. So Postman's been a real godsend for all of us in the API community.

Awesome. So, Patrick, any tips for folks that trying to get more into performance testing? With API tests, is there a way they can develop their APIs to be more performant, or a way to develop scripts to make them so they make better performance tests?

I mean, I'm not an engineer. I'm sure there's all sorts of tricks engineers can suggest. I used to be a lot more technical than I am now. I'm getting dumber every day the more I have to do a CEO role. But what I can say is that the one thing they can do is just test it at every phase, every release test, get those metrics, get those numbers, and you'll have comparables. And then you could see if things are getting slower over time. I mean, I'm sure we've all seen the slow degradations happen with some of our favorite applications. I remember back in the Winamp days, Winamp 3 was so good. 3.1 was amazing. Then AOL bought it, became version 4, and suddenly it was a heavy beast. People stopped wanting to use it because the performance wasn't there anymore. It wasn't the the lightweight great MP3 player that it once was. And then version five, they rolled back to version three. Basically, like, there's something to be said for performance, and if you don't have those metrics, you won't realize that it's getting slower over time. You need to find a way to measure it and keep a history of it and make sure everyone's aware of it.

Okay, Patrick, before we go, is there one piece of actual advice you could share with someone to help them with their API testing efforts and let us know the best way to find, contact you or learn more about API Fortress.

Sure, we actually created API Testing University, which is sort of a subset of our docs. But I think we also if you just go to apitestinguniversity.com you can go there too. We're trying to put more and more information about what we're training our customers and the people that we work with, they ask us for advice all the time because we've been doing this forever. We're the experts in the space, so we're helping put more content out there, the sort of stuff that we do on our presentations on the website. So take a look there so you can understand better of like, you know, what's basic API testing and then what's more advanced API testing. When you start talking about being data driven and not using a small CSV but actually using a dynamic huge database or using an API to feed data into your API test, those more advanced use cases are really how you get from like a C minus test to an a test. Apifortress.com, i'm Patrick at if you want to email me thank you Patrick for.

Your performance testing awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com pplikeautomation16 and while you're there, make sure to click on the Try It Today link under the Exclusive Sponsor section to learn all about SmartBear's awesome performance test tool Load Ninja. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.