API Testing 3 ways: Postman, Playwright, and Jest Compared with Stephen Kilbourn

In a land of testers far and widely journey seeking answers, seeking skills, seeking a better way o the Test Illumination Testing Podcast Guiding testers with our nation awesomeness. From ancient realms to modern days, they lead the way. Oh, the testicle of nature Testing Podcast.

If you only know how to write end to end tests via the ui, you are really limiting your career. Potential Automating API testing backend services allows you to ship changes more confidently and also enhance your opportunities to help your team. Also, there are several popular tools that allow you to do this, but which one should you use and what are the pros and cons? In this episode, I want to share with you a private monthly Guild session we had for Automation Guild 2024 members. If you don't know when, you buy a ticket to our online Automation Guild, the next one being February 10th to the 14th, 2025, check out that in the links down below. You also get nine monthly sessions after the event is over each month to keep the learning and keep the community going after the official event is over. So you have support before, during and long after Automation Guild is over. So so in this session, Stephen Kilborn, a longtime Guild member and speaker who brings a wealth of experience in automation testing, shares his top three API testing tools, Postman, Playwright and Jess and dives into the pros and cons of each one along with demos. So if you're listening, you also probably want to check out the video as well. As you know, setting up, scaling and maintaining your own functional automation grid can be a pain. A lot of technical expertise and cross team dependency goes into achieving an automation setup that is both robust and scalable. But what if I told you there's a smarter way to navigate through these hassles with ease? Let me introduce you to Automate Turbo Scale. It's a high scale desktop browser automation solution that allows you to quickly set up grids on your preferred cloud provider. You can configure metrics like test concurrency, CPU limits, browser node cooldown, and setup alerts to ensure your grids are running with the most optimal resources. Now here's what's even more awesome. It even allows you to leverage a testing ready, fully managed automation grid on the Browser Stack cloud to run your tests. No code changes, no setup, no maintenance needed. Automate Turbo Scale can elevate your debugging capabilities as well by integrating with Test Observability, which also offers advanced test debugging and reporting capabilities. You also get instant access to video recordings, logs, quality profiles, AI beast flakiness detection and a bunch more. Automate Turbo Skill is an answer to a lot of complaints I hear from test engineers on my podcast to all the high scale functional testing needs that they deal with. So why not give it a try for yourself? Head on over to testskill Me turboscale to check it out for yourself.

Now go through kind of four steps. One is just the introduction of why Automate API testing and looking at the API that we'll test here and then we'll go through each of these three tools and write tests. The same exact tests with each of these tools. So why Automate API testing? This is API the automation guild, so I don't think I have to really sell anyone on automating. But APIs generally are going to be what you want to test outside of the front end. Or sometimes you're just making an API that is your product and if you can test that in isolation without the front end, you can catch regressions faster a lot of times. And the whole goal of our testing is to prevent regressions or find bugs before it's even gets out to the door in front of your end users. If you're testing at the API level instead of going from the front end to an API, it makes it a little easier to investigate failures. And I think it's very important whenever you're writing tests to do your best to use the same repo to contain your test code as your API code. So if you're writing an express app or express API that's in Node, I think it's very important to try looking at ways to keep that code there. That makes it easier for you to locally on your machine as you're testing run the API and your test code. It makes it where you can give a developer that's writing a new feature the ability to run tests locally before they try even making a pr. And then it just simplifies your setup when you're trying to do CI. If you're in GitHub or Jenkins, you don't have to pull down a test repo to run the test against the API. So those are kind of the four big goals that I try to incorporate and to walk through. How we do this is publicly available just so if you want to go to the GitHub link, I'll show you. Share this repo in a bit, but you can get your own API key and actually use this for free. I didn't want to do something where you had to do a lot of work to run it locally or you were limited. So weather API is Free, you just sign in and you need an API key because they don't want you to DDoS their free tier. Yeah. So we have two endpoints that I'm going to write tests for and you can look at in here. One just getting the current weather for a location and the other one you pat you very similar, but you add in the number of days that you want and you get a forecast for that number of days. So a general strategy for testing an API I think is when there's some sort of validation authentication process to one you want to make sure that you can't hit the API without a key or token and make sure that you get the right kind of error message. And then once you have the key, start going down the both the happy path of you get all the data, you have a valid location. In our case, you get the right kind of response or you have a bad location. Does it handle that gracefully in the way that you document that you would expect it to do? So trying to go through that scenario with each of these three tools, I will start with Postman. So I think probably everyone here is familiar with Postman, but Postman allows you to write and make requests. I'm talking very heavily on rest APIs today, but you can do other things as well. But you can make tests, you can make a get request, write out the query parameters or a post. You can have your post body and see if the API is responding the way you want. But they also have a tab in there that we'll see in a second where you can write test and then Newman is something probably doesn't get the same buzz as Postman, but it's made by the same company and it's really just a tool to let you run their collections either on your own machine via your CLI or you can they give you a snippet of code to put and you can run it as a node app. And that's what we'll do today. So kind of seeing those steps, Postman go into the app or your VS code plugin, whatever you prefer. You'll write your test, validate that everything's working the way you expect, and then you save those tests that you write in what they call a collection and you export it in a JSON format. And then you need to set up Newman to actually run that file and also make sure that you have the variables that you need. Kind of looking at the app, I've already set this up and we have our two endpoints is how I put our collections together. And then the kind of three scenarios that I talked about. One is the happy path and then two different sad paths. One where I have bad location and one where I have a bad API key. So when you want to use Postman, probably most of us have done something where you send an API request you can set as variables your URL, your location and the API key. And for this, if you go to the Weather API and look at their docs, you can see in the docs the API key is actually just a query parameter here that you include. And when you make the request you can. You'll get back a object with your location and a lot of keys. And not everyone realizes this, but they have a scripts tag. Now it used to be I think a specifically called test, but you can write These scripts in JavaScript that will run either before or after your request. So this is how you would actually write a test with Postman, Postman walking through here for a happy path. The things that I want said I wanted to test and so I want to make sure the Response is a 200. I wanted to see the location was included in the response and that the current weather is actually returned with all those expected keys the way you would do that with Postman. They include let you import ajv, which is a schema validator. And I can write out all those keys that I want that what the schema looks like for the this response. So you can see like the region, I go the name, region, country. These are all things that should be in the response. And I can make sure that it really is a string that comes back. I don't really care that it's specifically this in my test, just that it is a string. So I try to be flexible on a lot of these because local time, the actual weather that in this API would be changing each time. And I don't need to seed everything. I can test it elsewhere, hopefully at a lower level that that was actually working. And then for your SAD pass you'd write very similar paths that they'll be a little bit more simple because should just get back in error. And in that error object you'll have a code and a message and I can write tests that. Oh, you can write tests that have the response of 400 and then actually test the error object to have the code that the Weather API defines and the message that they say that they'll send back. So forecasts would be very similar. You go through the same process and once you've written out these to export them, you have a In that dropdown, the actual option to export and save. And so once you save that, you put it in your repo. So this APIs or this repo is public. You can go look at it later, but I'll pull it up right now. And you can see I've created folders for each of these tools. And so for Newman is what I called it. I saved those two collection files and then I wrote this one script called Postman and zoom in even more. All of this from where this first line, line one, and then these Newman run statements are pretty much copied straight from the Postman doc you don't they make it pretty straightforward for running your collection. You just need to tell it the path of where to find the JSON collection file.

Steve, I know Postman introduced a lot of paid options over the free ones. Is this all free that you're showing right here?

Yes, everything I'm showing right now is freed. Once you get to where you want to start sharing the collections and everyone being able to run it that same file locally or in their own app, it starts getting different. But the runner itself you can run for free. So it's essentially parsing these JSON files and running it on your machine just like they do for you.

Perfect.

So two global variables I set up were the an API key and the API URL. And this key one is your big one that you want to make sure you never forget. Don't put your key in your repo. So the way to do that is to set it as an environment variable. You can put a env file and in my repo I have a sample and it's just pulling in these key value pairs in API URL you don't have to do. But I feel like it's a good practice to do because a lot of times you'll want to set up the ability to run like a dev environment and then a test environment and then like a production environment. The key you don't want to put out there publicly because then anyone can use your secret password. So to import it for your test, you just do this process env API key and it'll save it. And when Newman is running your collection, it knows the idea of global variables and it just automatically pulls it in because you said all the environment variables are called global vars. So to run is a it. You see the Postman weather test js so you can just run it with node. And to make things easy, I put in my package JSON a script. So going to try switching to a Different tab or a different app right now. All right, so we'll see what happens when you run that. And just writing the script you can see you're getting some output and probably about six seconds it ran everything and you get a list of all the tests that ran and green check marks that it passed and Postman and is nice enough to give you like some response times. You don't get the nice debugging experience that you can with the next tool that we've had, but it is fairly straightforward to see. Okay, if I had instead written my test and I had said like my expected response for one of these, let me in the background here change my 200 to saying I expect it to be 250 which doesn't exist. You'll see a failure. It didn't get that the response that it should. So you're able to start debugging and seeing what failed, but you have to manually go in and call the API and start debugging. You don't have any idea of a trace or a saved response. So pain points is before we jump to the next tool. So there's no built in retry logic with Postman. If it fails and you get that 200 or 400 response, there's no built in way where you can just call it again. And sometimes that's good. You want it to fail right away. I working in a space where we use a lot of serverless endpoints. When you deploy a new deployment, a lot of times your lambdas will be cold and need to be hit once or twice before it warms up and starts working. And so if you don't manually write in that each time you need it to happen, it's not going to retry for you. And when you're doing like test deployments, you need to do that. The other thing is it's a little bit hard to read. You're looking at a JSON file with the test. It doesn't have the greatest developer experience because if you need to update that test and if you're using a free version, you essentially have to save the JSON from your GitHub repo in your Postman app, start editing it and then upload it again, which that's prone to errors. And then you got to go up to the There are ways to sync with the paid version, but not always do you work with a company that wants to pay for those those features. And then the third one is that it's not universal. So it works great when everyone uses the same tool but sometimes you're working with developers that are used to something else and they don't want to use Postman to look at test. And then when they're used to their own workflow, that's maybe Paw or maybe some of these the open source tools. So overall you can do everything to test this API with Postman, but there are some things to think about. And so now we'll go over to Playwright. So didn't do a polling question, but I'm going to guess everyone here has heard of Playwright. As far as going end to end testing, touching the API or touching the front end and picking selectors and clicking through that, it does a great job of doing that. It provides a really good tracing mechanism. So if your test fails, you can see the screenshots of what happens, but you also get the network response calls. And that kind of goes into their ability to test the API because they had this idea of the request context. So not only can you use the request to configure things for your test environment before the Playwright test run, you can actually test the root API responses either in the UI based end to end test in conjunction or what I'm doing today is just install Playwright without all the browsers so it's a little lighter weight and just make API calls. So if you go their docs they have full on explanation of how to do this. So going back to that repo for Playwright, I have its own folder as well. I like the idea of setting up kind of a BDD syntax so that given when then without like getting very pedantic about being bdd but for our application I'm going to collapse these so you can see the flow of I have the idea of given an authenticated user and given a bad API key and then when I have I'm given an authenticated user, I want to do when we invoke the API with and this is going to be the happy path and then the sad path is when we have an invalid thing. So those when statements are actually just making the API call. And this is using that request context. And for our API I just need to pass in some query. They're all get requests. So you can see it's a request get with a specific URL that has some query parameters. And if you're not used to the typescript world, these parameters are things that I say have to be a string, but they're optional because of my test. I want to be able to test what happens if someone doesn't pass the location or what Happens if someone doesn't pass the API key. So even though they in the actual API, these things are required in the testing setup, I want it to be optional. So when I start writing test, because I've broken it into that given Win syntax, I can make it a little bit easier to read of. So when we make that happy path, it should return a 200 response. And so I just pass in the request context the location that I'm testing, which I just defined up here as a city, and then that I'm an authenticated user. And because of this app only being API token, that doesn't change given it's really just reading ENV very much like you did in Postman. If you have a different API setup where you're hitting a token endpoint and asking for a temporary token here, I would make that API call and define the token and save the token and then pass it here. But since our API key doesn't change from day to day, then I just have to save it as a variable. And so once you make that request, I can start looking at the status and response to be a 200. And here I make sure it's defined if I want to check that the locations in the response I just put the portion of the of the body of the response to make sure that it is what I expect it to be. And then for keys here I did not use ajv, which you can't. It is an NPM package you can install and do exactly like Postman. But I thought it was helpful to see like you can really just compare all the manually do a very similar check of making sure that each of those key values match up with the name and type of value that you wanted. And so.

You'Ll see the sad path would be similar that now I just want the Response to be a 400 and the error object that that sends back to have the right code and the right message. And then just like for the Postman and Newman's setup, you can write Playwright test as a script. And I like abbreviating because it starts getting long to type all these. All right, so when you run your playwright test in your terminal, you can see it runs pretty fast because one nice thing about switching over to Playwright is it can run several things concurrently. So it's able to make several API tests at the same time and be a little bit faster. So I think we were six or seven seconds in this time we're closer to three. The other thing is you get a report. So if you run that command I will, you'll see that it opens and then I will jump back to my browser here. And if you haven't used Playwright before, this is what a Playwright report looks like. You, you can jump and you can see a list of all the tests in each of the test files I had the current and the forecast test. And if I click on a specific test, I'm able to see all the steps and checks that it's doing and then I can open up a trace. And since all I'm doing is calling a single API once, there's not a lot to look at, but it does save the exact API call that it makes. I can see the, the URL, the with all the query parameters right there and I can look at the response header and the body. So if for some reason this fails, I, I have documentation of why it fails. And beyond the assertions that I'm making, I can actually look, ask a developer to look with me and see like, hey, this is what I'm getting back. You know, what would have caused this failure, caused this to happen. So the developer experience starts getting a lot better when you have Playwright. And if you have a lot of tests to run, being able to run in parallel is very nice. Kind of the pain points one is like, it's a new tool that's unfamiliar to backend developers. So if you're not working a full Stack team or people that have ever tried Playwright, it's just one more tool to teach and onboard everyone with how to look at those traces. Where do you get it, how do you navigate it, how do you write tests with this tool? And it's an extra dependency. So sometimes you do have to go through security or your team has to get approval to go to use Playwright because you're in healthcare, you're in banking situation where they need to know everything that's being installed, make sure there's no, no risk for them. So sometimes it's a lot of extra work just to use something like Playwright, even though it is very popular. And then the kind of, the third point is like, it might be something that is duplicating what the developers are already using. So that brings me to Jest. So Jest, if you aren't familiar, is really well known for, in the JavaScript and TypeScript world for unit testing. So most likely if you're working on a app that is in Node or React, it's going to have tests using Jest. In fact, Playwright's using a lot of Jest under the hood. But what's Less common is what I'm going to demo of just pairing up with API testing. And to do that you need to use a Node tool or library to actually make the API calls and let Jest do the assertion side of the test. So I'm demoing with Axios. There are some other options, but Axios you can go to their website and look at the docs, but it's a very simple format of how to make API requests either from a browser or from Node. So front end and back end apps can use it. And when we use this we can actually write tests very similar to the other two methods that I demoed. So jumping to the code now, I see a folder called Jest and in that folder you will see it looks almost identical to the player I want to have. Steps actually should be could be identical if I was cleaner when I set this up. But given you're just pulling in that API key and then you have a win and you have two different test cases. So when is really the bread and butter of making the Jest and Axios work? To start off, it is very similar. I'm going to have my two win scenarios of make calling the current weather API and the forecast weather API. I once again need to pass a string for the location and the API key. The difference gets into under the hood. Axios isn't giving this header, so you need to make sure you pass this header. So really.

These three lines are going to be just like in Playwright. I'm really just passing a API request get API request. But to handle an error you do need to do a little configuration. So I have this method called Make HTTP request and it's just taking the method URL string. And this is extraneous. But if you were to test some post endpoints, you'd also need data. So when you make the request with Axios, you're going to get back a response and that if that promise turns into an error, if you don't do this step, the test is going to fail. Which is great. If you're expecting a 200 and you get a 400 back, you want an error and then that that works as expected. But when you start testing the SAD pass the you need to catch that error and you want to look at did you get the error body that you expected? Did you get the error code that you expected? And so I do a little map, do a little mapping here. But because you do this little extra work, the actual test itself ends up looking just like your Playwright test with the exception instead of test, dot, describe, you just say describe so you can see I go response, body, response location, do all those same assertions. And then because I did that mapping for the error cases, I can once again catch the error that Axios API would return and then start investigating it without ending the test and not seeing a screen. So once you've made that mapping, you can once again run a script. And so with Jest, the test test Jest is how I set it up of just running the Jest command. So second to last time of jumping windows here, I will go here and kill my Playwright report server and see run test Jest and you'll see that it goes through. The tests are going to take a little longer than Playwright, probably very similar to running the Postman collections. But you get a report output that looks a lot like Postman. You don't get like a broken down average of all the response times, but you do see how long each test takes and then the total test runtime, five and a half seconds. So now with doing this, we've run the same set of tests of I guess 14 total tests, three different ways, and you get a little bit different output for each one. Some are better than others, there's a little bit different work, but we're able to test that same API with all three tools. The pain points of doing it with Jest and Axios's one testing using this combo is less common. So you're not going to see as much support if you go on the stack overflow or whatever. If you have problems specifically with your test. The you do have a lot of because Jest and Axios are both very common. If you have specific issues with either of those, you do have support. The other thing is it's a little unfamiliar to other QAs. So it's great if you're the one person working on that and you get to set it up and hand it off to a developer, because developers tend to be more familiar with Jest. But if you need to onboard other people that aren't familiar with this tool, with the setup or with Jest, then you have a lot of training to do so it probably wouldn't make sense in that scenario. And then the third thing is that configuration you need to understand like Axios is designed to fail and throw an error message if the you don't get a happy 200 response or 201. So you need to understand to set up those helper functions that'll catch your errors for you. But in doing, if you understand what's going on under the hood. It is possible to use this combo and not have to go through the the trials of getting Playwright or Postman approved or worrying about licensing with a paid account on Postman. So some scenarios it does work for your team. So that is the demo I have. I know it's a lot of code pretty fast to try covering three tools in a presentation, but I'm trying to leave some time here to ask questions. The repo is public and you can reach out to me later too if you have play with it. It doesn't work.

Thank you Steven for your automation awesomeness Links of everything of value we covered in this session. Head on over to testguild.com a513 and while you're there, make sure to check out our awesome sponsor of this episode, BrowserStack and their Automate Turbo Scale solution to help you remove the frustration and time of setting up, scaling and maintaining your functional automation. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed in creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to Collaborate visit test guild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With hoots and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wild fire through the land Guiding tester showing the secrets to behold.