API Testing using Postman with Kristin Jackvony

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we have a special guest. We have Kristen Giacvoni talking all about API testing using Postmon. Kristen's been on my to do list or my want list for a while. So awesome to have her on the show. She's a principal engineer for quality at Paylocity. In her role, she's charged with helping teams develop good quality practices for releasing software that is reliable, secure and maintainable. In addition to all her work at Paylocity, she also writes an awesome blog called Think Like a Tester and has a LinkedIn learning course about Postmon and is hard at work writing a book about software testing, which I'm really excited to hear more about. And hopefully we'll have her on the show to talk all about that as well. But you don't want to miss this episode. If you want to learn anything about API testing or automation in general, or how to succeed with it, listen up. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Kristen, welcome to the show.

Thanks Joe. I am thrilled to be here.

Awesome. It's awesome to have you on the show. Like I said, really excited to have you. Before we get into it though, sometimes I botch the bio. So is there anything in your bio that I missed that you want the guild to know more about?

No, that was great. I loved it.

Cool. Cool. All right, so as I said, I've been following you for a while. I think we met at my first automation guild actually in 2017. You're really really active in the community and since then you've really, really grown and seem to be really expanding in all directions in automation. So the first thing I to focus in on is API testing. Though it seems like a niche that you seem to be really into. So why of all the things you could do automation, you probably do do automation, why focus in on automation for API testing?

Okay, well I am just a huge fan of API testing and I am a huge fan of Postman and the way that I got into it. Well, I got into test automation in the Same way that I think a lot of people do. They get started as a manual tester and then they wind up in a role where they're told, oh, you need to have some automation. And so of course, the first thing you try to do is you try to automate based on what you usually see. So, like, you're logging into an application, you're clicking buttons, and so you create that automation that will do that very same thing. It was probably about 2015, I think, that one of my developers that I was working with at the time, this was a previous company, introduced me to the concept of APIs and said, oh, you know, you could do some testing with these APIs. And so I was doing some testing. I can't remember, it might have been with SOAP UI at the time. It was a while ago and I saw, wow, it's really fast to assert on these things. And then you don't have to worry about the whole, let's get the browser driver working and let's figure out how to log in and all of that. So I got really interested in it then. And then when I started working at Paylocity, developer introduced me to Postman and I thought, where has this been all my life? Because not only is Postman very easy to use, but I think that it's also a really good way to sort of visualize what it is that your APIs are doing. And so I was just so excited about it. And one of the reasons why I'm so passionate about it is because you can test things so very quickly. And a perfect example of that is I have an application that I've recently written that's designed to sort of help testers practice their UI automation and their API automation. And I've got some UI automated tests for it and some Postman API tests for it. I can do like 650 assertions on my API in about a minute and a half. And there's just. Even the best UI automators aren't going to be able to say that. So that's why I'm a huge fan.

Awesome. I guess what I'm curious about is I also know you're a huge fan of Cypress. I actually saw you on Future of Testing, Front End Test Fest. Actually, not Future of Testing, you were talking about Cypress. And so. So I think Cypress has the ability to test APIs as well. So, like, why Postmon then? I know you give a lot of good reasons why you like Postmon, but do you ever use Cypress for API testing? Or when would you choose One over the other.

That's a great question. And at paylocity we do have people that are doing, we have teams that are doing API testing through Cypress. And I think it really depends on the team and it depends on what both the testers and the developers want on that team. So if there's a team, let's say, that has a very heavy UI application and they just want to make some API calls in order to do some setup for their UI tests, then Cypress might be the way to go with that. If it's a team that is very, very API heavy and doesn't really have a whole lot of ui, then I would say Postman's the way to go. But again, it matters to the team. It matters what people are saying. I am a big fan of getting the developers participate in test automation. And so if the developers say, gosh, we'd really rather do this in Cypress because we'd rather be able to look in Visual Studio code and create our tests, then that's great. I mean, that's what you should do. But if the developers are comfortable with Postman and they're sharing collections, the testers and the developers together, and everybody knows how they work, then that's a really great alternative.

Awesome. So if a developer wanted to use Postmon, is there a language they use or is it more of a drop down type of interface? How do you create Postman scripts?

Well, let's see. To actually create a Postman request, put in the URL that you want to hit. Usually you would have a body of the request. Like let's say if you're doing a post that might require a body. And so that is often in JSON format, I mean, not always. Postman actually supports a lot of things, a lot more than our company is doing. So that's what you do. And then of course there's also the header section. So you would set up your headers like if you had an authentication token that you wanted to pass in, then to assert on the results that you're getting. That's done in a test window that is using JavaScript. And postman actually has, I don't know if you'd call it like a little hook. It's like a Postman hook where you're making an assertion, sort of using their special assertion language. But it's essentially JavaScript. It's like JavaScript with a little wrapper. So for anyone who knows JavaScript, it's pretty easy to put together.

Nice. So I know one thing people struggle with with API testing in general is not what they struggle with. But I see is they do a lot of positive testing. For some reason they get kind of tripped up on negative testing. How hard is it to do negative type testing with the Postman and API testing in general?

It's easy. It's really, really easy to do negative testing. So like some perfect examples would be the kinds of, like let's say you're doing a post and you're entering in a lot of information. Like let's say it was for a contact. So you had first name, last name, whatever, and you, you wanted to make sure that if you passed in a first name that was too long that you would get a 400 response with an appropriate error. And that's just super easy to do. I mean, all you would need to do is you would just in your body of your post request, you would put in the first name that was too long and then what you would assert on is you would assert on the response code that it was going to be a 400 instead of a 200 and then you would assert on whatever the error message is. And Postman has a, this really cool thing which I use all the time because I'm not really good at remembering code syntax that they've got this little thing called snippets. And so when you go to that test section and you say, oh, I want to do a response code assertion, but I don't remember what the syntax is for that they've got these little snippets on the right and one will be like response code is 200 and you can just click on that and it just puts that JavaScript into the window and then you can just make the adjustments as needed. So now you're asserting on a 400 instead of a 200 or whatever. So it's very easy for people like me that forget syntax.

That's cool. Can you create custom snippets?

Oh, that's a great question. I don't think you can, but yeah, that'd be a great feature request.

Nice. How does this work in a team though? Is it like a closed off application where you're doing your test and someone else is doing their other test? And how do they come together when you're working on a large team?

Well, Postman has the ability to share collections and it all depends on what level of Postman you use. There is a free level of Postman and you can actually even, I believe, share the free level with maybe one other person. I can't remember now, but if you have the Pro version, you can, as a company, you can share all of your collections. And so what we have at paylocity, because we've got a lot of teams, every team has their own workspace. And so in each workspace, all of the team members who are working with that workspace are invited. They can all collaborate on it. And so. But then there's also a personal workspace too. Everybody's got a personal. Or you can have more than one personal workspace. So if there's something that you want to do yourself and you just want to mess with stuff, or you've got a personal project that you're working on for a presentation or something like that, you can do that on your own. But then that shared workspace that those shared collections, everybody can work on together.

Very cool. So let's go back a little bit, take it up a high level. Could you walk me through the anatomy of an API test? Maybe I'm oversimplifying it. You make a request, you get a response back. That to me is a, that's an API test. And then you do assertions. What am I missing? How does that work?

Well, there are so many things that can go wrong with an API test. I mean, you know, just here's a simple one. And a lot of people don't know this. When you do an API request, if there's certain levels of responses, you can get response codes. There's a 400 level response code and there's a 500 level response code. A 500 level response code should really only be used if something is like really, really wrong. Like you can't reach the server at all. Whereas a 400 level response code is for things like you don't have permission to do that, or you've sent in, like my example earlier, you've sent in that first name field that's too long. So there's a whole host of 400 level response codes that you can get back. A lot of times testers will be testing APIs and they'll do something that they know they're not, that the user isn't supposed to be able to do, but they'll get a 500 response code back. And that's actually wrong. So that's something that testers really should know, being acquainted with what the response codes are and why they would be used. Another good example is the difference between sort of authentication and authorization. There are some things when you make an API request that the person is not authenticated, so they should probably get a 401 response whereas if the person was authenticated but they didn't have permission to do the thing that they were trying to do, that would be a 403 response. So even just something as simple as response level codes, there's a whole lot of stuff that you can mine there. Then you get into things like headers, what kind of headers are you sending in and what kind of headers are you expecting back? There are security headers sometimes that APIs have, and you'll want to make sure that, for example, at Paylocity, our APIs, most of them are all locked and private. They're not public. People can't just hit them on a whim. And so we have API keys, so when we make a call, we need to include the appropriate API key in order to have that call go through. So it's very, very important to test that you can't get a response to that call if you don't have the API key or if you have the wrong API key. So that's another level. And then just things like I was mentioning before, when you're doing posts, what kinds of restrictions do you have on the elements that you're trying to send in? Like here's something that's supposed to be an int and you're sending it in as a string. What kind of response do you get for that? So just all the same kinds of things that you would think about when you were testing, like a form and a UI contact list or something like that. Like, you know, you think about all the different things that can go wrong and then you just make them in your API tests.

Absolutely. And I guess though, like you said, within the UI test, developers probably know that people are going to be using the ui. They can see it, they can interact with it. I would assume sometimes APIs may be something used by the application behind the scenes you need to test, but not necessarily customer facing. So they may forget some of these things that you're testing for. So I think it's a great point you said about security. I know how common it is. You also look at things that may be a clear text, make sure no one's actually passing a password because they think, well, no one's going to see it's an API test. Why do I care about that?

Yeah, I'm really glad that you brought that up because, I mean, here's the thing. Hackers and malicious users know how to look in the network tab of Dev tools when requests are made and they can see what API requests are being made, they can go ahead and they can lift any information they can get from there. And they can use a tool like Fiddler or like Burp Suite and they can replay those requests. So it's so important that they are locked down so that malicious users can't do that.

Absolutely. Another thing I was thinking about is you mentioned how even a single request is complicated because there's all these other things that could happen. I would assume this would be common. You have an application made of microservices and a workflow is made up of maybe multiple API calls that are chained together. Do you have that type of testing as well? APIs, kind of chaining them together, passing values between one to the other to do an actual workflow type test as well?

Yes, definitely. And that's another really awesome feature of Postman is that they allow you to create variables. They're called environment variables. And so you can have just an environment variable group for like dev environment for your prod environment. And it's really easy to create them and it's really easy to use those variables in your next request. And I've seen some pretty neat things that people can do. One co worker of mine had come up, he came up with this amazing script. It was all JavaScript where basically every kind of request they were going to make, he went through and looped through and checked all of the tokens to make sure that they were there and whatever token wasn't there, made calls to get those tokens and set all of the environment variables. And so as soon as that pre request script was finished, he had every token he needed to do everything in his test suite. And then I've also seen this was at a Postman conference I attended. Really interesting, sort of a conditional set of requests where the requests were chained together. But once you got to a certain point, it was kind of like if this, then you skip over down two tests to the next one. So it's really, it's an extremely flexible tool, which is another reason I love it so much.

Now, earlier you mentioned something about testing if it's an int or if it's a string. How do you know that? How do you know what to test? When it's an API, do you normally get requirements or is it like a Swagger file or a schema?

Yeah, ideally. Ideally you would get a schema from your developer and that could either be like a Swagger schema or Postman also has the ability to use schemas. So yeah, hopefully you get that information from the developer where they're saying you know, here's the information that I am expecting and here's the kind of form I expect it to be in. Here are the length limits, or, you know, is it an int or is it a float or whatever. I think sometimes teams can forget to do that. The developers might make everything work and then forget to communicate that information and so in. And that's one of the reasons why I really encourage developers to get involved with API testing. And one of my favorite things is when a developer says to the tester, we've created a new endpoint for our API and here are the acceptance criteria, which you've already seen, but here's also, I made a little Postman collection for you so that you could see what the requests look like. That makes me so happy because then the tester can jump in immediately, do that happy path testing because it's already right there, and then start thinking about all the different ways they could break it.

That's very cool. Now, we touched on kind of security testing with API testing. I know you do all types of testing. How about performance testing? Do you ever get any type of performance data out of these API calls and track that over time to see if something's less performant than it was earlier?

Yeah, you can actually do that in Postman as well. For every request, they record the response time. And so you can actually make assertions that say, I want my request to come back in, let's say, no more than 3,000 milliseconds. And so then if it doesn't come back in that time, you're going to get a failed test. And I actually did that as, funny story, with my application that I created, I wanted to have those response time assertions. But the application I created is in Heroku. And Heroku, I guess, goes to sleep if not enough people are using it. And so then that first call takes like 30 seconds. And so my Postman automated tests I was setting up were failing every day because of that first call. So. So, yeah, so I. I changed my tests a bit.

So that's interesting. Developing your own application almost can give you empathy for a developer rather than going, these people are idiots. Why do they do that? Like, how would you know that unless you actually developed your own application? Right?

Yeah, yeah, it was so humbling. The thing I think that humbled me the most was just the validation. You know, I remember when I first started testing, you know, I would test something like a date field or something like that, and I'd find these bu and the developer would fix them and then I'd find five more. And I'm like, why doesn't this guy know what he's doing? Well, now that I've done that, now I see just how hard it is. So yeah, it's a real eye opener. I recommend it to anyone, actually.

That's so cool. So I read on your blog you did write this application. Now, is this application in conjunction for eventually your book? Is that why you wrote this? Because eventually you're going to write a book and you're going to use that as the backend for your examples?

Yes, that's what I did because I thought so many times, there are example applications that people can use to practice UI automation, and there are example applications that people can use to practice working with APIs as well. But sometimes even those are a little bit too hard, you know, like maybe you want to, let's say, get started using Cypress and you just want to make like a simple end to end UI test. But the test application you're working with has tried to be sneaky and they're making it really hard for you to get all your elements and you're like, I just want to get one test working. So. So that was the idea that I was going to create just a very simple UI and a very simple API. And then throughout the book, the book's about a whole bunch of stuff, but the book does talk about UI testing, UI automation, and it talks a lot about APIs. And I wanted to have something where the reader could say, oh, I think I understand about this post request, but I would like to try setting one up myself in Postman and seeing if I can actually get those results. So, yeah, I'm really excited about that.

It's a great idea. Not because what you mentioned though. I wrote a book in 2014, 2015. I used public APIs around the web at the time. And so even now I get people saying, your book's out of date, I can't access this endpoint anymore. It's because the endpoint doesn't exist. So what you're doing is awesome because that's going to save you a lot of time in the long run because you have your own control of the application. So good for folks to know if thinking about writing their own books. For sure.

Definitely.

Cool. So Kristen, how do you get all this information then? Where do you store it? Is this integrated in CI CD like any other application or any other test tool? The results say I'm using Postman and I have a bunch of test scripts and I'm running Tests, where are the results stored? Can I store them in CI cd? Does it integrate with Jenkins, that type of deal?

So you already mentioned, I think Newman. So Newman is the. It's an NPM package that allows you to run Postman tests from the command line. And so what we do at Paylocity, so for our build system we're using TeamCity, and for deployments we're using Octopus. And so what we did was we had one of our very clever developers created a PowerShell script that actually calls Newman and calls Newman with all of the different variables that we would want to pass in to make that request to run those tests. So we have things set up in OCTOPUS where when we go to do a deployment, Octopus will say, okay, I see that we're deploying to production and we want to run some smoke tests and Octopus knows what all of our API key values are and everything. So we're going to pass those into the PowerShell script, which is then going to run Newman, which is then going to return our results. And then we also have tests that we run nightly in TeamCity, where it's just a TeamCity job. It runs the PowerShell script. The PowerShell script says, oh, does this build machine have Newman on it? If it doesn't, it installs it, it runs Newman, and then it returns our pass fail results, depending on how things go.

Actually, I've never heard of Octopus before. It's a CI system like Jenkins. What is it again?

Yeah, it's like Jenkins. It deploys software. Yeah, I'm not sure we're a Net shop here, so I don't know if it works specifically with Net applications, but that's what we use. And yeah, it's just like Jenkins.

Nice. So I'm trying to see what we may have missed. You actually have, like I said, you have a course on LinkedIn called Postman Essential Training. It has over a thousand members, or actually 18,794 learners. So, yes, you go over all this in detail. You go over the introduction, getting started with Postman, Easy, Postman assertions, Advanced Postman assertions, running your test collections and automating your collections. Could you just tell me again what a collection is in Postman?

Yeah, so a collection is a group of requests. The way kind of the breakdown is sort of got your workspace. A workspace is where you can have all of your collections and you share those with your team. A collection is just a whole group of Postman or API requests that you're making. And another really cool thing, that Postman does that I love is you can organize all of your requests into folders, and you can nest those folders and you can go crazy deep with those folders. I once tried, just for fun, to see how deep I could go, and I think I wound up getting, like, I don't know, five or six folders deep before I said, okay, I'm tired of trying to see how deep this goes. But that's really nice because it's such a great way to organize all of your requests. And so you've got your collections, your folders in the collections, and then finally, you know, the last level would be your request level. And then separate from that are your environments. Your environments are just basically collections of variables, like what's the variable and what's the value? And so you can have an environment for all of your QA environment tests, and you can have a separate environment for all your production tests and so on.

Awesome. So I haven't authored a LinkedIn course. Do you actually get questions from people that take your training? And if so, are there any common questions or misconceptions people have been running into when they're trying Postman or API testing in general, say?

The questions probably come in, like, two different categories. One is a lot about Jenkins. And I was kind of actually kicking myself after I created the course because, I mean, it's a CICD platform. I mean, it's a lot of steps to set up and different people. You know, if they're taking the course on their company computer, they might not have Jenkins or they might not be able to set up Jenkins. So I've gotten a lot of questions about Jenkins, and if I could do it again, I'd say, just leave that off. But I think another one is I get questions about, hey, I've got this request and it's not working for me, or my assertion isn't working right. And why is that? And I just have them paste in the code that they're using, and it's usually something very simple, like they forgot a bracket or they misspelled something. And I think that's actually something that's a little different from. Well, definitely different from manual testing. But I think similar to when you're writing code, is that just something tiny, like you didn't capitalize your variable, but your variable name is actually capitalized. Those tiny little things can actually make a difference. So that's a learning curve sometimes.

So, Kristen, how far along are you with your book? Think like a tester.

I am currently on the second draft of My book, I'm hoping that I will be finished by the end of June. And then in the July, August, September, I'm going to start sending out sample chapters to people who have subscribed to my mailing list and starting to get feedback. I'm really hoping that I can get it published by the end of the year.

Nice. How can people get on your email list?

Then to get that they can go to my website which is, oh, and of course I closed the link, but it's thinking Tester.com is my website and if you go over to the free resources, there's like an email sign up link right there.

Awesome. Okay, Kristen, before we go, is there one piece of actual advice you can give to someone to help them with their API automation postman testing efforts and what's the best way to find or contact you?

Okay. Well, I would say one of the most important things to do in test automation, whether it's your UI automation, your API automation, is get your developers involved. Try not to live in a vacuum. Tell them what you're doing, ask for their advice, ask for their suggestions, teach them how to do what you do. I think a lot of testers are worried about that because they're afraid they're going to work themselves out of a job. But they're not because developers are never going to be passionate about testing like you are passionate about testing. So get them involved. They will have all kinds of great ideas like, oh, you know what, I can make a little test harness for this that will make it easier for you to do your automation. Or you know what, I think we should organize our variables this way instead. It'll be easier for everybody. And that just makes the team just gel and just work so much faster. And you can contact me, let's see. Well, you can find me at my website, thinking tester.com you can also find me on LinkedIn. I post there frequently, I'm on Twitter, I post there less frequently. So you just look for my name. Kristin Jackvone.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a354 and while you're there, make sure to click on the Try for free today link under the exclusive sponsor section to learn all about Sauce Labs awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them, so that's it for this episode of the Test Guild Automation Podcast, I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.