Contract Testing with Lewis Prescott

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Tuning in for a special treat. Today we'll be talking with Louis Prescott all about contract testing and API testing in general. If you don't know, Louis is a QA lead. He's worked at multiple companies like Cancer Research UK and ASOS previously. He also has a course on Test Automation University, which you should definitely check out, which I'll have a link for in the show notes after this episode. And Lewis also specializes in multiple things, but his hot topic seems to talk about a lot is API testing, API contract testing for microservices, and a whole bunch more. If you want to stay on the cutting edge of how to do API testing, you don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSourceLabs and click on the exclusive sponsor section to try it for free today. Check it out.

Hey Louis, welcome to the Guild.

Hey, thanks so much for having me.

Awesome to have you. Before I get into it, is there anything I missed in your bio that you want the Guild to know more about?

So I have a podcast myself about how to get started with contract testing, talk to people in the industry and the community. So yeah, check that out. And my website, pacman.co.uk, has all my blog, my videos, all that good stuff on there.

And what's the name of the podcast, Louis?

So it's how to get started with API contract testing.

And if someone googled that in like Spotify or something would come right up, right?

Yeah.

Sweet, sweet. And we'll have a link to that also in the show notes, so no worries there. All right, cool. So Louis, I've been following you for a while and everywhere I look you're on Udemy, you're on University of Test Automation University, you have a blog. It's all about API based type testing. So why are you so big on API testing?

Yeah, so it came from my time asos, really. I worked on a backend team there. We were in charge of the registration and login for the whole of the platform. So we had a lot of public facing APIs as well as internal APIs for our business. And yeah, that's where my interest really grew for it and got on from there. Really nice.

So I guess before we get into it, people hear contract testing a lot. They may not know what it is if they're doing more non API type testing. So at a high level, how would you explain what is contract testing?

Yeah, it's difficult to explain without diagrams and stuff, but on a podcast, the way I like to describe it, it's like flipping integration testing upside down. So if you think about the traditional way the that you would do an integration test where the API service makes a request, checks the response, contract testing brings another dynamic to it. So the web application or the mobile application will actually create the scenarios and then the API service will replay those scenarios on their end. So it's kind of adding another dimension to your API tests.

All right, so when would someone create contract testing? Would they have to have a base of APIs that they could leverage for contract testing? Is there a different, completely different way you would develop these type of tests?

Yeah, so there's kind of two ways that you can do it now. So there's consumer driven, so the consumer side of things, which is your web app, they will generate the contracts. Their side that's all mocked, that's all isolated. They don't interact with the service in any way. And then they publish those contracts to a broker which stores all the contracts and then the API service can run those tests locally. So you don't actually need anything to be set up. And you can do this on one endpoint, on 100 endpoints. So yeah, it's very nicely packaged so you can utilize it in however you like.

So if anyone's dealing with any type of APIs, would they all have contracts that they could then perform this type of testing against?

Yeah, exactly. So if you think of a contract like an API schema, but the contracts that you have are those behaviors that you're interacting with. So all those scenarios that are built into your API schema, like a bad request or a 404, all that kind of stuff you can replay in your contracts. And also the contracts will be a subset of that. Right. Because the web app will interact with the API in a different way, the mobile application will interact with it in a different way. So then you can see those behaviors inside these contracts. So yeah, it's kind of building upon that schema based testing and then going up a little.

Who creates the schema? Is it developers? And is there any way that someone goes, the developers don't Create a schema or do all APIs have a schema out of the box?

So you should have it out of the box. Right. If you're using a framework like NET Core has open API built in, generates the contract for you. Traditionally, if you're working in a development team, you would define those up front. You can use a tool to define them, or you can just put it in a wiki or in a document. They should be built as you're developing. Right. The difficulty of that is maintaining those and keeping those up to date. And that's where these contract tests can really come in and add value.

So these types of testing is just validating that the API is doing what it says it's doing.

So.

So if someone's consuming the API, they can trust in knowing that all the functionality that's supposed to be able to perform with that API is handled?

Yeah, pretty much. Right. So when you're interacting with an API, you want to make sure that the response is coming back as you expect it behaves in the right way, it throws the right error codes for the condition that you are testing against, and ultimately that's it. But APIs develop over time, I. E. You might need to change a field, you might need to change the type of field. There may be different versions of the API. So all those kind of things get very complicated very quickly. And that's where it makes it much easier to put it into a tool where that is all documented very nicely for you.

Cool. So because you have the schema and it has basically all the rules of your API, how much does it take to actually write a test? Do you just throw it in a tool and it knows all the values and just spit out a, a script for you? Is it that easy? Am I overthinking it?

So it should be that easy, Right.

But it's usually not. So PacFlow have just released a new feature which is called bi directional contracts, which allows you to plug in to your existing tests. Right. So if you're using Cypress or Service Worker, where you've already got mock set up, that can just generate those and they become your contracts. On the API side, you just use your existing schemas, as you say, just replay what is inside the schema against the contracts and then happy days. But in order to make it really work well together, you want to have the contracts as close to the code as possible. Right. To get that feedback as early as possible, you want to have it right next to your implementation. Cool.

So usually performs the contract testing, should it ideally Be a developer before they even release it. Because I'm sure your application probably because it's an API and it's a microservice, other microservices might rely on it. So it's not even consumer based necessarily. It could be in house based.

Yeah, yeah. So when you think about a consumer. Right. That could be anyone. Internal, External. But the it should be a team approach. And I think this is one of the big advantages of using this type of testing is that actually it generates conversations and it's not necessarily the tool that is the thing that's providing the most value. Right. It's the conversations that you're, you're coming that's coming out of it. So it's definitely the API team that will care about the results of this the most. And they should be wanting to find out how people are interacting with their APIs. But as it's described, it's consumer driven. So yeah, it's difficult to pin it down, but I would say it's a team approach.

So when I hear contract testing, I hear consumer and provider. Are there different types of tests running as if you're a provider or a consumer or is it just an end to end type thing that covers all both scenarios?

Yeah. So in a microservice environment. Right. It's mimicking that end to end behavior. And that's where you can really, can really reduce the amount of effort that you put into your end to end tests if you have these contract tests. Because you can clearly see how these behaviors are happening across the end to end estate without having to spin up an end to end environment.

Nice. So whenever I hear, once again, contract testing, I always hear the tool Pact. There are other tools you use or what do people normally use? You mentioned Cypress. If someone's using Cypress, can they also do contract testing with Cypress or Selenium?

Yeah, so usually I use pact. Pact is the tool that I have kind of adopted and has worked well for me. Java has it built into its Spring Cloud package, so that's native to that language. The benefit of PACT is it has support for most languages, but also again, it's just about the type of testing. So what I've done before is just stored the contracts in GitHub and then you can write some code which pulls down the contracts. So there's no need to necessarily use a tool. The benefits of the tools are that they have all of that comparison and all that kind of stuff built in for you. The new way of adopting contract testing is potentially using the bi directional contracts which PacFlow have just released, which allows you to adapt to your Cypress tests, for example, to allow you to do that. They're building adapters at the moment, so they're looking for contributors. So if someone's keen to get a Selenium one in there, then they'll be happy for someone. It's open source, so they can contribute as they want. But, yeah, there is ways that you can use the existing tests to adopt contract testing.

Nice. Now, when I think of microservice, also I think of dependencies. If you're testing one microservice in isolation in a contract test, I. Using mocks, it doesn't matter is because you're just. How do you test what's being returned to you if there's. If that contract relies on another contract? Do you ever have those type of relationships? A contract relying on another API that has another contract?

Yeah. So it's all isolated. Right. So you're only ever testing one integration. You're never testing a dependency. But there is a way to set up a start state, a beginning state, which is called a provider state. So you can set that up, right, to say, okay, I've got this dependency. So then when the provider replays the test, they can set up that dependency before the test runs. But the contract itself is always just between one set of integration.

So I've never done this type of testing myself, so it may seem like a stupid question you mentioned, Pat, but when I look at your courses, you have something like PACT using Postman, PACT using Pack js. So PACT is a tool that then you can then use with other tools. How does that work?

Yeah, so the thing about PACT is you're using it to generate the contracts and store them in the broker so that both sides can use it. So you can put open API specs into the broker, and then Postman can just call those and use those. The other reason why I use the broker is I pull down stubs from there so I can use those in my tests as well. And I can say, okay, use this stub, because I know this has been verified by the API service. I know that this is a good stub. It's not one I just generated and I'm using that in my tests.

Cool. Cool. All right. So we touched a little bit on the consumer. You also have a section in one of your courses called Provider Test. Why would that be broken out differently than Consumer Test?

Yeah, it's just. It's purely because of. They sit in different code bases, so different person is writing Those different tests. So that's where the separation comes in. Ultimately you can't have a provider test without consumer test. So one of the difficult things with contract testing is getting buy in from all, all of your stakeholders. Right. So if you're a provider and you want to run this type of test, then you need the consumer binding. So when I was working in an API team, I would reach out to the consumer teams and say, I've generated you these contracts. Are you happy with these? I'll use them inside my tests and then you guys can update them. But it's. Yeah, it's all about that buy in relationship. So that's where you need to separate it because it's two different services potentially, but it's all one and part of the same thing.

Nice. So it seems like you mentioned, I think you said these are more like integration tests so you can shift left earlier. So I guess if your team's starting to work on an API, even if it's not fully flushed out, I guess you can always create a new contract every time and just make sure that a reiteration that it's fulfilling the contract.

Absolutely, yeah. So you can version them. Right. So you can make sure that you're getting keeping on top of what the latest version is. And you can tag them as well. Right. So you can tag, this is production. So the consumer will always just be saying, okay, can I release to production or to an environment? Whereas the provider might be saying, okay, I'm going to release this beta version and I'm going to test that against the consumers. But the consumer will never touch that beta version until it's an actual live version.

Cool. And I could be wrong. I can envision this in a CI CD pipeline, like the first gate, the checking in. Like don't even put it into the code base because, you know, we know off the bat that contract is hosed. So I guess, would that be a standard user case?

Yeah, absolutely. Yeah. It really fits nicely into your CI pipeline if you use it properly. And again, about the buy in. Right. The value really comes when you've got all of your consumers bought in. You're not going to see the value straight away when you're just testing it against one consumer. Because ultimately that gated check isn't telling you the full story.

So what are some other benefits then you could give to why contract testing would be important?

Yeah, so one of the clear benefits is not having a dedicated environment because it all runs in isolation. You can spin it up when you want the reduced amount of integration tests and end to end tests that you need to have because you've got that confidence lower down. And for me, the greatest benefit, which I keep on coming back to, is the communication between teams and the communication within microservices is it really builds into that conversation about API design, because you can see, you can visualize, okay, all of these consumers are using the API in this way. Maybe there's a field you can remove from there, or maybe there's a field that can be moved to another API. So there's real deep conversations that you could have about API design through using this data.

So are you saying then that a developer could have made assumptions and so once you create that contract, it actually highlights. Well, is this really what it's supposed to do? Because it's actually in black and white now.

Exactly that, yeah. Assumptions, right. And that's where the description about flipping integration testing upside down is that the API team is usually making assumptions about, okay, I want this to be like this. And there's a concept called Postal's Law where you should be open in what you receive, but strict in what you give back. So that's what you should be doing when you're building your APIs. Right. The consumers should be open in how they use it, but the API should be strict in saying, okay, I can't accept this and that for this reason, but that should be an API concern, not a consumer concern.

Absolutely. So once again, could be off the wall here. A big trend I've been seeing is like security testing. Trying to get security testing earlier. Can you use contract testing to make sure that there's no kind of wonky things that it would accept that could cause an exploit like SQL Injection or something? Is that something that would be part of this type of phase of testing?

I don't know how you would build that in, to be honest. But that may just be my ignorance around security because I haven't really dug into that myself. But yeah, I don't know, I just.

Threw it out there. I could be completely off base myself. So if you're listening, let me know.

I mean, exactly. People might have some good ideas on that.

Cool. So one of your courses, you do mention Docker, you said you need someone needed to have some Docker knowledge to do one of the courses. Now is that just because of the way this course was set up or is when you're doing API testing with contract testing, usually Docker or Ktinas are involved.

So it's for hosting the contract store. So the pack broker, they Have a Docker implementation where you can spin up the PAC broker. But there's also a software as a service with pacflow. So whichever one you want to go down the use cases I see for both is if you're in a very locked down environment, you probably want to host your own. Right. And have that in your own infrastructure. Whereas software as a service for us, it's super easy to spin up and store our contracts there. And they have nice pricing model as well.

Cool. So we talk about pact, which seems like it's an essential tool that someone should need to know when they're getting into contract testing. I think in your Test Automation University you mentioned some other tools like Supertest and noc. So I guess what's Supertest and what's knock and how do they fit into? I don't know if that has anything to do with contract test or if that's just API testing in general.

Yeah, it's just API testing in general, but those packages, yeah. Offer nice features. So Supertest is just a layer on top of the super agent which allows you to make HTTP requests. But it's really nice you can chain your assertions straight to the request. So there's no like getting the response, waiting all this async await, all that kind of madness. That's all built into the package, which is really nice. And then NOC is a mocking tool, so it allows you to mock your requests. So if you've got any of those external dependencies in your tests, then you can use KNOCK to mock those out. The other really nice thing about NOC is it's got a experimental feature which allows you to turn it on or off. So basically then you can have fully integrated tests and mocked tests all in the same suite. So yeah, that offers a nice feature as well.

Nice. So with either contract testing or API testing in general, is anything you see a lot of companies doing wrong or people doing wrong or misconceptions.

Yeah, in my experience there's kind of two different approaches where it's we mock everything or we don't mock anything. And both come with their challenges. Right. And also sometimes you can lose sight of what you've covered at which level. And that's really important when it comes down to test coverage is identifying whether you're actually testing the service end to end or whether you're mocking some of the dependencies. So yeah, I think for me that's one of the things that I see done wrong and that usually comes down to definitions because everyone has a different definition for what is a component test? What is an integration test? What is a system test? Everyone has different definitions for that and I think that gets very confusing around this, this topic.

Awesome. So should everyone be doing contract testing then? If they have APIs in microservice, do you think it's like, you know, I mean it's just like unit testing if you're a developer, if you have APIs, you should be doing contract testing?

I think so. I think there's lots of benefits which are outside of just that. Test the visualization of your consumers understanding their real behavior. Like if you would map this to understanding your users and your internal users are your users. So why don't we put that level of detail in there? That understanding of behavior is really key for me. And then microservices is difficult to test when you're doing that end to end process. So isolating it in this contract environment just makes sense to me.

Yeah, absolutely. I'm listening. It seems like a no brainer for sure. So I noticed I've mentioned two courses here. I just want to make sure folks have the reference here. The first one is API testing and JavaScript at test automation University. And you have another one, you probably have others. The other one is on udemy, next level API testing, contract testing for microservices, which I'll have links for. And the next level API testing course. I think you have over 300, almost 400 students curious to know like the type of statistics you get from a course like that. Do you see which modules are most popular? Or maybe are you getting questions that you're like, huh, I didn't think about that. Made you change your curriculum to maybe handle something that people may be struggling with or want to learn more about.

Yeah. So I think up front it's written in JavaScript and. NET and it would be nice to be able to support other languages because yeah, not everyone's doing it in that way and it's hard to explain the concept without using examples and then you have to provide code, which is difficult. So yeah, I guess that's one of the biggest feedbacks. And I don't go into a lot of detail. It's very much an introduction course. It'll be nice to delve into matches and all of the setup, the provider state and all that kind of stuff. I don't really have the time to go into that much detail. But yeah, is what it is.

Cool. Do you do consulting? So if people wanted to take a deeper dive or it seems like you're working full time, do both full time and consulting on the side.

So I've now got a daughter who's one year old, so going to spend more time with her and I'm going to consult a bit on the side as well when I have some free time. So yeah, absolutely. If you want to get started with contract testing, feel free to reach out to me and we can have a conversation about what it looks like in your company.

Sweet. So once again, off a wall question, a lot of things I've been hearing more about is GitHub Actions. I think in one of your courses you mentioned GitHub Actions. So how does GitHub Actions work with contract testing?

So it nicely fits into this, the CI pipeline. You can just trigger the test from there and you can have those GitHub triggers to trigger the other pipelines because you've got the consumer and the provider side. When you make a change, then you can easily set up those triggers on GitHub, which makes it work nicely together.

Awesome. Okay Louis, before we go, is there one piece of actual advice you can give to someone to help them with their either API testing or contract testing efforts? And what's the best way to find or contact you?

So I guess my top tip is to live in the code base. Write the test as closely as you can to the code base, whether that's pairing with the developer to get started contract testing. Often you need a developer to get you started. So yeah, I guess that's my top tip. Pair as much as you can and then yeah, if you want to hear more from me, I'm@pacman.co.uk, and check out my blog, check out my courses and listen to my podcast. So yeah, thank you.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com.

And while you're there, make sure to click on the Try for Free Today link under the exclusive sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes. Amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.