Contract Testing in Action (Book Review) with Marie Cruz and Lewis Prescott

In a land of testers far and wide they journey seeking answers, seeking skills, seeking a better way through the hills they wandered through treacherous terrain but then they heard a tale of podcasts they had to obey O the Test Kill Animation testing podcast Guiding testers with our nation awesomeness from ancient realms to modern days they lead the way O the Tuscalon Ovation Testing podcast With sweeps and lyres the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing them secrets to behold O the Chesskill Automation Testing podcast Guiding testers with our nation awesomeness from ancient realms to modern days they lead the way oh, the testkill Automation Testing podcast.

Oh, the Tescal Automation Testing podcast With lutes and liars the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing us secrets to behold.

Hey, do you want to learn more about contract testing? If so, you're in the right place because today we're going to be talking with two awesome experts on all things automation. But more on especially contract testing today with Louis Prescott and Marie Cruz to talk all about. They have a brand new book. I think it's still in the works, but you could still get, I think an early version of it called Contract Testing in action. Really excited about this. If you don't know. Luis has been on the show before. He's a test specialist at IBM, has over nine years of experience in software testing, is a recognized champion in contract testing and a course author at Test Automation University as well, and a totally crazy active member and mentor in the testing community. And we also have with us the one and only Marie Cruz, who is a software tester with over 10 years of experience. Currently she's a senior developer advocate at Grafana Labs and she's previously worked as a engineering manager responsible for driving continuous testing and quickly improving many improvements across many organizations. As a principal engineer focused on introducing recommended best practices for testing and test automation frameworks. One of them being around contract testing. Really excited about this episode. You don't want to miss it. Check it out. Hey, Lewis and Marie, welcome back to the Guild.

Hey, Lewis. Hey Joe.

Hey.

Hey.

Yeah, really great to be here.

Yeah, awesome to have you both here. It's been a while. So you two, like I said, are really, really busy and I'm always curious. I always ask authors why, why write a book when I know you two are crazy busy? So Marie, let's start with you. Why? Any thoughts? Why write a book at this time?

Yeah, it's always in my bucket list to write a book in terms of like professional. And I reached out to Louis like over a year ago because I know Lewis is really passionate about contract testing. And we both noticed that while there are a lot of blog posts packed, documentation is great, but there's no dedicated book about contract testing. And I think during this time as well, I was inspired by Mark Winteringham's dedication in writing the book about testing web API. So I reach out to Lewis. I asked if he will be comfortable writing a book with me because it's always, I guess, easier to do it with someone that you know from the community because taking it on just by yourself, it's gonna be a mammoth. So yeah, it basically came from that. And yeah, about a year after the book is here. So it's been a long, long hard work. But I'm glad that we both did it.

Absolutely awesome. Like I said, it is a resource I think the community really needs. So Louis, you've been talking about contract testing forever. I think last time we had you on the show you were talking about contract testing. Very passionate about contract testing. So why a book now on contract testing?

Yeah. So as, as Marie mentioned, it's all her fault. She got me involved in the book. But yeah, no, I think as Marie said, like, there's no dedicated book and like one of the biggest hurdles. Right. Is getting started with contract testing. So like, we really wanted to address that with the book. But yeah, I think it'll be a long time before I put pen to page again.

Nice. So talking about it's going to be a long time. I noticed it's in. I don't know if people know about Manning. They have like an early release program, it sounds like. And it looks like there's four chapters out of, I think 12. So all those chapters are already done. And I know this is like a real publisher. It's not like one of my books that is self published, it's a real publisher. So it's already in the works and you're just waiting for them to be looked over and make sure they're blessed before they go live. How does that work?

Yeah, pretty much. Lewis and I have been really working hard delivering the rest of the chapters. We still have one more chapter to finish, which is around the bidirectional contract testing and another round of reviews as well. But I think what Manning does is they try to regularly provide updates to the users. So if people are going to purchase it as part of their early access program. The goal is to deliver each chapter every month or some new updates. But yeah, most of the chapters are done. We've had amazing reviews as well from external people from, you know, manning side as well, as well as from the community. So we've incorporated most of those and I think we have another round of final review. So fingers crossed that yeah, it's going to be really, really out.

So awesome. We're going to have a link to it in the show notes down below. I know I've been seeing a lot of posts on it on social media and it looks like it's getting a lot of reviews already on the, on just early chapters. So really excited for people to grab the full copy. But you can get an early copy now and the links down below depending when you're listening to it or the full version as well. So I guess let's kick it off by people, if they're just listening, they're like, oh, this is great. What's contract testing? Let's start off principle one. Louis, so how would you explain what is contract testing before we dive into more of the book?

Yeah, so we really like delve into this in the first few chapters of the book. So we're really trying to outline what the different components of it are. So who is a consumer, who is the provider, who are the interactions between? Where do you put the contracts once you've generated them? How are they different to like your open API specifications? We really kind of delve down into those myths, into those preconceptions of what it is. We delve into those in the introduction and the way that we explain it is through like that mental model of, okay, this is a scenario that you have in your application, which is, for an example, an API. You have an API which communicates with a mobile app or web app and that forms your contract, right? The two things communicating together, the contract sits in between and then you've got all of the nuances of updating that contract to I need to make a change to the web app. How am I going to make that change with this API? Is that going to break it? How is it going to behave? So yeah, in a short stop, it's about that communication, right, between two systems. That doesn't have to be an API, that could be an event or a GraphQL interface, but it's that communication between the two systems.

Awesome. And Marie, why is contract testing important, you think nowadays to write a book, full book on it. So is it Something you've been seeing. I know you said there's not a lot of resources out there, but are you seeing it's a technique that a lot of people are missing out on?

I definitely think so, because I think nowadays as well, there's a lot of focus, especially with microservices. And as we increase the number of APIs that we have to consume, how do we then test those different integration points? Because while that type of approach is really scaling well, I think the integration test that we know currently, which is we make a get request to a specific endpoint and then verify if all the data, if everything there is correct, that might not scale as well. Let's say if you have more than one, or if you're like Netflix or Amazon, who have loads of different microservices, how do you then ensure that any small changes that each team deploys to production doesn't break any of dependencies out there? Because a common scenario that I've seen in the past was we have a consumer that could be a web app and then we have a data provider. Both of them will have their own set of tests. The provider might deploy some changes from their side and then the tests that they have in their pipeline are all running. But it's quite common to miss the sort of scenario where have they communicated it to the consumer, because most of the time we end up with the changes being deployed to an environment already and not knowing that that has broken some sort of integration that the other team is using. So I think nowadays it's a type of testing that both Lewis and I believe has that great impact, because it's really going to let you know way, way earlier on if any of your changes are breaking any of the expectations from your consumer or it just gives you that early confidence that whatever changes you're about to make is not going to impact any other systems out there. Whereas with traditional integration tests, you typically have to deploy that change first, because you test with actual test environments and you end up, I guess, having a longer feedback loop because then you've noticed that there's a bug and then your tests are failing and then you then have to communicate to that other team. Whereas with contract testing, it detects that issue earlier on. So before even deploying those changes to an environment.

Awesome. So let's dive into the book a little bit more. It's broken up into three parts. The first one is, I think, introduction to contract testing. So if anyone's starting from scratch, this is obviously where you start. Everyone starts, I think, part Two is consumer driven contract testing and the third is provider driven contract testing. So in the first part, I think we went over the why the how. I did give a technical overview in chapter two. Do you have like a. I think you gave a pretty good example there, Maribo. Any other examples of contract testing principles as they applied to maybe a real world scenario?

Yeah, I think in chapter one, I've definitely provided an example where I think most of us have encountered where imagine, you know, you're working on a Friday and I think it's good to deploy changes on a Friday, but sometimes teams are still scared to do it. But to teams who want to deploy on a Friday, they have their tests running on the pipeline, but then suddenly there's now some sort of alert or notification that something's been broken in production. And that has happened quite a few times. So the way we've introduced contract testing is imagine if there is a way for you to even catch all those changes before deploying it to an environment. Another way of explaining it is using the contract example that we have with Manning. So I've explained it in a way that contract testing is like when you sign a contract with someone, you have to make sure first that everyone agrees with what's written in the document, whether that's from the consumer view or that's from something that the provider has to provide to the consumer so that there's no surprise element right at the end. In this scenario though, the one that's driving it is the consumer. So the consumer will say, okay, if I make a GET request to this particular endpoint, I expect to receive this status code, this different data that has this particular data type, and so on. And in that contract as well, it will define what sort of other scenarios that may be quite difficult to emulate in an integration test because you need to, for example, emulate, okay, what data do I need to make sure that I receive a 404 code? Or what scenario can trigger if it's going to be a 403 unauthorized. So all of those will be documented in this contract. And then with this approach, the provider can then say, oh, okay, this is what my consumer wants. I don't need to over engineer my API. So it's helping the provider in a way as well, so that any changes that they work on is really what the consumer needs. So that's the focus of the whole consumer driven approach. Yeah.

Nice. Louis, any favorite points in the first maybe three chapters that you want to highlight or you think maybe people struggle with that? This book Addresses.

Yeah. So I think one of the things that we really like, touch upon is how you would sell this. Like, how do you get your team bought into the whole contract testing process? Because I think lots of people get stalled at that point because you do the proof of concept, you're like, oh yeah, we can definitely start to use this. And then you go down the route of, okay, we need to deploy an instance of the broker. Why would we do that? We've already got integration tests which test this scenario. So weighing up those decisions of why would you choose this approach over another approach? And then, yeah, providing those benefits of that early feedback. Being able to debug your API issues quickly using this tool, like so, providing those benefits so you can sell that back to your team and have a strong argument.

Love it. And so this is the first part of the book. Second part of the book. I think it's the meat of the book because I think it has like seven chapters. Consumer driven context testing. So you know, you cover things like web application, mobile clients, provider contract test, graph, SQL, event driven architecture, storing and contracts, CI, CD pipelines. It's very thorough. So let's pull a few of these just to get people want to learn more and grab in the book. So first one is what do people need about contract testing as it applies to maybe web applications, since that's one of the chapters.

So in chapter four, really, we focus on the most common scenario where people will apply contract testing. So this is when you have a web application and it could be like a REST API as a data provider. So in that chapter we really showed step by step how you can write a contract test from both the consumer and the provider perspective. We've chosen PAC because it's the current de facto tool when it comes to contract testing. And it's something that Louis and I have also used a lot of in previous projects. So if there are people who want to get started with pack but they don't know how. So we provide the actual GitHub repo like step by step, we provide the different scenarios that are quite common between a web consumer and an API and we walk them through how they would write the different configuration, starting from defining where the pack should be stored. And we've also done it in a way so that rather than setting up a pack broker by themselves, because we want to make sure that they can immediately see the value of contract testing, we've chosen to use the free version of packflow so they can just upload the contract there and see everything in action. Although in later chapter we do have a chapter around the setup of how you can use your own pack broker if you're going to host it yourselves. But mainly that's the meat of chapter four, really showing the basic concepts. And then from the provider side it's making sure that whenever they pull the contract that they can replay the interactions that the consumer has registered as part of that contract and then making sure that the expected response coming in from the consumer matches their actual response. And that's where we sort of explain that if there are any miscommunication or if there are any mis like expectations, then that test can really capture those feedback earlier.

Nice. So out of the all the different types of contract testing you can do is one more difficult than the other or trickier like say comparing web as opposed to mobile or GraphQL. Any one of those maybe require a little extra effort or it all depends.

So yeah, I mean it's supported in all of them and they all follow the kind of same flow. So the, the implementation and the setup and everything should be quite similar. The only one that we kind of flag that requires that additional effort is event driven if you are tightly coupled to the event system that you're using. Right. So you would need to kind of refactor it or come up a way of making it agnostic to that technology. But that's the only thing is if you're already tied into it, otherwise you can just pick up and play with it. As PACT has all of the supported like languages, all the main ones and then yeah, like GraphQL and stuff, it's got those plugins to enable you to do that.

Yeah, I think with the different tests like what Lewis said, it all follows the same approach. However, in my opinion I think the biggest area where teams start to find the whole concept of contract testing a bit more complicated is when they integrate it as part of their continuous integration pipeline. So that's why we have a whole chapter dedicated and I think it's the longest chapter if I'm not mistaken. So we really cover it step by step how to integrate the contract testing as a pipeline if you're using it GitHub actions we also acknowledge that there are other continuous integration providers out there, so we provided snippets. Say for example, if you're using CircleCI, this might be an example config. But we focus with using GitHub Actions as our main CI tool and one other concept as well that me personally found a bit challenging in the past. So I wish someone would have Showed me the way to do it. I had to really do a lot of experimentation was if I need to trigger the provider verification automatically if there are changes in the contract. So this is where having some sort of examples around that can really help the readers. Because while the documentation out there online is all good, I think diving into it step by step will really help the readers solidify why they need to also trigger the provider verification automatically. So that's why we spent a lot of time in that particular chapter.

And I guess before that one, you also have a chapter on storing the contracts. Is that an issue you see people struggling with where, where do they store them to make it more, I don't know, secure or scalable.

Yeah, exactly. And it's one of those conversations that you have at the start as well is whether you host your own contract store or you use the software as a service packflow offer. So, yeah, we weigh up those pros and cons and also, yeah, we go through the kind of different functionality that you might need depending on your circumstances. Yeah. Whether that be authentication in a certain way, whether that be storing the contracts from in Docker or spinning up in whatever format your company already uses.

Nice. So do you see most people don't have contract testing tied into the CICD pipelines, or do you see they try to and they fail? And that's why you created a chapter that's probably really one of the meatiest ones in the book.

I mean, I can speak from personal experience, but the way I've experienced it was we would try to integrate like the basic sort of workflow. So if anyone looks at PAC documentation, they have this concept of PAC nirvana. It's their way of sort of expressing the different levels that people need to achieve to make sure that they get the full benefit as possible. With integrating contract tests as part of pipeline. And when I first introduced contract test, I always go with the smallest value that I can provide, which is, okay, we can run one consumer test. So we need to make sure that the provider is also verifying it. Great, that part is done. But at the same time, if there are any changes in the contract, what normally tend to happen was in the past like the whole provider pipeline was being triggered manually from our side. So to reduce that manual sort of approach, we had to integrate this thing called pacwebhooks, which basically sends this message to the provider that, hey, there's some changes in the contract, but rather than triggering the entire provider pipeline, I'm only going to trigger the specific job to verify the changes. That the contract has. So that has given us a lot of benefits. So there's no more manual intervention. But it took me quite some time to make sure that the webhooks are all configured correctly. There's a lot of trial and error. So I just thought that other people might also experience the same issue because different CI workflows will have different ways of, I guess, in terms of their configuration, but the concept is all similar. So I feel that that's quite an important piece of information to share because other people might also experience the same issues where if there is a change from the consumer, rather than triggering the entire pipeline, which I think depending on how your pipeline has been set up, that can take some time. So you would rather just have that immediate feedback loop of just triggering the specific. Provide their verification job.

Nice. So that brings us like. Like I said, people need to grab this book because there's so much info on this, which jumping to part three now, where you go over Provider Driven contract testing. So I think it's the last two chapters as well. Transitioning from integration tests. What does that mean, Louis? What's that all about?

Yeah, so they're quite closely linked to last two chapters. So Provider Driven is kind of that approach of you've already got integration tests, you've already got your open API specifications, and you want to retrofit in contract testing. And that's the scenario lots of people are in, right? Like, they've already got the tests in place, so they want to start improving what they've got. And so that's where Provider Driven comes in. So you can use your open API specification, you can use the tests you already have, the integration test you already have, and then that translates them into a contract verification process. And so there's some adapters on the consumer side, so you can use the mocks you already have, and then they join up in the middle. And PacFlow has a magic contract comparison where it compares the mocks with your specifications. So you don't need to have that duplicated effort of saying, okay, now I need to go and write my contract test. You can use what you already have. And in the last chapter, where we talk about converting your integration tests, that's where we're talking about, okay, what benefits are you going to get from? And what scenarios can be converted so that you can see what still needs to be integration tests, because you will still need some of those, what can be reduced in order to give you that earlier feedback and that stability of using contract testing.

And Marie, let's Wrap it up. Any thoughts on the last two chapters of what you were maybe excited about or something that you struggle with or people should know more about?

Yeah, I think in the past I definitely struggled in introducing contract testing when we have like public APIs or third API or third party APIs where we really don't have much control in terms of their process because they have their own way of working and especially with public APIs where you don't really know who your consumers are. So the consumer driven approach that we've introduced might not be best suited for those scenarios. So that's why Louis and I have also included a chapter or two around provider driven or bi directional contract testing because we feel that now that's been introduced, there's really not much excuse for people to not look into contract testing because they can just take their existing test. So from a provider side it could be their existing open API spec, upload that to pacflow, and then from the consumer side, whether you're using Cypress or I think Playwright is also supported. Lewis, correct me if I'm wrong, but if you already have existing functional tests, you can just use that as a basis and use the different plugins that PacFlow provide to convert that into a contract so you can reuse the existing tests. And then what this bidirectional concept will do is it will cross verify the two contracts together, one from the consumer and one from the provider and check if the expectations are similar. So I think this is a game changer because in the past there's been situations where while the whole concept of consumer driven really works, but in some scenarios it might not be the best approach. And this is where bidirectional or provider driven contract testing can help. And I think yeah, to people who are scared of contract testing then provider driven could be your answer.

Love it. And like I said, this book is going to be the definitive guide to contract testing. You're going to want to get your hands on it. Like I said, I'll have a link to it in the comments down below for sure. It also has three appendixes where it goes over what tools you recommend, project requirements, specifications, and even have a chapter on exercises. So it is a must grab. So really excited that you both wrote this. But Louis and Marie, before we go, is there one piece of actual advice you can give to someone to help them with their contract testing efforts? And what's the best way to find or contact you or get our hands on your new book on contract testing?

Yep, I guess one piece of advice. Just grab the book and learn the different examples. But like kidding aside, even if we're not talking about the book, just contract testing in general, it could be quite daunting to introduce a new testing type. But I think if you focus first in delivering the why so why do you need contract testing? What is the value that you're going to get out of it? Then it's going to be much easier to convince the different stakeholders around the benefits and introducing these different tools. And also at the same time, I also want to just provide an advice that contract testing is not a replacement for actual communication between you and your teams. So contract testing can enhance it, but it can never replace the actual human communication. So you still need to communicate with your different data providers or different consumers. And yep, you can reach me via LinkedIn. I'm still on X but not very active, so LinkedIn would probably be the best medium. And I still have my blog testingwithmarie.com so I have my contact form there if any one of you want to, yeah, talk more about contract testing in general.

Yeah, just echoing what Marie said. Really like contract testing is a communication tool and it's really great at that and it really enables in depth conversations around your communication between systems. So yeah, communication is key for me. And yeah, if you want to reach out again, reach out to me on LinkedIn or on X or find me on my website pacman.co.uk.

And as I mentioned, you definitely want to check out this book and for loyal test skill listeners, we have a special code for you. Use Test Guild 24 to get a special. I think it's 40% off the book right now until August 24, so you definitely want to take advantage of it now. Check it out, let me know your thoughts and find it in the comments down below. Thanks again for your automation awesomeness. The links to everything of value we covered in this episode head on over to test guild.com a500 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.