238: Karate API Testing with Peter Thomas

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe and welcome to another episode of Test Talks Today. Join me and enter in the Testing API dojo and as we speak with Peter Thomas, the creator of the open source API testing tool Karate. If you're new to programming or automation testing, Karate is really easy to use and requires no Java knowledge whatsoever. It's really easy to get started with API testing. I think it's really going to help a lot of teams out there. It's actually one of the reasons why I asked Peter to join me at the Automation Guild this year to talk all about how to use karate for microservice test automation and how simple it is to get started. He has a demo how to use it against all different kinds of services as well as using it for performance testing. So a really awesome session. You don't want to miss that. When you join us at the guild, you also get an opportunity to ask Peter questions live. So Peter will be joining us live and you can ask him any question you might have around Karate or anything to do with API testing. If you haven't already, go ahead and head on over to automationguild.com register today to check that out, as well as 20 other awesome sessions dedicated 100% to helping you succeed with creating automation testing by some of the top experts in the world. Join us at the Guild register today. AutomationGuild.com test talks is sponsored by the fantastic folks at Source Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testtalks.com and click on the sign up now link under the homepage Sponsor section.

Hey Peter, welcome to Test Talks.

Hey Joe. Thank you and it's good to be here.

It's great to have you finally on the show. I've been wanting to get you on the show for at least a year now, ever since I heard about Karate, which we'll talk about today. But I want to talk about API testing and karate. But before we get into it, can you just tell us a little bit more about yourself?

Yeah, sure, Joe. So I work for Intuit and I play the role of a distinguished engineer and I've been there for six years, was with Yahoo, kind of before that, been around in the industry for a long time, probably too long, and have worked extensively with Java and so a Lot of that experience is currently coming out, turning out to be pretty useful with this journey I'm having with karate. So that's about me.

Very cool. So I'd love to get into karate. But before we do, I just want to talk a little bit about API testing. I think it's really important, but I was curious to know why you created a tool for API testing. Why do you think API testing is so important?

Yep. Great.

Well, to be honest, Right. I wasn't that much into test automation until very recently, although they may. That may sound a little hard to believe. And what happened was I was.

In terms of a little background. Yes. I work as part of a team that's responsible for a whole bunch of platform services. Intuit, my company, is very big on services. We've been having this microservices journey happening for a long time and we've made some pretty good progress and there's a lot of fun stuff that's happening. I.

Kind of had to look at a particular test suite almost two years ago in my team and I didn't like the look of what I saw. And I am a little particular about the way I write Java code and so on. And this particular in house framework that was being used by this team, it was homegrown, based on the Apache HTTP client, all the usual suspects. I found it really hard to understand what was going on, hard to debug. There were layers and layers of code because various people had come in along the years and tried to add improvements and added more edge cases and features and framework stuff, which I think kind of did more harm than good. So, yes, I've kind of rambled a little bit there, but I didn't like at a high level, I didn't like what I was seeing in terms of what was happening across the company in terms of how people approached API test automation. And I thought there needed to be a better way and I kind of decided to try to create something on my own and that's how it all started.

Very cool. I love how you took that initiative because I see a lot of people not taking the initiative. They see an issue and they just complain about it. So what made you want to take that initiative though? Like you start on your own as like a skunk work project and then once you had a nice work and model, you rolled it out to your team and then. And went enterprise?

Yes, I think you guessed it right, Joe. It's pretty much kind of that's what happened. This essentially happened in December 2016 when I actually Sat down in December. I did have a little bit of spare time. It wasn't that hectic at work, kind of understandable. It's kind of the end of the year and all that and I just decided to take this time off. And by the way, my company encourages a lot of white space time, have this unstructured time concept, which is pretty useful and cool in my opinion. So I kind of made use of that time, sat down, used cucumber as a base and created this.

Prototype that eventually became karate like we know it today. So Kunk works pretty much. I actually showed it to a bunch of folks. I didn't get some very positive reactions, to be very honest. And I think the breakthrough for karate happened when it was like no one really believed in.

Seemed to be cool, but people were having their own priorities and all that. But I think once I open sourced it, that was a huge learning for me and the company as well because once we saw some traction and yes, people like you, Joe, who picked it up and thank you for the initial boost that you provided karate with your blog post and the video tutorial. Like one year back, one and a half years back, it kind of took off. A lot of people used it and yeah, it all took off from there and it's kind of taken on, taken on a life of its own in a cool way.

Awesome. So you did mention you learned a lot from open sourcing it. Can you give us an example? What did you learn by open sourcing it?

Oh, yes. Oh man. I can talk about this for hours. Many things. And one data point which I love to tell people is that if I had known what I've learned now with this one and a half years with karate, I probably wouldn't have open sourced it because it was so incomplete, it was so raw, it didn't have a bunch of features that are there today. So tremendous learning. I think the amount of feedback you get from random folks or passionate folks all across the Internet, you know how it is. Like, you know, you open source something, it's like pretty much the entire world is going to look at your code. Of course that's a brave thing to do because you're going to get criticized for your code as well. So I've got a bunch of that. But you get some very genuine feedback. People try it and then the you will have these one or two people who are really passionate and who would give you feedback and give you more ideas. And I think it's just, I think 80% of karate the way it's Become right now is based on a lot of requests and people saying, hey, why can't you do this? And why not have this feature? And I think it's been amazing and I would thoroughly recommend open sourcing to anyone. And it's been one of my pep talks that I give a lot nowadays to engineers within the company.

Awesome. So like I, like you mentioned, I did look at this like a year and a half back, but it seems to have been evolved, evolved a lot. So it's great that it seems like you've taken that feedback, you've baked it into the tool to make it even better. I'd like to dive into that, but before we do, I guess maybe we should mention what is Karate at a high level.

Oh yeah, sure. So Karate primarily is framework for testing any web service API. So it's very good for REST services and it can do SOAP as Well, and even GraphQL, which is kind of picking up right now as a technology option. But Karate is probably one of. I like to think of it as one of the world's best HTTP clients. It can make all kinds of HTTP requests and all the edge cases that you can think of, like file uploads, form authentication of URL encoded form submits and all that. But I think a lot of people kind of under appreciate that Karate's real strength is in asserting JSON responses and even XML responses. But I personally think a lot of people miss this point when they kind of go about evaluating or even building their own frameworks. Is that the biggest job you have as a QE or a test engineer is to actually assert on what you're getting back against expected results? And Karate, I think, personally does a very good job at that.

So with Karate, you did mention rest APIs, and there are.

A large amount of solutions for that. So it does work with soap and WSDLs or what else can it test? Or does it only focus in on REST services?

It's mainly used for REST services, but as you said, Joe, it can test soap. In fact, that has been a huge surprise to me. Two things I want to call out. For whatever reason, I think when I created the original prototype of Karate, I was actually having to test a SOAP service and this is kind of an inside joke in my team. And what happened and.

What it has translated to as of today is that Karate has excellent support for XML because I kind of built it in from the start. It may be a surprise to many that JSON kind of came in as a phase two of the initial prototype. What I was getting to Joe is that it surprised me that a lot of the usage I'm seeing right now in the wild, people who are using Karate, they seem to be using it for testing SOAP services and I think there seems to be a lack of good options or good free open source options in the industry currently for SOAP testing. Happy to say that a lot of people are using Karate for soap. You don't. It is actually interesting. It's not WSDL based. Karate treats everything as, you know, plain text. You just send raw text to a server and you get back raw text. It can be JSON or xml, but then Karate does a great job of asserting and allowing you to unplug things from the response and all that. So it's kind of surprising to. Maybe surprising. It's not wizard based. You just once you have this XML envelope which is plain text, it's very easy in Karate, by the way, to kind of take that and create edge cases and multiple scenarios out of this one base happy path payload. And yeah, Karate is used for SOAP and it seems to be working well.

Well, yeah, I'm surprised that people are still using soap. So that's interesting. So I guess for someone getting started, do they need to code up the solution? You mentioned some envelopes working with the SOAP response. Do they need to know that is the front end intuitive enough for someone that's not a developer to start writing test against a SOAP or a REST API?

Great question. I personally think Karate is actually quite accessible to people who don't know programming. So as I said.

I'm going to go on a tangent here. One of the great things about HTTP is that it's plain text, right? It's a very simple. It just boils everything down to first principles in a way. And I personally think that's why REST is so successful and you know, why a tool like Karate that's kind of embraced those principles kind of seems to be doing well and it's caught on and adopted and all that. To break down my answer a little further, I think it's easy as a developer, let's say you need to come in and write a SOAP test suite from scratch in my mind. And this is what I see a lot of teams do and this is what I advise teams to do. All you need to do is, you know, just get this one sample payload of a full HTTP request and possibly a response, and it's plain text. You just cut and paste it into your script. And I think one of the good things about Karate, where it actually has an advantage over traditional programming languages is that it supports you pasting these great big blobs of text right into your script. And what I would recommend is you can kind of split them out into separate text files which lie alongside your main script so it doesn't clutter your test script and all that. But yes, you just need this one request and then Karate would post it, do an HTTP post, get the response. And I've personally found this is a very quick way for teams to get up and running because there's a variety of ways in which you can get plain text samples of your HTTP request to the server. In fact, if you're talking rest, typically the development team would have that you would have these things in an ideal state. Obviously you should have these text files as part of the API spec against which possibly you're developing. Yes, to summarize, you just get plain text, it's either XML or JSON and then it's very easy to just build a test script starting from that point.

Very cool. And when I first looked at this, I think you were using Cucumber as a front end, so someone didn't have to write any code to get started. Which is given I have this endpoint, when I return this, I should get this value back or this response code, is it still leveraging Cucumber or have you moved on beyond Cucumber as the front end type interface?

So that's a great question, Joe. And in fact you might have seen Nano. There's been some recent chatter on the Karate forums and so on that we've actually moved away from Cucumber. But if I rewind all the way to the start, I used cucumber version 1.2, something I don't exactly remember, as a base and it was great as you just called out Joe. Right. Cucumber does this fantastic job of giving you this front end or the user experience that you think in terms of English keywords and you have. And I personally like the fact that given when then kind of reads very well when you are kind of eyeballing a test script and so on. So Cucumber does a great job of giving you this keyword driven test scripting approach, so to speak. And there's a great way that Cucumber connects these keywords to the backend Java code. People call it the glue code that is really well done in Cucumber. And thanks to that I was able to create this prototype that, you know, worked well and people who are non programmers love it. You know, it's not True bdd. And of course, you know, that's. That's a separate topic topic in itself as to whether karate is BDD or not. But karate in the early days. So I was saying yes, when karate started out, it used cucumber as a framework. But I found that, you know, the cucumber team keeps changing. There's been a lot of, you know, developments and changes to the API and some of the breaking changes were a little hard to deal with. Karate kind of locked into cucumber version 1.2 for a while and I kind of took a decision very recently that's about three months back to move out of cucumber, but at the same time retaining these syntax 100%. So that's good news for people who are actively using Karate because there's no change. The upcoming version, which would be 1.0.9.0, would just be a drop in replacement. There's no more cucumber behind the scenes. It's just custom code. Karate. I mean, that we have written to replace all of cucumber, but it would still work and give you that same, in my mind, the pretty awesome experience of just writing what kind of looks like plain English, but definitely very readable, but at the same time be able to do this heavy lifting of making HTTP requests and asserting against all kinds of responses.

And like I mentioned, you have been making a lot of changes. Another one I think is performance testing. Are you able to leverage Karate now for some sort of API performance testing?

Oh, yes. That to me is actually, I think, one of the things I'm most proud of in this whole Karate journey that I keep talking about. Performance testing as a capability of karate was released, I think, in June or July, sometime around that time frame. I think it's pretty cool. It's actually in my mind, I think I have been chasing this for years whenever I had to get involved in the past in various companies to do testing. I've always found it irritating that you have this bunch of functional tests, but then someone needs to go and rewrite all those flows into something like JMeter or Gatling or similar tools. I considered that as the holy grail, quote unquote, of test automation, where you should be able to reuse your functional tests as performance tests. And Karate can do that. Now it uses Gatling to actually execute the test. And in my mind, it's a very cool integration because for those who know Gatling, one of the learning curves involved is that you need to learn a little bit of Scala and figure out how to make all those constructs and HTTP requests. You don't have that problem now with karate. You can just take your existing karate test, slap a load model on top of it, which pretty much is three or four lines in your Gatling script that says, okay, this many concurrent users, this is how many users should wrap up over how many seconds and it just works and it's brilliant. And on top of that, if you get assertion failures, you can actually. The Gatling report would actually have the line numbers of which karate script failed. So I think that part is also kind of. Well, it turned out really well. There's a few people using out who have reached out to me with some very positive feedback. So I'm kind of, I'm totally pumped up with the whole performance testing of karate, which is kind of new. So I hope more people look at it.

I love Gatling. I didn't know about this. That was used under the covers. I definitely need to check it out. So where are you getting time to do all this? You do work full time. How did you get time to implement performance testing and all these other changes? Do you have a larger team now that's helping, moving it forward. Do you work on this full time? What? You have a community that's grown around that's been actually contributing through the open source model?

Yeah, that's a great question, Joe. And it's mostly me to be honest. And I think you can see this for yourself if you look at the commit history of karate. I have got a lot of comments about this. It's mostly me, as you said. The good news is that because it's really been, you know, it's doing well in terms of adoption. There's a lot of companies using it. In fact, one thing I love calling out is the fact that it's actually appearing regularly in job listings and there's been a few folks, not too many to be honest, but I've had pull requests. So, you know, people are contributing in terms of time. I think clearly people who know me would vouch for the fact that this is a particular peculiar obsession of mine, this whole karate and API testing and you know, kind of changing the world in some ways. I pretty much, you know, spend a lot of time on this, certainly after work. A lot of my hours are spent developing karate and responding to feedback. Stack overflow is amazing right now there's a huge uptick in questions, which is one indicator of I think, how well it's doing. And yes, I do spend a lot of my weekend time on this. You know, of course my Family doesn't like that that much. But yeah, it's, I think, an occupational hazard you need to deal with. So, yes, it's. It's a pet project of mine. I just love working on this. I really hope it doesn't burn out. At least I don't see that happening in the near term.

Awesome. So if someone has an existing Java project and say they're using cucumber already, are you able to plug in Karate and get the benefit of using it, or do you have to start a brand new project?

Yeah, that's a great question. Now you can. How do I say this? Most people, yes, they start using karate from scratch. You know.

The good news is that you can have karate side by side if you have an existing Junit test suite. And we see a lot of teams that do this, even in Intuit, my company, which is obviously, you have this legacy test that have been written in, let's say cucumber, or at least typical at Intuit as people have used, rest assured, or the Apache HTTP client. The great thing about Karate is the dependencies are so minimal and I think that's something that I worked really hard to preserve as Karate Evol is that in terms of maven dependencies, the jars that it pulls down are very few and most teams don't run into jar conflicts. So. And even if you do, you know, there's a little more I can talk about, which is Karate has two options. You can either swap in Apache, which is the default recommended HTTP client, or you can switch to Jersey. So if you really have, you know, project jar dependency library conflict, most people overcome it. And basically I've never heard people complain about that fact. Now, in terms of if your question is on whether you can interrupt with existing cucumber code, that's a little more tricky. And as I hinted earlier, Karate is a little opinionated about how it deals with cucumber and how it approaches the given when then and the scenarios and the features and the construct that cucumber has. So at least till now, till the current version, which is 0.8.0 as we speak, it's actually not possible to mix karate with cucumber. You can't have it both ways. The good news is that since we just talked about Karate now, having had this engine overhaul where we moved out of Cucumber, going forward, I'm looking forward to people reporting back that they would be able to, you know, mix Karate into an existing test suite where they have cucumber and it should work. We haven't really tested it yet. It's still very new, but it's going to be possible or even if, you know, we see some initial hiccups. And I'm quite confident that we can work through resolving these things.

If someone has a Karate project, is any sort of reporting that's generating that extra benefit that they get from using Karate.

Oh great. I think Karate has a very good reporting story. You know, many people are using this currently. It's probably no surprise if you think about Karate's origin, you know, as a layer on top of cucumber. And we all know cucumber has been around for a while. It's got this massive ecosystem with plugins and reporting and all that. So the great thing is that Karate inherits all that, right? So specifically karate has direct support for, you know, it generates the JUnit XML format by default, which means it's straightaway slots into all the CI CD tools that you can think of like Jenkins. And beyond that, I think most people have. Now most people use this particular report, which is the cucumber HTML report. It's probably the most popular cucumber third party plugin. It's actually a maven plugin, but we have this particular recipe that is on the Karate documentation. If you just add this one dependency and five lines of Java code, you're going to get this wonderful HTML report which a lot of people love. And I think one plug for this particular report, if I, you know, take the liberty now to talk about, is, you know, it actually within the typical cucumber HTML report that people are used to, it actually embeds the, the HTTP request and response, the full logs, including headers, the full payload as plain text. It's, it's there in the HTML report. This works even if you run Karate in parallel, which again is one of the things that, you know, people love about karate because API testing is slow and so on. So yes, I think karate has this great story for reports. And the best part is since we use the cucumber output format, which is kind of a pseudo standard right now, because many people use actually opens up Karate to be plugged into anything that supports cucumber. And that's a lot of tools if you think about it.

Awesome. So I was looking at some other new functionality. One of them that I came across, I'm not very familiar with this API testing double. What's an API testing double?

Yes, some people like to refer to this as mocks or stubs. And yes, thanks for bringing it up, Joe. This is something we talked about, the gatling and the performance testing stuff, the mocks kind of landed a little before that. This was probably early this year, 2018, and I am. Okay, this is a big favorite of mine in terms of features and what this means. And I would like to talk about this. And I think this is something a lot of quality engineers should really appreciate. And anyone who works in microservices for that matter. What Karate's Test Doubles capability offers is that you can using the same Gherkin scripting language. And yes.

As an aside, if the original creators of cucumbers saw this code that I'm now referring to, which is cucumber styled or gherkin code that can actually implement a REST service, yeah, they'll probably lose their minds and they'd be pretty upset. But what I was saying is that Karate has this new capability or new ish capability that a lot of people are using, which allows you to boot your own HTTP REST web service using very little, very few lines of code. It's very carefully designed so that with a minimum of coding and a minimum of typing, you can stand up this fully featured REST web service. And you can imagine straight away that this is going to be tremendously useful, especially when you are in a situation where your E2E or your pre prod environment of your dependent systems. And dependent systems are a big challenge. As we all know in microservices, typically you are working with a team that is still in development or they don't have this stable test environment, or it's a flaky environment. You've been through all this, Joe, I'm sure. And Karate gives you now this option where you can write, you can very quickly stand up the service and use that to integrate with and make sure that the service that you own as your team is working fine. And then, of course, swap it out when the time comes. But one way to express this is that now with Karate, you have this superpower in my mind to actually run your entire CI CD pipeline without having your network connected on your laptop. And that's pretty cool, if you ask me. And one more plug, and I think this is something that a lot of quality engineers would love to hear. Another way to think about Karate's Test Doubles capability is that it gives you, as a quality or a test engineer, the power to actually create an HTTP service much faster than your counterparts in the development team would be capable of. I really think this is one of the underappreciated, secret features of Karate. I've been trying my best to market it, but I Would really encourage a lot of people to take a closer look at this thing. Yeah, there's a lot I can talk about this.

I love what you're doing here and hopefully more people after hearing this will help contribute. It is open source, so definitely I think more people involved to help you out would be awesome. Okay, Peter, before we go, is there one piece of actionable advice you can give someone to improve their Karate API automation testing efforts and let us know the best way to find, contact you or contribute to the karate project.

Oh yes, let me see. In terms of advice. Ah, yes. I have this very strong recommendation for people using karate and having questions and my advice is please read the documentation.

But yeah, but seriously, you know karate, you know, I've spent a lot of effort in the documentation and I think it shows. There's a lot of people who actually complain that there's too much documentation, but I think that's a good thing for an open source project. It's kind of rare. I think a lot of the questions that people have when they start out with karate can be avoided if they do take the time to read through the documentation. It's a lot, but it's in my mind at least pretty well structured, so that's going to help. And in terms of reaching out, the best way to get support on Karate is Stack Overflow. There's actually a thriving community. In fact. Yeah, one of the things I'm quite proud of in the last year or so is that, you know, I've, I've actually crossed 10,000 points on stack Overflow. So I'm kind of a, you know, have some super user rights and all that. All because of all the answering of karate questions that I do nowadays. And so yes, if you are looking for quick answers to your questions, it's either me or now there's a thriving community. So you're bound to get some quick answers and you'll be well on your way, I'm pretty sure of that.

Thank you Peter, for your API karate testing awesomeness. For links to everything of value we covered in this episode, head on over to testtalks.com 238 and while you're there, make sure to click on that sign up for a free trial link under the exclusive sponsor section to learn all about Source Lab's awesome products and services. So that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks. Podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles and other automation awesomeness. 16.