Karate DSL: Performance from Zero to Hero with Artem Bondar

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we're going to be talking with Autumn all about Karate DSL Performance From Zero to to Hero. Autumn is a lead automation engineer at Signify Health with more than five years of experience. He's built multiple automation frameworks from scratch with either Selenium, Selenoid using Java, C Sharp for companies like American Airlines. He also has experience with Angular Unit Testing, API Automation, and other frameworks like Rest Assured and Karate dsl. He is also a true Cypress enthusiast and a mentor at Test Pro. On top of all this, Adam is also a professional certified Scrum Master and he is an instructor at Udemy, where his course on Karate DSL performance From Zero to Hero is located. And that's what we'll be talking all about today. He's a really smart guy with a lot of different topics I'm sure we'll touch on. You don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Load UI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performance testing to learn more.

Hey Autumn. Welcome to the guild.

Hey Joe, thank you so much for this introduction though. So you told a lot about me that probably I forgot personally thank you so much.

Awesome. You do a lot. So I'm curious to know how you even have time to create online courses.

So you know, the idea was simple. When I joined Signify Health in 2019 and the team told me hey, we're using Cypress for automation at that time I'm like what is Cypress? I never used that tool before. I found a couple of courses online and took those and later figured out that probably I can do that better. I knew how to use this tool but there are no much tutorials, good tutorials about that and I created Cypress class first how to Automate with Cypress and then Karate DSL was the next step because this is the second tool that we've used in our company and Karate DSL didn't have any tutorials at all online. So it's just the document on the GitHub official documentation only. And I'm like, all right, I know how to use this tool pretty well so why not create the second class? And I did this. So that's the, you know, the short story behind that.

Very cool. So a few things there. First is not really performance related, but I guess it could be is why Cypress? I've been seeing more and more people really embracing Cypress. It seems to be really growing in popularity. I'm just curious to get your take of why you think maybe Cypress is growing in popularity.

Cypress is a is a new generation of the frameworks JavaScript based generation framework. So you can find like Playwright, Test, Cafe Cypress, you name it. So there are a bunch, there are a family of new JavaScript frameworks. The reason why they are booming right now, they're just, you know, faster, more reliable, they handle more for you instead of.

So in Selenium you have to write a lot of code by yourself like different helpers, different handlers and all that kind of stuff within the new frameworks like Cypress and so on. A lot of handled by framework itself. You focus just on the test. Another benefit why JavaScript frameworks has a benefit is because they are executed inside of the browser. JavaScript is the only one language that's supported by browser, right? You don't need WebDriver technically. So that's why it's faster, it's more reliable and all those benefits you are getting from it, that's why great.

Now I know a lot of these frameworks have other functionalities like Cypress. You could do accessibility testing. Can you do any sort of performance testing with Cypress? Maybe like just front end performance or have any experience with that?

When we talk about the performance, usually it's not about the performance of the ui. Usually we talk about performance of the system as a whole. All your backend, your backend from front end, your API, your services, how overall system behaves under load UI technically is your bridge between your backend and front end. So usually testing for the UI is not performed. I mean performance testing for the UI usually is not performed. Usually performance testing either for the backend systems and there are a bunch of frameworks you can find on the market to test the backend systems.

Great. So why Karate DSL then you're using Cypress. I believe Cypress can also because it's a JavaScript you can download other libraries in JavaScript test API. So where did Karate DSL come in then?

So Cypress actually support API testing? It's built in. Why Karate dsl? Few things. First of all, it has much powerful assertion libraries to verify your schemas, verify your data types in the response. It is more friendly to write API tests in Karate compared to the Cypress. For the Cypress support of API testing in Cypress is more complementary when you need for example make certain action in UI and make verification through the API call or you need to create headless authorization, making few API calls, getting your token saving to the cookies and authorize your application headlessly right away. So that kind of stuff. But when you need to assert a giant JSON object with 10,000 lines of code and that kind of stuff and it's this huge schema that's calling other schema. Cypress not play really well and Karate do really great job in that stuff and talking about performance. So Karate has very great integration with the Gatling performance framework. Probably you heard about that. And what's beauty about that? That when you integrate Gatling and Karate together, technically what Gatling is doing, it's scaling your API test. So you reuse your API test to run performance, that kind of stuff and you generate at the end a very nice, beautiful, detailed performance. Reporting on Gatling. Using Gatling itself is pretty not difficult, but require more skills because you will need to write a Scala code to write performance test in Gatling. In Karate there is a kind of gherkin syntax, you know, given when. Then it's not cucumber as many people may think. Hey, karate is cucumber. It's a shell around the Java to or a shell around the rest assured to run cucumber test. No, it's not. It looks like cucumber, but it's not. It has predefined steps that you can use. And then you call and when you read this test, it's pretty, you know, logic, hey, given path this when I make this call, then result this and assertion int. Is this pretty straightforward? But it's not cucumber. And yeah, and you just connect hook the Gatling to the Karate and you can you take your API test, scale it and you create a performance. Pretty simple.

Awesome. I'd like to dive into that a little bit more before we get into that piece though. What knowledge do you need to know in order to get up and running with Karate for performance testing? Are there any key concepts or key key functionality you need to be aware of before even diving in.

For the performance testing? I think the key concept in the Performance testing is first of all understand what you are testing, right? Understand your system under test and what you are trying to actually simulate. Why are you testing this? So I saw sometimes, hey, we're creating the load, I don't know, 10 requests per second. My next question, why, I mean, why you make 10 requests? Why not hundred requests per second?

What is next? Do you have any dependent system that after this call should happen something, something else, something different. So the test engineer who is developing performance test script should think from the user perspective how users using their system. I mean the exact user, for example, right? System user log into the system, making some search, then filling out some form, then submitting some form. So user performs certain scenario. So as automation engineer, you have to find out those main scenarios for your application and then simulate those user flows, write the script to simulate those flows and then scale it. So that's the right approach to make a performance test and not just hitting and you know, destroying single endpoint of your application. Trying to find out, okay, when this blows up, right? No, that's not. How does it work? You need to think about user flows, think about the main key load scenarios for your application. Then write the script to simulate this flow and then scale it using the performance testing tool.

Nice. What's the flow like at your company then? So when they're doing UI testing, they use Cypress and then when they're doing API testing, they know they I need to now use Karate. And then are you able to leverage that Karate script and repurpose it as performance or do you need to rewrite it in Gatling? I'm just curious to know how, how your flow is and what the decision is at each point to know what tool to use for what, what functionality.

So we do combination of both. So one of our application, which is kind of complex, we use a pure Gatling framework. Why we use pure Gatling framework because it has more different commands and methods, how to tailor, if you will, your script to better simulate the user behavior, Something like that. But if you need more straightforward scenarios like sequence of the steps that you perform, the delay between the steps, or API to API call Karate works very well in this. Especially when your scope of performance testing is just single core API. Let's say you have, we are in the medical field, we have such as member API or provider API and we know to perform, we need to know how that core API performs. So karate works perfect there. So this is a sequence of 2, 3, 4 calls and you just write the script, put your karate API test in this Sequence hook gatling to this run it and it just works. It's just magic happened now after that.

Nice. So for performance testing piece do you run in CI cd? And I know a lot of people are trying to incorporate performance testing as part of when someone checks in code. It also runs like a high level performance test. It almost sounds like karate might be a good lightweight, quick performance test to run on check ins. I don't know. Is that something you would do?

It is, yeah. It's just one step in the CI cd. It's not part of the deployment script but is a separate hook that is triggered after the deployment. So when the performance test is executed, we just get the notification to email that hey, the performance test executed. Go and check the report of the execution and then you just compare the benchmark of the previous test run, current test run and between deploy. You know, okay, is the current build or last build? What's the difference between the performance is the main metrics are in the good shape. I mean the response time, the error rate and that kind of stuff. Then it means we good. But if we noticing some spikes or something like that, this is when we look and this is what requires our attention.

So I know one of the benefits of Cypress is it's an easy install so developers don't have to change context.

True.

How's it work with karate? Do they install karate on the machines? Is it hard to get karate set up?

It's very easy to be honest. This is a single maven archetype that you need to execute in the command line and the layout of the project will be built on your computer right away. So with a single feature file that you can execute right away. So you will have runner, you will have folder structure, you will have config file, you will have everything. And that's the good starting point. Just move from there so you can create a new feature file and start to write the test feature. This scenario is this when my URL is this one, when I make get request this one then the response 200 and the assertion is this and that. So you know, pretty simple, very straightforward.

So I guess sometimes people get worried with like you said it's not cucumber but some people can write really bad cucumber. I assume you create really bad karate code as well. How do you make it so you don't have a lot of things hard coded and I guess in cucumber language it'd be like a feature file. You could have like I guess thousands of feature files if you had hard coded give me 200 call this API endpoint and that API endpoint changes how does it make it so it's easy to update or maintain? I guess.

So you can use variables. Of course there is a support of that stuff and in Karate it is also feature files the same way. How easily cucumber. By the way, you can create different helper files. Helper features. You can create usable function. What's beauty about Karate? It supports Java and JavaScript. So you can use either Java to write your different methods or functions that you are probably missing or logic that you are missing or you can use JavaScript functions to write your custom code as well. That's going to handle this as well. So it's more, you know, up to you what you are more stronger in Java or JavaScript. You can use both if you will. And that's by the way one of the beauty of the karate that this natively supports JavaScript and JSON. You don't need any kind of custom parsers or something to read this data. So yeah, and talking about the paths that you say it changes. So when we talk about the path, we cannot say that it is hard coded. This is defined requirement what your URL should be or what the path should be. You define your base URL in the config file where you keep your old variables URL config but path to the URL is a separate thing. And this is separated in the karate. You have base URL and you have path that is appended to the base URL. So your base URL keeps you keeping in the config file. The path is in the part of the scenario.

Absolutely. I guess a lot of people they like to blame tools for behavior when it's really the way they design tests. That is usually an issue. So as long as you set up your karate test correctly, you're not going to have. It's like any. Any other tool. It's. It's only as. As good as how you design your framework to actually utilize.

Exactly. Yeah, you're absolutely right. So the test design and test architecture is a separate piece of knowledge that you have to figure out, okay, how correctly to design the framework. If the framework designed poorly. It's nightmare to maintain this stuff, even if by nature this framework designed pretty well. You know, I have students that send me sometimes their screenshots of their framework. I'm like, oh my. Really? Why did you do this? Why are you calling your JSON file from the feature file? So it's easier to keep your JSON in the JSON file, not in the feature file or text file. Why you complicate your life by yourself. So that kind of stuff and. But you know, it all comes with experience. This is what I can tell. So the way how you can learn it the best way is just to write the code. Look the best practices read documentation people for some reason not reading documentation. GitHub for the karate Peter Thomas, who is developer of the Karate framework, did a great job to put all the documentation a bunch of examples of different codes and features how to implement the best practices with Karate. Just don't be lazy. Look at this stuff. It's. It's really good and just going through this you can find out the best approach and practices, how to implement.

That's a good point. I. I've interviewed Peter Thomas a few times. He puts a lot of work into this tool and it's always changing. He's always making it better. So I guess that's another indicator of whether a tool should be used or not is how active it is. How active is the main contributor. And it seems like he's really kept on top of this since he released it for sure.

Yeah, you know, and he keeps improvement of this framework. So recently Rudd is supporting UI automation also so you can automate browsers as well. So it has mock supports for your APIs. I didn't play around with it, to be honest. I know a couple of folks in our company using it. I did not yet so probably add this in the nearest future to my Udemy class. So it has a lot more than just API testing.

With performance testing. Then how do you know when to use Gatling? Does it take care of that behind the scenes or do you create like a simple performance test in Karate and then you say okay, wait a minute, I really need to scale this up for a lot of concurrent users. So therefore I need to leverage Gatling to get that functionality.

You know, it depends on the scenario. So so far in our company we have just single framework that is in pure Gatling. Just because of the complexity of scenarios of the user flow. Only because of the user flow complexity. That that's just that. And eventually it was written in Karate. And then I found out that I am missing some useful methods that Gatling has that is not supported by Karate and Gatling integration. Because Karate plus Gatling you can just scale technically your API tests. So you have one call, you can make ten calls or a thousand calls or ten hundred calls per second, that kind of stuff. But in pure Gatling they have More built in methods in framework, how you can for example drive your traffic or split your traffic to different scenarios, those kind of things. With Karate plus Gatling you cannot do this, unfortunately.

So I guess the interesting question here is then why did you choose Gatling? Was it because you originally were using Karate and I happened to use Gatling or did you look at the other open source tools for performance testing like K6 and whatever and you realized for your particular company your scenario Gatling fit better.

Exactly. This is what happened. So we used Karate. Then we found out that Karate has support with Gatling, we started to use that and then when we faced with certain limitations that we needed, you know, more tailoring for our scenarios, then we wrote a pure Gatling script.

Nice. So can you talk a little bit about your Gatling framework then? I know you said you had to make it separate. Is it, was it hard to set up Gatling then as a dedicated performance solution? Any best practices with it?

It is harder. Yeah, it is because it is require more.

Technical skills from us from engineer. It's not that straightforward as Karate or Cypress to set up. You know, when you look at Gatling documentation on how to set up Framework, it's not really straightforward. It doesn't have the magic, I don't know, Maven archetype or NPM install package or something like that. So you, you need to configure this and Scala scripting. It's also not really often you can find the Scala language usage anywhere in any framework. So it's all new learning curve for automation engineers. When you see the Gatling for the first time, that's why. And you just write pure script in Scala. Yeah, and as I said, it's just not really straightforward and not much guidance you can find online. You just read in documentation, write the code, try it fail. You try this, it fail. You try. Oh, here we go. It's working. Okay, next step they need to figure out, okay, how to pass my headers, how to pass my, I don't know, token, how to get the variable and pass it to the next scenario and you know, each step. It's complex. I would say it's not that straightforward. But what's the benefit of Gatling is it shines when it perform actually. So it use Akka under the hood so it can scale the number of users like dramatically. You can use the single laptop to simulate thousands of calls at the same time without needing a powerful server or powerful machine. So it's very powerful when you run it. But when you write the test it can be pain a little bit.

So when would someone need know when to use Karate's for performance rather than straight up Gatling? You mentioned there were some functions you realized you needed, but are there any other scenarios where you think someone would be completely fine just using Karate and you would use Gatling behind the scenes?

Those two, I think it's covering all our needs in the company. So we also experimenting here with Locust framework with K9 and we decided to stick with Gatling still. So we did evaluation of those tools as well and decided okay, Gatling so far looks best for us I guess.

With Karate though, how would you use Gatling as a keywords or do you just say how does it know when to use Gatling for performance tests?

Very simple. You just change your command in the command line to run Gatling test and you write a simple Scala file where you say, hey Gatling, run this feature file with my API test. That's it.

Nice. So does karate then if you ran just API functional tests and then you did a performance test, does it combine it all into a report in Karate.

That you can see it's a separate report. So Gatling will create own Gatling HTML report for performance testing separately. So API testing has one report, performance test has separate report.

Great. So I know you've ran a course. I love giving people work when I don't have to do it. You ran a course on Cypress, you ran a course on karate dsl. I think there's probably a need for people to know in general performance testing. So how much do you think people when they come to karate for performance to use it for performance testing, that they know enough about performance testing to even leverage the tool correctly? Don't know if that makes sense. But like how would you, what would you recommend someone know about performance testing before even trying to jump into using any tool for performance testing?

As I mentioned earlier today, first know what you are trying to test. I mean what you are trying to get out from the performance testing. So performance test is a simulation of the load. So before writing any script or choosing any tool, think about what are you actually trying to simulate. So how many users which endpoint, what user flow? Think about it in advance before you touch the script. And when you know what you try to simulate, you choose the tool.

Absolutely. Great advice. I know you have over 2,000 students. It's basically a five star rating for the course. I'm curious to know, you probably get a lot of questions. Are there any common questions you get that you'd see people struggling with or doing incorrectly in regards to performance testing specifically.

You know, surprisingly, I'm not getting a lot of questions about performance testing. Maybe it's pretty much straightforward because really so the part of the performance testing covered in the course within less than 10 lessons or so. So it's pretty simple. The biggest part is the about Karate framework itself. How to write the script, how to configure the script. But when you come to the performance testing, it's technically you are using all the knowledge that you've got from the karate and just scaling it with Gatling. That's it. So to be honest, I don't remember the questions about the performance testing for my karate class. Interesting. Good question though.

Yeah, you have to look into that and let us know when you come back. I guess the next question then is you mentioned this karate Peter, he's always modifying, he's always making it better.

Right.

I guess with that there's probably functionality that people may not even know exists. This always drives me crazy about people as well as there's a framework that has built in functionality to do something and you look at their framework and they've created their own implementation of something that's already part of the tool because they didn't know it existed. Is there anything part of Karate that has that, that you've seen that you like? How come people aren't using this, don't know enough about it to know it's in there that can help you with testing either functional or performance testing wise?

Yeah, as I said not and probably not about the performance but.

You know, probably some, some very basic stuff about, I don't know, reusability. For example, you know, people keep copy pasting the code instead. Hey, you can call this feature just again and it will return you the result. Why copy pasting the code or what? I noticed people just not going attentively enough through the class and when they go to the lesson number seven, they start making mistakes because we did, they didn't pay attention on the lesson number five where I explain certain basics in theory that is needed to move to the next to the next step. And this is where probably mistakes start to happen and they're rushing copy pasting, not using optimal way of organizing and structuring your code. And then in one month of this not right direction they have kind of framework that's hard to maintain and hard to manage. Yeah.

So Adam, are there any concepts you think should be summarized get people to really understand the concept of either performance testing or using karate dsl?

Yeah. So let's start with the performance testing. So the key concept of the performance testing Think about your scenarios, about the user flows, about what you are trying to test before you write the script. The good way is to go to some monitoring tool of your production environment, let's say open new Relic and find out how users using your system in order to understand how much load and what kind of load do you need to create to your system to performance test and stress test your system. It will be a good benchmark for you.

Nice. Okay Arnold, before we go, is there one piece of actionable advice you can give to someone to help them the performance testing or Karate DSL efforts? And what's the best way to find contact you or learn more about Karate DSL API Automation and Performance testing.

From Zero to Hero when we talk about the overall frameworks UI automation, API automation, it doesn't matter. Think about the architecture of your framework first. Think about the reusability if you repeating your code. If you copy pasting code at least three times somewhere, stop right there. You're most likely doing something wrong, so think how you can do it better. If you're noticing copy pasting, stop right there. This is my advice number two. My advice number three Read documentation guys. It's your friend. So don't think that you find always on the YouTube a better guy who can explain how to do the best way. Just go to documentation first. Read it the developers of the frameworks they already created, most likely for you the best practices the best approach how to use this tool or that tool. Read it first and then find the additional guidance if you need it for.

All right. You can also find me on the LinkedIn. Of course it's Art and Dashbondar. I'm always there and I respond to pretty much everybody who is sending me messages. Also udemy.com, this is where I posting my classes. They are only on Udemy by the way. And that's the two main channels how you can contact me and ask me any questions. I'm always open.

Thanks again for your performance testing awesomeness. Links to everything of value we covered in this episode. Head on over to testguild.com P69 and while you're there, make sure to click on the Try them both today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UiPro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.