How to Create Testing Magic with Zero-code with Shubham Jain

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. And today we'll be talking with Shuham all about how to create testing magic with zero code using a tool I just learned about. And I think it's really going to help you with your automation, especially with API testing. Shoham is actually the maintainer of Kaploi, which is an API testing platform. API testing is becoming more and more popular and I think this is a solution that's going to help everyone be more efficient with their API testing. He is the former consultant at AWS India, contributor to X Wiki, and he also has led data and platform engineering teams across the world. He loves talking about developed productivity and next gen problems and solutions and he's going to share one of those solutions with you today that I think you're really going to get a lot of value from. You don't want to miss this episode. Check it out. Hey Srivam, welcome to the show.

Hi Joe, excited to be here.

Thanks. Great to have you.

Thanks for having me.

Yeah, absolutely. I'm excited to learn more about Kaploi. Before we get into it though, is there anything I missed in your bio that you want the Guild to know more about?

Oh, I think that covered most of it. Something I think interesting would be I've been part of Google Summer of Code. It's a very interesting program run by the team at Google. It got me into open source, both as a contributor as well as later start mentoring for Google Summer Code with Xwiki. So I think yeah, that has been really phenomenal. And the team at Google Summer of Code is really cool. I really enjoy still being part of them. Part of the program with Kepler.

How do you become part of that program? Do you submit an application and then they go through it?

Yeah. So you can apply as an organization and once the organizations are picked. Earlier it was only open to students. So students apply and now I think it's open to more people. Yeah. Then the organizations pick based on the contributions and the student profiles.

Awesome. So here's a dumb question right off the bat. What's the difference between a maintainer and a contributor? Since you are the maintainer of Kaplo, are you the lead guy that says no or yes to the final build?

Kind of yes. Yeah. The maintainers obviously have more access, more involved in the decision making process once you've done a certain level of contributions and you control a big part of what's going on. So yeah, that's basically difference between the maintainer and a contributor.

Awesome. So what got you excited about Kaploi that you wanted to be the maintainer and contribute to it?

Well, so my team started it. So since we started, we had to be the maintainer to ensure it runs. Yeah.

Why did you create it then? What was the main issue? You saw that you like, there's open solutions everywhere. There's solutions vendor based but not meeting the need of this particular pain point.

You were seeing at my previous teams, what we were doing was we were mostly focused on R and D and you know, building out, trying out new ideas, taking things from ideation to users in less than two weeks. Right. And that's when the way our team worked did not. Essentially the way we worked back then was we would change things, we would deploy nearly on a daily basis. Whatever was out there and whatever we could try in terms of testing it required a lot of effort. We were like, okay, we want to test if our application works. We want to make small changes and deploy immediately. So can I just use my production traffic or what my users are doing and validate if I broke anything really quickly that eventually led to Keploy. Turns out everybody has. I mean it's a very common problem. Everybody wants to make small changes and validate if the application still works. And yeah, so we open sourced it and now here we are.

Nice. If someone asks you, hey, what is compleit? What does it do? What's the main tagline or the main feature feature that it provides developers or testers?

So it's the only EBPF based solution which can capture live traffic and generate test cases and data mocks which you can run as test. Like you said, it's also open source, completely open source.

So anyone can use this and is there a paid version or no. Okay, nice.

We are working with select companies as beta partners to try out deploying more complex scenarios where we have streaming or we have Async workers and things like that. But yeah, as of now, what's open source will mostly stay open source. It's all built on top of this.

Nice. So I'm trying to visualize this for the people that are listening here. So you run is it like a sniffer in production and it actually listens to real traffic for your application without you guessing? And then it will generate API tests for you that mimic exactly what's happened in production. Is that the main use case?

Yeah, exactly. That's what it does. It can be anywhere, right? It could be somewhere on a dev's laptop where I'm developing an application, I can call the API and still generate a test. Or in any live environment or a production environment where you could do something like that.

And this is important because as we go cloud native, we may be using services and APIs that we don't even know about. Like we may have a feature on our application like Submit and lord knows what it's calling behind the scenes. So actually having this in production is able then to create those tests for you and know what endpoints to interact with automatically.

Yeah, that's exactly right. What's crazy is we have seen users who have been using different SDKs, which they did not make. Turns out a very simple operation was doing like 50 HTTP calls or 50 database queries. Those were largely hidden. And the tests that they wrote earlier were mostly using mocks. So they would mock the entire interface out and never know about it. When you use Deploy, you're able to see everything that the application does, record everything like database queries, external calls, and rerun them exactly the same and see if the application still works. So apart from testing, it gives you a deeper visibility into what's really happening.

All right, what do people have to do to get this? Do they have to like instrument their application with open telemetry or something so you can get this type of data in production? Like what needs to be in place before they can use a solution like Kaploi?

Nothing really. I mean, so something I mentioned, ebpf. We use EBPF to trace system calls and instrument application without any code changes. So using Keploy is just you download Keploy, you do Keploy record and just run your application. And that would generate the test and mocks.

All right, so for folks that don't know what is EBPF then an interpreter for languages or something.

EBPF is Extended Berkeley Packet Filters. It allows users to trace system calls. So for example, if I'm using Linux, my application would be using system calls, do different things like read packets, execute any piece of code, get access to memory to network. So using ebpf we can instrument or change or understand the application behavior without actually injecting an SDK inside the application build. Traditionally it has been used for observability, especially by companies like New Relic. They have a project called Pixie. It's Also being used heavily in security because you can observe applications without needing to change the code. So. So yeah, we use the same technology to go one step further. Not just observe, but also simulate. And yeah, and so kind of use that for testing.

Nice. So I know one of the benefits is eliminates the need for data dumps. Looking through data dumps, stubs or mocks for dependencies. How's that done? Say you're using a third party dependency. It's in production, it's obviously interacting with it. But then you go to staging or development and you're like, I don't want to use that service, it's going to charge money or it's not available. How does it get rid of mocks then if are not in production?

Yeah, it's simple. Once you record something, deploy essentially acts like a proxy you can capture from any environment, let's say production, and then you can rerun it literally anywhere. So essentially when you use something like deploy, environments don't matter at all. You could capture different things from different environments, run them together in a single environment, and all that environment will be running is the application and deploy. So no databases, no external services, no internal services. So typically what this helps people achieve is they're able to within like within five minutes, run maybe 500 integration tests in their CI. And if you use something like GitHub Actions, you just. The entire thing runs inside an actions container. So you don't need any environment to run any of this. So yeah, like a use case.

Nice. So who's the main use of this? Is it. Would it be a developer before they do a main commit, they would run it through spin up, I guess an environment, run all the tests. Okay. Green promote to the main build.

I think that's the most popular use case we're seeing as of now because Keploy also integrates with integration testing libraries like Junit, Pytest, Go Test adds to your code coverage. So essentially normally writing integration tests is hard because it requires effort, plus it's slow, it's expensive. Kepler kind of helps you create those integration tests that also add to your test coverage, which are insanely easy to make. You just need to use the application. As a developer, I can run it as part of my CI. But while that's the most popular use case you're seeing as of date, I believe it can be used further in other use cases, maybe debugging or augmenting the testing process. Especially when it takes a lot of time to write or a lot of time to execute the existing Tests in any testing infrastructure.

All right, let's dive into that a little bit more how you can use it for the testing process. So you'd run it in production, you'd have the tests. How would you know what's being covered or how much coverage you have? So you have existing functional UI tests and they're slow and you're like, oh, boy, we could probably get a lot of this API testing, but I have no idea what is being touched by this ui. How do you know? Okay, I'm good. I have these 50 API tests and I know that it can replace these lower or slower running UI tests.

So the way that works is Keploy would, while it records tests, it can also capture the coverage information that, okay, what coverage in my code is this able to achieve? What Keploy can then do is it can use that and figure out, okay, what are the top 50 test cases from, let's say 10,000 or millions of test cases that I should run? Yeah, I mean, that way I can prioritize and set up, you know, running millions of requests. And since Keploy is able to also capture the data calls, it can actually isolate those 50 requests and actually run them in a lower environment. So that's basically the very, very high level idea.

Do you have any use cases of customers that have done that? And you're like, wow, that actually saved them a lot of X amount of time. I didn't know you'd be able to do that.

Yeah, so we keep hearing about different users from the community. For example, one would be Nutanix. So traditionally they would need to spin up virtual machines and they have. Nutanix has their own data center products and they have their own managed services. So all of them take time to spin up. So one interesting use case for Keploy is that traditionally, I mean, that takes time and effort to maintain as well as run. They were able to step out most of that and use Keploy as a proxy server and run the tests in like mere seconds, which compared to something which would take like hours because it has to orchestrate everything and then do the validations.

So does it save money on environments then, or containers, runtimes? Because you don't necessarily need thousands of containers spinning up and spending a lot of time being open. Does that make sense?

Yeah, yeah, yeah, definitely.

All right, so what are some other features then that we may have missed? You mentioned debugging. How can it help with debugging? Does it bug, bubble up insights or does it, like, how hard is it to get the data. I'm sure it collects a lot of data. And then you're like, okay, like how do you know what's relevant, what's not?

Yeah, so it's quite simple since it's again, depending on how somebody configures Keploy. So if I would use Keploy to capture something like capture request in a live environment and I would like to debug a particular request, I just need some way to get some kind of a request identifier, maybe some kind of trace ID that we see in OpenTelemetry. I can search that in kiploid. That should then give me that exact particular request along with everything that happened with respect to that request. So yeah, that would essentially help me isolate that particular thing and replay for debugging.

Nice. Have you seen people use this for performance testing? To see, hey, how does this service handle billions of requests? I don't know, almost a transaction based, without actually having to go into production, do anything nuts like that.

Yeah, that's a use case I've been thinking about. But I think that as well as chaos engineering, chaos tests and building different instances of application and see how it acts under stress, I mean we have not done that yet because we feel that the tooling needs to mature further before we can get into performance testing. Because I feel we would need to really, because we're emulating dependencies, we're emulating databases. So the ideal way this would work is when you record, you would also need to record those failure scenarios. Right. Or we would need some way for Keploy to emulate or you know, make a database failure, send a failure calls so those things can be explicitly done. But I think it's still early stages for Keploy for that particular use because I think it's a long way to go.

Is it possible to overuse Keploy where you just completely rely on it and say trust it completely. I don't have to do any full end testing in a full blown environment. I know. Is that a concern?

I don't think as like in the current state it can replace everything that we have currently. Because there's definitely value in testing with real endpoints. Right. I feel the value is more towards getting a quick response and being able to cover a lot more use cases than just core flows. So yeah, I think end to end tests, having real network interactions, those are valuable. But people tend to prioritize the core flows because you know, we have a lot of stuff to do. I feel there something like deploy can add value but yeah, there's definitely still value in running real network interactions.

Yes, I'm thinking like my old job, developers don't want to wait 3, 4 hours for end test to finish before they get the okay to submit. So this seems like a good compromise. We'll run all these tests. Looks good. Let's put it to the next stage of the pipeline. Then we'll run maybe our end to end test overnight or something to do the extra coverage. So it seems more like an assistant rather than a replacement then.

Yep. In fact, we have seen use cases where the way the team started using Keploy was they would convert the existing tests to Keploy tests and the Keploy test would run much faster, like in minutes compared to hours. And then they started adding more tests Keploy. So essentially most of the scenarios that would be caught by the regular testing pipeline was would be caught by Keploid first. Unless there's something very weird happening in the network. Right. Something which has not happened before at all. Maybe you updated the database or something external changed. Until something that very weird thing happens, it would be caught a lot earlier and a lot quicker. And then there's always the other pipeline.

How do you convert test to Kapli? If you had existing tests like you.

Just mentioned, you run the existing test suite and you record everything with Kiploy and yeah, there you have Kiploid tests.

Cool. It's that easy. So it doesn't matter if someone's using Selenium, Playwright, Cypress, it's just going to pick up the traffic and almost act like a sniffer like I said. And automatically generate. Yeah, nice.

Yeah, exactly.

You mentioned it's not fully mature yet to handle maybe performance testing. It may not even be something that's on your roadmap. Anything like AI or anything like that you see being baked in in the future. I know AI is a hot topic. Just curious to know if you're messing around with anything funky over there right now.

We have done a lot of experiments. Most LLM models today, I feel they're not really reliable as a test generator per se. Right. So they could generate something really funky, something they would imagine libraries. They are definitely not reliable, but what they're really good at is reasoning. And given a particular input, they could probably think of some really interesting edge cases. Something we've been experimenting with and trying with some of our beta customers is. Okay, you are using Kepler to capture and generate test cases. Normally you would have to do it yourself or get a live user traffic to do it. What if you could take your API specs swagger, OpenAPI 3, give that to an LLM model and yeah, ask it to generate different edge cases based on that and use Keploy to actually generate the real tests. So that way the syntax will always be correct. The mocks which are recorded by Keploy are real interactions which happened. And Keploy can also, based on coverage, filter out all the noisy and faulty tests. So yeah, I think that can be interesting combination. We're trying. That's where we are with AI.

Awesome. I saw on the website something called Keploy's accurate noise detection feature. What's that?

Oh yeah. So we've had this from the beginning. What we do is whenever we're recording something and replaying, there could be noisy parameters, right? There could be timestamps, it could be something which is what going to always change regardless of application behavior. So what Keploy would do is it would run whenever it would capture something, it would rerun it again. But when it reruns, it will mock everything out. Right. Similar to what it would do during testing. And then it would assert, okay, did anything change? Are the timestamps different? Are the any field which changed? And I know the application is still correct because it's the same application, same version. So that way I can find noisy parameters, not compare them during testing.

Awesome, awesome. We mentioned cicd. How hard is it to integrate? It sounds like easy, but I don't know, like how hard is it to actually get it in an existing software development lifecycle to plug it into as a plug and play?

Basically it's a plug and play. Just add keploytest as a stage. Just one line. Yeah. And then you're good to go.

Nice. I always ask, when it's open source, how's the community? Sometimes you get great solution, but you're like, ah, this community is dead, that's going to die. I'm not saying that's going to happen with Kaploo. How's the Kapplay community? How active is it?

I think it's going really interesting so far. We get different kinds of users. Most of them want to maintain test coverage because, you know, that's part of the job. He recently got a lot of community members from Japan. I think there's a hacker news equivalent there and somebody mentioned Keploy and yeah, we got a bunch of users from there. I think so far the community hasn't really gained. We've got a lot of feedback and yeah, so far, great. I think the community is still There.

So being a contributor and a maintainer yourself, how do you get people to contribute to open source? I mean, a lot of times people make a plane in the community, but it's like not many people actually take up the hard work of actually implementing something. Do you have any tips on how to get started contributing or how hard is it to contribute to the Kapoi project? If someone says, hey, I'm not a master developer, what can I do to help around here?

So what we typically do is we would mark it would label some issues as good first issues. Whenever we get an issue from a community user, we would try to break it down into different tasks which are relatively small and can be done by somebody who is not an expert. And Keplo is also written in Go. And we focused a lot on readability, so contributing is typically relatively easy. We try to document most of our methods and ensure that, okay, if somebody wants to contribute, they can go to the good first issues label, they can get started. So we get a lot of contributors, especially during GSOC and Hacktoberfest.

Awesome. So applications are always changing. How do you make sure your tests are up to date and comply? Is it just a matter of do you regenerate tests often? And because it's so fast to create tests like how do you make sure your tests are fresh and up to date? I guess with the latest and greatest releases.

Like you said, since it's easy to create, they're easy to update. So if something fails and you would get a diff, the user can decide, you know what, if this is the new expected behavior, you tell Keploy to normalize it and yeah, that's it. Your tests are updated.

Nice. Cool. If someone's listening to this, like, oh, this sounds great, but maybe I want to get some more in person. Kaipa see it in action is any events going on or online resources they can use to learn more about Kaploi?

So the first place to start would be our slack channels. We have a community there and we are active. The second place would be like if you want to have a more direct conversation, the best channel would be our community calls. So we have that monthly. Yeah, anybody can come and share their experience if they have a use case they want to get answered. That's another channel. Yeah. So I think those two are the best channels from the community.

Awesome. And we also have a lot of folks in India. I believe you're based in India and I know there's a lot of meetups that go on that they like to attend. Is there any live events that if someone is in that area? I know India is huge, but any live events they could actually attend if they're hearing this? Like I said, there's a lot of good skilled members from India.

So we have an event coming up in collaboration with Google to focus on open source contributions and getting started with open source in general. So it's in collaboration with the Google Startup team as well as the Google Summer of Code team. It's happening on the 29th of Feb in Bangalore. So yeah, I think that'd be a very nice event to check out, especially if somebody's interested in testing as well as open source.

Okay Shuban, before we go, is there one piece of actual advice you can give to someone to help them with their mocking kind of Kaploy automation testing efforts and what's the best way to find contact you and learn more about.

Kaploi Best way to connect against Slack we are available and I'm also available one to one and advice. I think that would be to keep mocks realistic to record real behavior. I have faced a lot of problems when I would assert my mocks or when I would make mocks myself, which are most of the times not true. I mean they're not the real behavior. So I think that has bit me a lot. So I think yeah, let's keep it as real as possible.

Awesome. Head on over to Kaploi IO and definitely check it out. It's also on GitHub actually and it has 3000 stars so it seems like it's well liked. So definitely check it out. Thank you so much for joining us today. I think it's a great solution that people definitely need to try and get their hands on right away. So thank you so much.

Thank you Joe.

Awesome.

Really fun.

Thanks again for your automation awesomeness. The links to evident value we covered in this episode head on over to testguild.com a 4x6 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you Join the fam@test guild.com and if you're in the DevOps automation software testing space, or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.