Behavior Driven API Testing with Ido Cohn

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast and today we'll be speaking with Ido cone, all about behavior driven testing and much more. Ido is the co founder and CEO of Loadmill, so really excited to dive into what his thoughts are on this topic as well as learn more about Loadmill. So you don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSourceLabs and click on the exclusive sponsor section to try it for free today. Check it out. Hey Ido, welcome to the Guild.

Hey Joe, thanks for having me. Super excited to be here.

Good. Great to have you. It was a really short bio I gave for you. So before we get into it, is there anything in your bio that you want the Guild to know more about?

No, I think so. I don't know. I started my career as a testimation and engineer so I feel like I've sort of like a part of this community for a really long time. I also like listen to your podcast now pretty often. So no, I think it's nothing special to add.

Awesome. So in the pre show I believe we talked about some main topics or your assistant and I talked about some main topics and something that came up that was interesting and I'm curious to know your thoughts on it. Is what are your thoughts on the test pyramid? Is it still relevant as we go into the new year?

Yeah, I mean so that's. I like to talk about it with most of the customers and I think it's like a super interesting topic. So obviously it's not like one size fit all, it's not perfect for everyone, but it's sort of like make a think about the priorities and the upsides and downsides of every type of tool we're using for test automation. And I think obviously if you have a very UI heavy app with a little logic on the UI side, it might make sense to have more UI tests. But it's always good to try to optimize and use the best tool for the job you're trying to do. And obviously unit tests have a lot of benefits. So you want to have your test automation based on Unit tests. And we especially see a lot of customers that have, I would say, like too much UI testing. So they're like very heavy on the top of the pyramid. It has a lot of downsides. So first it slows them down. UI tests are not always the best fit to run for every commit, for every pull request because they may take long, they are expensive to maintain. And we see a lot of customers that we talk to, we try to like push some of the tests, automated tests that they covered through UI testing, lower into the pyramid and try to implement them through service layer tests, through API tests, through all that. So even though it's not like perfect for everyone, I think it's like a really good practice to think about it and try to optimize your test coverage to be the most efficient way you can do it.

Nice. So you have a lot of customers, I speak with a lot of people. So I'm just curious to know what you've seen out in the field. Obviously, best case scenario is developers are doing testing all along the pipeline. But I think the reason why I see a lot of companies have a lot of UI tests, they have more like a tester type QA person involved and the developer's not necessarily involved in testing. Do you see that as an issue, as maybe why there's so many UI tests rather than other types of tests in the pipeline?

Yeah, I think I've experienced that myself because when I started I was a manual qa. And when you get to your first automation job, I think you're used to think about the product through the UI like the customers usually do, and you're thinking about automation. So let's try to automate everything that I used to do as a manual tester and click all the buttons, just do it automatically. And once you sort of like free yourself from the thought that you have to do it through the ui, you understand that if you have like, I don't know, maybe a form, but it has like hundreds of variations, I don't need to actually like re click the form every time. I can do it like automatically through the backend. So you start to understand that there are like other layers you can automate through. And also I think so, because in many companies also the automation engineers also went through this transition, being first manual QAs and then transitioning to automation. I don't know, it's sometimes hard for them to look at the backend of the application to understand the APIs involved. And actually this is something we're trying to do at Loadmill. To really help them look at the backend APIs and understand what they're doing. Also when you're trying to automate stuff, UI scenarios are very replayable. So every time you click the button it's like the same button. But if you're working with API and the values in the API and the queries are changing, it's sometimes harder to automate. So I think it's sort of like there used to be like a lot of UI testing and a lot of companies when they start new automation projects, they start by looking at through the UI and automating through the ui. But I'm gradually seeing like the shift to like lower down the pyramids to automating more through the backend and even more like with automation teams. Something that was used to be done only by developers and now more automation teams are also getting into the API layer and stuff like that.

Nice. So you know, I've been trying to promote API testing. I wrote a book in 2014 using an older tool doing API testing. But I always get into still today I still get into discussions. Well, you shouldn't use API testing because you're not using the application like a real user would. So do you still hear that type of objection or do you see people understanding now API testing is just part of the whole test suite and you may have some still high level UI tests that go to the actual forms and such as if they were a real user.

Yeah, I still hear that. But I think most companies know that. Especially if you are like an established company and you have like an automation team. It's not that you will have one tool that will fix all of your problems. You have to test through different layers. I mean you probably don't hear that when you're talking about like unit tests. It's obviously that you have to test like the tiny bits of logic, right?

Yeah.

And also I think many companies discover that it's not really scalable testing all the many different variations. You know, they're like it's impossible to do everything through the ui. And you know, look at, there are a lot of stuff that you can do to optimize even UI testing. You can parallelize like the execution and do a lot of stuff, but in the end it doesn't scale. If you have a form that you have to fill with like I don't know, 100 variations, doing it through the UI might take like, I don't know, 20 minutes and doing it through the API can take like one minute. So where I think more people are coming to understand that. But still the adoption is not where I would expect it to be. See, even now I think the go to for many automation projects to start will always be through UI automation. But I think there's like a gradual shift.

I definitely agree. So you know, along those lines, I guess you said there were some challenges, see, even with API testing. So are there any common API testing patterns you've seen or challenges from your customers that you think this is something that needs to be resolved to get really more effective tests?

Yeah, so it depends. There are like several different approaches to API testing. You can test every API separately and try to test all the different variations of every API separately. But then it sometimes kind of get hard to like set up the data to be able to test the application this way. And we're more inclined towards like, sort of like, I would say like the behavior driven approach where you have like an end to end flow of API requests, where you're like, basically, I don't know, you order a plane ticket, you change it, you delete it, you do like a flow of actions on a certain entity. And when you come to do that, you face a few challenges. So the first challenge is not all the internal APIs are always very well documented. So if it's like an external API that's used by external customers, it usually will be pretty well documented. But the internal APIs are not always so well documented. And it's a challenge for the QA team. Start to talk with the engineers, understand which API returns which value and how do I use this value in the following API request? And I think so this is the challenge I personally had when I was working on test automation. And this is what we're trying to solve here. Basically we're trying to like look at the recordings of API calls, trying to sort of like backtrack and analyze. So this value comes from that request and sort of like create the test for you. So this is sort of like one of the challenges I see a lot of people facing. Also trying to map the APIs for you and create the documentation, show you the list of all the APIs that you have in the system. Sometimes it exists already developed by the developers and sometimes you have to work really hard and sort of reverse engineer your own application to understand which APIs are being used. This is one of the challenges, One of the more recent challenges that we see is there are a lot of technologies involved. So you have RESTful, FBI, GraphQL, GRPC, and also WebSockets involved. And sometimes in the UI, you probably won't even notice that because they're like working together and everything is fine. But if you're trying to mimic that at the API layer, it's sometimes pretty hard to understand how the data is flowing from the REST API to the websocket request and stuff like that. So that's another challenge we see.

Nice. You mentioned a recorder. So I know that when I speak with a lot of folks, when they're doing microservices and they're trying to do a flood, they don't know, like you said, it's not documented. They don't even know what dependencies are on a particular API that calls out to another API. So what is this recorder? Can you talk a little bit more about the recorder you mentioned?

Yeah, I mean, so we actually developed a few different recorders the way that in the heart of our system there's this algorithm who analyzed the recording. So, and the recordings can come in like many different forms. It can be just like an HTTP dump and it could be like a HAR file you download from Chrome, for example. So, so we basically built some of our few recorders of our own and also using free, several open source ones. But the most popular one that we have is basically like a Chrome extension that you install in Chrome. It opens as a new developer tab and when you open the devtools in Chrome and you can see all the API calls being sent, you can upload them to loadmill to get them analyzed and see the relationships between the different APIs. So actually recording the API calls is not such a difficult task today. You have a lot of tools to do that. You have a lot of HTTP proxies that you can use. Even if you want to record the HTTP traffic coming from mobile apps, there are a lot of solutions to do it. So we integrated with a lot of these tools, you can basically export the data from them and upload them to our platform. We also see another interesting thing we see. So even if you use user monitoring tools, tools even like fullstory and stuff like that, you can also look at like real user sessions and export their network traffic and analyze the APIs from them. So sometimes you can even look at session of customer who had, I don't know, some bug or something like that, you can actually like export the scenario that they were playing and upload it into, for example, loadmill and create an automated test based on that scenario. And I think it's really interesting because it's sort of like the ability to look at the recordings and Analyze them from QA perspective, sort of like bring you closer to the actual customer and being able to replicate behavior performed by customers. We all know that sometimes it's very unexpected how customers are using our apps. So for me especially it was really interesting.

Yeah, it's a great use case. Even just having the customer do that and send it to you. You'd probably find some good test cases you weren't even aware of. Yeah. So another hot topic is AI. So you mentioned recording, that's the easy part. But the analysis piece, I don't know if you have this in your tool or something, but do you use AI at all? And if you do, is it used to analyze all this data or how is it implemented?

Yeah. So I think in general, if I look at the bigger picture, AI is sort of like a really hot buzzword in test automation and QA in general right now. And there are many different use cases for AI in test automation. So we've seen many different companies using it for test stability and making the test more stable, applying really smart selectors for UI testing, stuff like that. There's companies who are doing it for impact analysis, like analyzing which tests and impacting which code changes and optimize for test execution. And we use AI for the test creation itself. So we basically look at the scenarios that are performed by users in your app or even played by QA engineers on one of the recorded applications. And we sort of like try to analyze how the API interact with one another and create a test for you. So we actually basically write the test code for you, we display it in sort of like a low code UI for you to edit. Because I think, I don't know, no one really likes to read other people code, especially not code that was generated by an algorithm. But so I think this is really interesting use case for AI in test automation, like actually helping us with the test creation. Obviously there's still human work needs to be involved, like adding some assertions and I don't know, maybe tweaking the test a little bit. It's really fun. I mean, the algorithm can basically create like the skeleton of the test for you, like do 90% of the hard work for you. And then it's just like the human factor to add like the insights and the smart elements on top of that. So yeah, we do use UI for that. UI is always some like somewhat of like a vague term when you talk about like solutions like ours, but it's basically sort of an algorithm that create a test and sort of like gets better over Time, the more recording it sees of your application and the more feedback give back to it when you tweak the tests. But I think in general, we'll see more of that coming up. Like, in general, like, a lot of companies are trying to sort of like, target this market and introduce a lot of smart AI solutions. Yeah.

So nice. So I'm curious with that AI or in general, a lot of times people say, oh, I'm a tester, I know what I need to test. But I would think with API, like you said, because they have so many dependencies, it's not well documented that you may have a lot of tests that supposedly cover the API, but you may have a lot of gaps. So when you do the scan and you're analyzing, are you able to say, okay, you have coverage in this area, but you're missing this part, and this is really critical. Is anything like that built in?

Yeah. So we map all the APIs we see when you record a test, and we sort of like create a test for you, and we'll also show you a list of the APIs that you have covered in your test. So, so we can tell you what we did see and what you have tests for. And it is something we're going to put a lot more emphasis in the future. One thing we don't know is if we have APIs that you didn't record through us. So we actually, they don't know that they exist because we don't integrate, like, from the side of the code. We actually, like, just see the network layer. So we only see, like, the actual behavior that happens. It also raises a lot of questions about, like, will loadmill help me catch, I don't know, bugs in scenarios that no user ever performed? And so the answer for that is, first it's, is it important to catch, Is it like the top priority for us to catch bugs in scenarios that no one ever performed? So I would say it's still important, but the toppest priority for us is to actually be able to simulate the user behavior to catch bugs in what they actually do daily. Because I think when we're talking about regression testing, what we're basically trying to answer is that is everything that my users were doing yesterday still going to work for them tomorrow after I'll release the new version? And you always have to prioritize. You can't test for everything. And if you have to prioritize, I believe that you have to prioritize for what your users are actually doing. So obviously it is interesting to catch like edge cases in very rare conditions. But I guess you will still need human beings to be able to do that. So. Yeah.

Nice. So are developers involved at all with this? Like when they write code, do they get feedback if it's an API quicker? Because a lot of times people say, oh, there's too many tests that slowing me down for checking in code. Is there any type of integration where developers get faster feedback because it's already checking it as they're maybe merging up or something?

Yeah, so we do work with some developers who are think like the full ownership model or the developer themselves are writing the tests. And we still see a lot of teams that are working with like dedicated QA engineers. But I have to say that in order to run API tests, you have to have like a deployed environment. Unlike unit tests, they can't run against static code. Right. So we can do that if you have like a locally deployed environment, sort of like a hot reload every time you reload the code into your like local server or something like that, we can run the tests. We have sort of like an integration to do that. But I have to say that most of the time the tests that we create are like best fit to run. In the CI especially, we see a lot of customers running them against like dynamic deployed environments where you have maybe for every Pull request on GitHub or GitLab, they automatically deploy a new environment with the change and then they automatically run the API test against it. So it's still pretty fast feedback. You don't have to wait, I don't know, all the way to the staging environment or something like that. And you see it sort of like you can't merge a pull request until these tests are passed. And also API tests run pretty fast and usually you won't see. You can see like hundreds of tests running in sort of like, I don't know, five minutes. So it's, I feel it's not too long to wait for before merging a pull request if you have to wait five minutes. Sometimes, you know, if you have like UI tests that are running for 30 minutes, might be a hassle to wait for them. But yeah, so it's sort of like an in between thing between unit tests where you run them locally even before you commit. And these are the tests that you run before you merge your pull request.

Nice. So, yeah, like you said, it's a faster validation for every code change almost where they're not waiting around, but they're getting that feedback that they need to. So, you know, another hot topic Always is testing and production. Don't know if you have any thoughts on it, do you? Something you recommend or something you see your customers doing?

Yeah. So obviously testing and production is a big thing. I mean, a lot of customers are talking about it. We like to say that what we're helping customers are doing is basically like bringing the production environment into the testing environment, because we can, like, capture user scenarios from the production environment and bring them into your testing environment. But we still do see customers actually using our test to test in production. It has a lot of complications involved with that because if you want to test in production, there are like, security and stability issues involved. You don't want to mess with the production environment too much. But we do see that happening. And I think you want to make sure that what you're doing is actually working for your customer. You'll have to test it in production. Maybe not all of your test cases, but still maybe like your sanity suit. I know that it's. I don't know, it got somewhat unpopular lately, but I still really believe in good staging environment. We've talked with a few companies who've built, like, amazingly realistic staging environments where they have, like, all the services deployed, they have real data, simulated data, and in the staging environment, and they're getting, like, really good results from that. So I still believe that you can get a lot of value from the staging environment and it's worth investing in. But yes, sometimes you want to make sure that what you're doing actually works in the production environment. So we see customers doing that.

No, that's a great point. I've been approached by a lot of different companies now that have environments as a service and the focus on staging, be able to throw up these environments really quick and tear them down. It really makes sense nowadays. And back in the day when you actually had to have bare metal in place, for sure.

I mean, it's pretty easy to achieve today. I mean, there are, like, amazing tools to do it. There are a lot of companies and we have that internally. And I actually can't imagine my life before we had like, dynamic testing environment where I can just change something in the code, deploy a test environment. It's also like you can give it to the product manager to have a look. Is this actually what you meant? It's extremely useful, and it's extremely useful for test automation. You can run your API, test, your UI test against these environments. I'm a big fan of this approach.

Nice. So I should have dove down into this a little bit more. So I Just want to switch gears. You did mention you started off as a QA engineer. And I love speaking to founders that were actually engineers. So what made you create loadmill? Was it something you were noticing as a QA engineer and you said, oh, I could do this better or what was the path your story from going from QA engineer to say founder of loadmill?

Yeah, so yeah, I started as a manual QA engineer and then I switched to test automation. And after that I was a developer in a startup called Nolio. We're also sort of like a DevOps platform for the enterprise and we were later acquired by CA Technologies. And then I was an engineering manager there. And so it's not like, it's not that I came up with all these ideas when I was a QA engineer, but I think just starting my career as a QA engineer gave me like a really interesting perspective into how the work of a QA engineer looks like. And we're talking about giant Excel sheets with manual tests that you have to go over every day. And then when you do that, you understand how important is automation. Because when you actually experience doing day to day job of manually testing a lot of different features, it's become completely obvious to you that you have to automate it or at least some of it. And when we had engineers working on test automation back then it was written in Python and in the robot framework. We saw that a lot of the work they were doing was sort of like repetitive. So they're basically like writing these API tests, integration tests, and there was a lot of dirty work involved, like basically doing a lot of the work again and again. And there was, you know, a lot of back and forth talks with the developers, how this works and try something and it doesn't really work for you. And then I don't know, the developers changes it and your test breaks. So I actually really experienced the pain of creating that. And also I found myself constantly when I was an engineering manager feeling that we don't have enough test coverage. You always feel, I mean, at least we did that we don't have enough test coverage, that we have too many escape bugs that are getting to production. And also the feedback loop, it felt like we don't actually know what it's doing until something gets to the customer. I actually experienced that actually back then it was like an on premise product. You actually had to fly to the customer, install it on their servers. And after you put a lot of work, like two months of work working on that version, after five minutes when you install it on their servers. You find obvious problems that you wonder why we didn't catch when we were testing the product. And this gap between how we build the product and how the users are actually using it was always really interesting for me. And we tried a few different ideas before we came with this one, but basically always the idea was to get the data from production and leverage the information that we have about how users are using the product to create better tests and to actually get this fast feedback loop that we were looking for. So we were actually building a product for ourselves, but it became what it is, what loadmill is today.

And what is loadmill? I don't think I asked you that. So what is loadmill?

So basically this is what we do. It's, we're focused on backend and API testing. We analyze user behavior and we turn it into automated API tests. We're an Israeli startup working with more than a few of like the biggest companies here in Israel and some companies around the world. And we're basically all of us were like, have a background from the test automation space. We had some people coming from what used to be HP Software and Mercury working on loadrunner and tools like that and CA Technologies. And so yeah, that's us.

Okay, note before we go, is there one piece of actionable advice you can give to someone to help them with their API testing efforts? And what's the best way to find you, contact you, or learn more about load mill?

I think maybe it's like a bit general about test automation, but if you're writing automated tests and you're still triggering them manually, you're not like getting the full value. So you have to connect it to some CI platform, you have to run it, trigger it for every code change. Otherwise you're not getting the real value of having this fast feedback loop. Because the way I see it, we know that tests are about quality, but for me, it's also about agility and the ability to move fast. I mean, we obviously won't compromise for quality. We want the best quality for our product. But for me, automated tests are this fast feedback loop where you change something and you immediately know that you didn't break anything and you could move forward. So that's it for me. You can find me in a lot of places, on Twitter, on LinkedIn, but best to find us is go to loudmill.com and sign up for the product or book a demo and talk to us.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode Head on over to testguild.com a406 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Labs awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show, and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.