AI Self Healing Test Recorder with Eldar Kravetsky

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Elder, the CTO and one of the co founders of of Test Project, all about Test Project's new AI powered self healing recorder technology and more. Elder has over 15 years of experience in the field and has a passion for software development which began somewhere in the mid-90s as a kid and he's always been interested in programming. So if you want to discover how AI powered self healing capabilities and test projects, Smart Recorder can automatically handle test maintenance and debugging and keeping your test running smoothly. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Alde, welcome to the Guild.

Hi Joe, good to be here.

Awesome to have you. Before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

I touched many things during my 15 years in the field. I worked with some, if we're talking about automation here, I work with some legacy tools like QTP and TestComplete, you know, from before. Also created some automation infrastructure for non UI automation before I went into UI and when Selenium came around. So I got into it and started to do some nice stuff and playing around with it. So basically this is me.

All right. So before we actually get into the new AI features of Test Project, I know you are one of the co founders of Test Project. So how did Test Project come about?

Well, Selenium in Appium started to get more and more popular. You got to hear about it more and more and the community grew exponentially I guess. And we saw that in order to do something that provides you great options in terms of automation, but you still need to have developer skill set and not everyone has that or even wants to go into that. Right. So we thought maybe to do something around it. So this is like one of the reasons and developing infrastructures are hard because there are so many variables to consider and in terms of flexibility and maintainability. Right. So then we thought maybe we can put our experience into it like the Web applications, they started to be more complex and everyone started to move their business to the web. Right. Like SaaS, software as a service solutions. But test automation, it just. All the other tools, I guess, like stuck behind. I don't know how to put that. And there was no real solution that allows you to collaborate with your team in terms of test automation, which is very important for the quality assurance of your applications. Right. So this is the primary reasons, I guess. And we went into it.

Nice. Now, I know you mentioned you use QTP and Test Complete earlier in your career. I actually started my career using WinRunner and tools like that. And when Selenium came out, I kind of. I know this is not a popular opinion, but when I came to Selenium, like, Selenium is just an API. It's missing all this functionality I had in this other tool where QTP had a recorder integrated with Tessa with Quality Center. So we were able to have requirements and everything managed. The API of Slim to me was more complicated because you have to create everything from scratch. So it seems like Test Project is almost a free version of a. What would have been a vendor tool back in the day that has all the infrastructure baked into it. So when you start, you're not starting from level zero, is what it seems like. Is that Mike correct? How someone can interpret what Test Project is?

Well, sort of, I guess. But we are much more flexible and we're going ahead much faster. And I guess all the legacy tools got a little bit like behind because they didn't adapt in time.

So we moved to the web, right, to be alongside with the, you know, the greatest and latest technology. And so we are moving pretty fast. And so I don't know if we are like these other tools, but I want to believe that we are much better than that.

So talking about adapting, one thing that's been in the news or in the testing field for a few years now is the buzz around AI. I believe you have a new release of the Test Project Recorder that incorporates AI. So how do you explain what AI is? Because I think a lot of people have a misconception of what AI is in testing before we actually get into that new feature.

Well, if I need to sum this up, we probably several complex algorithms that will evaluate what's happening during the recording and when executing and make some of the decisions for you.

Right.

So we are recording things when you create your tests and we store some information in order to get a better understanding of what you are trying to achieve. And once you execute your tests afterwards we just, you know, doing our best to make your test pass, of course, without any false positives.

So this feature, when did it come out? Was it this week? Like how recent is this AI feature and what does it actually do within test project?

Okay, so we'll, we have the option of releasing some of our features to like doing pilots and releasing them to specific users because of the community that we have. And there are many users that are willing to help us to test new features and early adapters. So it was like I think a month, maybe a little more than a month ago, we released it in the closed version. So not everyone got it right away. Right. We just enabled it for specific users and we saw that some of the users were struggling with the specific things that didn't work for them, like iframes for example. So we gave them this new recorder and eventually released it for everyone. Around two weeks ago, I think. Something like that and around two weeks, I guess.

Great. So I think it actually helps in three main areas. I thought we'd dive into each one really quick. First one is self healing. Can you talk a little bit about what self healing means with test project with this new AI powered technology?

Well, the thing is that in many cases test automation will start okay and will run and everything will be just fine. But over time you will see that each new version of your application that is under test released, things are starting to break up and in some cases we can actually prevent your tests from failing and still achieve the goal. Right, so basically this is it trying to sum this up.

All right, so I guess basically what happens is that if someone's using an ID or an identifier and when it's running at runtime, it doesn't find it's able to self correct itself or find an identifier that is able to find that element and then carry on.

Yeah, exactly. So when you using during recording, we will actually evaluate each element that is being interacted with, that you are interacting with. And we store all kind of locators which you can later on.

Manipulate and change or enhance or even add some of your own. Right. And during execution we will try to find the elements and to interact with the element in different ways. It's not only the locators, but we like developed some algorithms to try and reach the elements and interact them, like execute the action that was required and recorded. So yeah, I guess this is the way that we do it in a very high level.

Awesome. So I believe also it's supposed to help with automatic test maintenance and Debugging. Can you talk a little bit about how that actually helps with maintenance and debugging?

Yeah, well, besides the fact that your tests will continue to pass. And let's take for example when you run your tests and let's say overnight, and you go into your test reports at the morning and just to see if everything passed and how it went, you will see indicators on some of the steps that self healing was applied. So this means that the test would have failed otherwise and we just, you know, did something to heal this. So it will allow you to drill down into that step and see what really happened. Right, so was there a pop up or maybe like a hover menu or the original locator? The ID has changed, so we use some other strategies to find your elements, stuff like that. So this is like when you're troubleshooting by looking at the reports. And in addition to that you can open your test in the recorder once again and then you run it. When you run it like that, you will see the indicators right away and we will suggest better locators. And by click of the mouse you can automatically update your elements and tests. So basically if you have like say a submit button that something has changed about it and then we were able to find it in a later run, you will be able to just, you will get a confirmation window and you will be able to update this element for all the other tests that are using this specific element. So basically you can debug and update all of your tests from one place by a single click.

Yeah, I've been playing with this for a little bit, the recorder, the new one, and I was really impressed by it, especially against a Salesforce application, which is notoriously hard to automate because they have a lot of dynamic elements in iframes. So I guess another part of this feature is the AI. But also I believe you introduced support for things like, I think it was iframes and pop ups and dynamic elements. Can you talk a little bit about why that's so important or like how Test Project handles those types of elements?

Yeah, well, at some point in time I think people were sure that iframes are going away and this is like a thing of the past, but apparently it's here to stay and people still using them and the more complex applications like you mentioned, Salesforce and et cetera, but not only them. So because of that we know that many of our users in some cases were not able to easily automate whatever they wanted because like Test Project always had the option to automate Iframes from coded tests.

Right.

You could write your own coded test and do that easily from there. But not everyone has, I guess, the skill set like we mentioned before, and it becomes like more. It's a harder process, I guess. Right. So the new recorder will handle all of that for you in the background. Right. So if you are familiar with Selenium, and I know you are, there is a concept of contexts, right. So you need to switch into the iframes and you need to know where you are all the time. Right. So we are tracking that and we are, we're sending the commands to the correct context and you don't even know that this is what happens.

Right.

What's happening in the background. So basically this is the thing about iframes that we had many complaints and we created. One of the reasons for the new recorder was the best iframe support there is.

I used to work for a very large enterprise and we had a large team, even with developers and one application that had multiple screens and you had to change context to test different flows. It was highly unreliable just because, like you said, the folks really didn't have the skill set. Put it right. So it sounds like the recorder handles all this for you behind the scenes. So I'm not saying it's magic, but it does a lot of the heavy lift is what it sounds like.

Yeah. Yes. Like we put in a lot of work, but for the user at some point it may look like magic.

Right?

Yeah.

So, you know, another thing the team struggled with was pop ups and a lot of times they had to use like auto it to handle like a pop up that was like a window pop up. So when you talk about popups, does it also handle that type of pop up or is it just browser based pop ups?

Well, mostly browser based pop ups, but when we created the mechanism, overall we've put in some base infrastructure and the new recorder and all the advanced features. This is not the end of it. Right. So we are constantly improving on new ways to locate end elements and tackle various problems like pop ups, dynamic elements, etc. So I expect that in the near future we will provide a better solution for each of these cases.

And I guess another thing that people have been struggling with modern applications that's become more popular is shadow doms which make it tricky to interact with the elements. So does this AI or this new recorder help identify shadow DOM elements as well?

Yeah, in some cases there are some complex applications and like you record something, you interact with the element. And while Selenium Executes it. It doesn't work because it actually expects something like a live interaction with the element. Yeah, we typically know that this is the case and we will try to do some behind the scenes magic to make that work for you eventually. Right. So Shadow Dome, you see one element and when you move the mouse you see a different element. So you recorded something and then when you run it, it's not there anymore. So we will deal with that in most of the cases, but it will probably get better and better with time.

So when someone's coding the Selenium API, they need to know, hey, I'm dealing with an iframe, I need to know, I'm dealing with the Shadow dom. When they're using Test project, do they need to actually be aware of, hey, this is an iframe, so therefore I need to use these methods or this is a Shadow dom, so I need to use this other method or does it do it for you? It decides while it's recording what to use and able to handle it.

Well, when you use our recorder, you don't really have to follow that and know that each step that is recorded is like assigned to a different iframe. If it's within an iframe. And when executing we will know to switch to that iframe.

Right.

So basically, after all, we are built on top of Selenium and Appium. Right. So we will use selenium code, selenium APIs, with all the things that you have to enhance in order to achieve what you want. So we will do it for you. You don't have to be aware. But still in the advanced section of each step, you will be able to see if there is some other context like an iframe that is being used here. So you will have like advanced control if you need to adjust something over time or maybe change something.

So I actually think that's an important point. A lot of people think, oh, this is just a tool for beginners, but like you mentioned, it's great for beginners, but it's also great for experts as well because you can go in and tweak everything as well. It's not like a black box that you're like, oh, we're not showing you what it's doing. It shows you what it's doing. You could tweak it if you want, but it seems like the recorder after each iteration is getting better and better, so you probably need to do less and less tweaking.

Yeah, that's for sure. And we will probably release mobile recorder with features like that in the future. Right. So currently we are tackling web.

Because we already have like a great recorder for Android and iOS, but still there are some things that we can enhance there. But yeah, you're right, we will do most of the work for you, but if you are a more advanced user, you can basically go into some advanced menus and see what's happening and tweak. You can always generate code from your tests and we will give you the Selenium code behind it, I guess, but using our SDK, our open SDK, which will be available soon, the code generation for the open SDK. So you can like, if you hit a wall and you want to go further and do something more advanced, you can always like take your recorded test, generate code and continue from there. Right. This is an option and for like intermediate users that don't want to code, but they probably know better what they are doing, you can always use the recorder even to create tests without recording. Right. So even if you don't have the application available for you, but you do know how to define the elements, you can go into, test, project, create all your element repository, create your steps manually ahead of time. Right. So when you have a live application to test, your tests already be available. Right. So these are all options for various types of users, I guess.

Absolutely. What I also like is a lot of times people say, well yeah, it's using AI, but I don't know what it's doing or if I can have confidence of what it's doing. But it sounds like you also have an algorithm baked in as well that over time as it's running, it's only going to interact with elements that has a good confidence ranking for, I guess so that it learns over time as well, I guess overruns. So when it dynamic ID is always changing, it usually knows after a few runs, you know which one is the most likely to identify that element. Is that how it works behind the scenes?

Yeah, more or less. We will evaluate an execution of tests and see what we were able to heal. Right. And then we will take a decision if we do update specific elements to make it run faster for the next time. And it will improve over time because we are constantly looking at it at the end of your execution to see what happened and if something broke along the way and we were able to salvage it. So we will make the adjustments if it's necessary to make it run faster for the next time. Yeah.

So besides AI, which not just besides, it's a huge, huge feature, I believe you Also made some other improvements, like you streamlined the user interface. Can you talk about what changes you made to the user interface that helps maybe make creating tests quicker or be able to manage tests easier?

Yeah, well, the previous recorder, right, the Legacy Recorder, we call it, some of the features and the options that were spread around, you had like top bar menu with some of the options. You have some additional buttons with tools popping up and all the steps and some context menus. So first of all we consolidate everything within like one recorder widget, if you wish to call it, and without anything like besides that. So everything in the same place, all the tools. We yesterday we released the Element Explorer version for the new recorder. That allows you to get some insight in your dom, like into the HTML elements and attributes and stuff like that. We have the locator and all of these already tackling the iframes, right? Because you can see the DOM of each of the iframes that you have and can locate elements. It will look for the elements inside all of the iframes. So these are major enhancements there. We got rid of some internal menus like in the previous recorder we had menus for managing parameters. So we consolidated it and put it in separate easy to use stuff for managing some unnecessary windows. If you're creating steps manually, you had lots of pop ups and things sliding in, sliding out. You had to move your mouse like a lot and many clicks. In some cases to create a step you had to do as much as like 10 clicks maybe. So when you're working with this for the whole day or several hours at a time, it eventually consumes lots of time just for this overhead. Right. So we eliminated many of these and like step creations is more waterfall now ui so you choose an action and then you automatically get the element, the selection, select an element and that's it. So three, four clicks and you're done. And we have many advanced features for the test and for the steps and before that they were all there, which like it was pretty cluttered, I guess. So we just, you know, collapsed it and minimized it and only if you need, you can go into that and see what's there and if you need to change anything, but in most cases you don't. So we just removed it. So it won't be in your face, I guess.

You know, I think we touched on this a little bit, but I just want to go back to it because I think it's an important feature in that we mentioned that Selenium is great. Selenium is awesome. You use Selenium behind the scenes to deal with web browser applications. But I know a lot of teams struggle with testing. These codeless applications were developed in like Salesforce or ServiceNow or SAP. So can you talk a little bit more in detail about like how it helps with Salesforce applications? Or why couldn't you just code Salesforce test in Selenium or what does test project add that makes that easier to do?

Well, Salesforce, for example, was a great case study for the new recorder, right? Because we came to understand that if we will be able to automate Salesforce, we probably will be able to automate anything. And the thing is that you can code anything, right? If you know coding, you can code anything, you can do whatever you want there. And if you do it right, it will probably work for a long time. But again, it will require skill set, a specific skill set and not everyone has it. And it will probably take you more time to achieve what you need, right? To automate Salesforce, for example, you will have to code and it will take some time. And if you are creating an infrastructure, even if it's like coded infrastructure, it will take even more, right? So we just applied everything that we know in order to tackle this Salesforce situation, which is like many iframes and dynamic elements and stuff like that. We are doing our best there to do what needs to be done in order for it to work.

I guess so I guess you should do it as a case study. Did you actually work with Salesforce? And then they say, yes, this tool actually works with Salesforce applications and we bless it or how we recommend it or anything like that?

Well, not really, but because of the community we got lots of questions regarding not only Salesforce, right? It was one of them. But people were struggling, right? And we are constantly doing some internal tests and we are like trying to identify more complex applications and not only use like demo websites to prove that our system works, right? So we like the community inputs and what we see. During our development process, we came to the conclusion that this is something that needs to be done because people are struggling, I guess.

Okay, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts? And what's the best way to find or contact you or learn more about test project?

Well, you can always, if you need to contact me directly. So email and LinkedIn, it's the best way, I guess. And test projects, we are actually like, we have our website and the blog and our forum. We also have the documentation all the ecosystem of this project. And of course if you're using the application and even if you just visit our commercial website and you have the chat option, right? So we are not using bots. We have a customer success team and they are always here to assist. So basically this is the best way, I think to contact us. And regarding your other question, well, if you are a developer that is going to develop, I guess if you're going to develop your own automation infrastructure, you should always think about maintenance and flexibility of your infrastructure, right? Because many automation solutions, and I saw it many, many times, especially in house solutions which like some companies decide not to use any external tools and just go ahead and develop something of their own, it eventually fails in most cases because it becomes harder and harder to maintain and you have new demands and you are not flexible enough with your infrastructure. So eventually it becomes the elephant in the room, right? So it's there and no one uses it and talks about it. So I guess like if you do decide to go with your own solution, always think ahead about how easy it will be to maintain this infrastructure and the tests that you will build on top of it and how flexible it will be for changes, right? New technologies, et cetera.

Thank you Alde for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com a3 22 and while you're there, make sure to click on the Try it for free today link under the exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.