Improve Your TestCraft with Eran Kinsbruner

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. Today I'm really excited to be talking with a group great friend of mine and a great friend of the show. He's been on the show many times when it was test hooks, but now I think it's the first time when he's officially on the Test Guild Automation rebranded podcast. So really excited to talk to Yaran Ginsbruner. All about testcraft, which was recently acquired by Perfecto. If you don't know, Aran is the mobile technical evangelist at perfecto. Over 20 years of experience in the dev and test industry. He's also the creator of the quarterly Digital Test Coverage Index and he is an influential blogger and speaker at global conferences like Star East, Eurostar and my own online conference, Dedicated Automation Automation Guild. He is also the author of the book titled the Digital Quality Handbook, which I had the honor of writing a chapter for. So as you can hear, he has a lot of experience. He does a lot for the testing community. Really exciting to have him back on the show after a while. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial, visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Ron, welcome back to the Guild.

Hey Joe, happy to be here again, like anytime. So super excited to talk to you again.

Awesome. So before we get into Ron, is there anything I missed in your bio that you want the Guild to know more about you?

No, Actually I did author two books, the Digital Quality Handbook and Continuous Testing for DevOps Professionals. And as you mentioned recently, Perfecto, which is owned by Perforce, acquired Test Graph. So I'm excited to be here and talk about that.

Great. Yeah, so that's the main thing. I don't know if you know, but every year I write the beginning of the year or the end of the year, I write my predictions for the upcoming year. And in 2019, one of my main predictions was that more and more companies would be acquired by larger companies. So a lot of these smaller testing companies be acquired into these big. I was calling it the vendor merger madness. So it's another example of perforce acquiring other businesses. So in the testing space, just curious to know was it about testcraft that you think made it a good fit for Perfecto? Let's start it off with that.

Sure. So if you want to understand the rationale behind this acquisition we are looking today and actually over the past, I would say few years as continuous testing and DevOps became reality software test automation was still wasn't lagging behind. It was definitely making it advancements. But at the end of the day test automation is not a great success in today's market. It is attributed to skill set to flakiness of some of the test automation frameworks. So when Perfecto within Perforce, which we are serving, you know, large enterprises that are doing automation using traditional frameworks such as Selenium and Appium and other frameworks, we saw that these organizations that own business testers manual testers are still struggling with test automation that actually works. When I say works, it works continuously upon each code change, upon each software iteration. And what we have learned is that business testers some of the dev skill set is not really writing good automation code unable to really fit into the test automation ecosystem. Therefore we looked at tools such as testcraft was a great fit because it's codeless selenium based which fits these less coding skills engineer and allows them to fit or join this ecosystem. In addition, business testers are a great resource for any software teams like feature teams because they understand the entire business flows of the application. The only thing that lacking is the good coding skills in Java or JavaScript. So by matching the business testers understanding of business flow with the benefits of a codeless testing solution like Test Graph we are enabling them to join this overall test automation or continuous testing journey.

Now so you touched on this a little bit but I probably I'm thinking of three, maybe four areas where how the acquisition of testcraft reflects the challenges that you've been seeing in the market. And like I said, you're a leader in this industry. You do a lot of analysis, you do a lot of reports on where we're going or what you've been seeing in the field. So maybe we cover some main themes that I've been seeing in 2020 and beyond and see how maybe Testcraft fits into that. So I guess one thing is AI. So I did a little research on testcraft before the interview and they do market themselves as an AI based solution as well. So how AI fit into the acquisition of Testcraft?

Oh definitely. So the codeless piece of testcraft is a given. The AI engine that is baked into the solution allows the scripts that are being recorded to kind of self heal themselves by monitoring the entire characteristics parameters of each object or element that is on the screen. Think about a website that is being automated via testcraft immediately the entire DOM tree, the object map of each page is being captured by testcraft and kind of being rated by the reliability, the likelihood of a test script to really find and succeed in clicking on this object or element. And this is done in runtime as well. So once you record the first script that is using AI to capture all the objects on the screen, it creates this map and weights and give actually ratings from 1 to whatever depending on the elements on the screen. So from the next execution and the third and the fourth, this machine learning algorithm actually establishes a very solid object strategy for the script for the automation. So if any elements is moving or changing, or if a new screen comes into play which makes the previous element be less reliable, the tool itself will fix the script and move to the next element in line. So from the end user perspective from the test engineer, it doesn't need to do anything. Okay, so that's the main AI piece in the script. Maintaining the objects which are in my mind for Selenium are the main flaky area of test creation.

Absolutely. So you know Imran, you've been in the automation game for a while and I know when I first heard about codeless automation or AI based automation, I would kind of cringe. So how real is this technology? I assume you're not going to invest in something that doesn't actually live up to the hype. So have you tried the AIB solution in Testcraft and how reliable would you vouch for it to people that are listening?

Sure. The main differences between today's Codeless Selenium from Testcraft and other cordless testing tools, you know, if you remember and me and you are in this space for so long, there was a UFT solution, right? UFT could have recorded scripts for you. But the main difference between these tools and today's TestCraft AI solution is that TestCraft is a dynamic creation tool which adopts, changes and fixes scripts for you in opposed to other codeless tools that are simply recording a static script that if something changed, you need to go back and either record the script or do manual changes. In today's codeless Selenium tool that Testcraft is bringing, you have near zero maintenance for these scripts due to the AI component. In addition, since the Codeless is fully compliant with Selenium Technology in opposed to other Codeless which are kind of proprietary. You are always up to date with all the browsers and web drivers. So if you have a Selenium technology that can click on an element, can scroll, can do whatever activities or actions on the web application, this is something that is being augmented by Test Graph. So you're not really losing any functionality that Selenium is giving you, but you actually benefit from high maintainable script which is fully compliant with Selenium without writing any line of code.

So I think another thing that people hesitate dealing with a black box solution. But it sounds like Selenium is being used and it's being abrasive, so it's using the standard. So if someone wanted to actually go into the guts and change some code, can they or is it a complete codeless type solution where modifying it on your own and looking at the code is impossible?

That's a very good question. So there are two ways for and obviously and if you will go into the tool, you will see kind of a checkbox if you are really up to it. So if you are an expert and you understand the code and Selenium deeply, you can do various things. You can obviously leverage the Selenium code within the Test Graph solution, but you can also manually audit and change elements that were actually defined by the machine learning algorithms. So let's say a Selenium script that was recorded by testcraft uses one element locator and it believes that this is the highest rating object. You can override this element manually and change within the code the element to be used. In addition, if you want to inject code into the script, JavaScript code, run API testing through test Graph, we can do it as well. So obviously there are much more advanced and dev focused capabilities within testcraft and the ones that aren't available will be added as part of the partnership with Perforce and Perfecto. But we do provide, or testcraft does provide advanced abilities for developers and these guys that are coders to do much more sophisticated things with the scripts after they've been recorded.

So I guess the next thing that might come to mind is when you think of Perfecto, you think of a large mobile device farm in the cloud that makes it easier to not have to maintain your own lab. So how does testcraft plug into this flow then? Is this like going to be an add on or like what? How. Why would Perfecto want to have a codeless type solution integrated with it?

That's a great question, Joe. So a Perfecto is not just A mobile lab. It's a web and mobile. We fully support Selenium Protractor, Puppeteer and other headless browser testing alongside the mobile. So we are full digital cloud solution for code based developers. What Tescof is bringing to Perfecto is again the missing component of business testers ability to join this ecosystem of test automation by allowing them to automate Selenium script without writing one line of code. So it fits perfectly into Perfecto and actually Testcut is already baked and embedded into the Perfecto UI in a beta mode right now. So you can actually go ahead and try Perfecto with Testcraft in a better method or mod as of today. And if you are thinking about if you are a feature team that includes SDEx develop and business testers, Perfecto will be actually the best fit for you because developers can use Selenium Espresso Xeuitest. Same goes for the SDETs for the test automation engineers. But the business testers will embrace the Testcraft codeless solution and complement either manual testing and exploratory testing without again writing any line of code.

So I believe Testcraft is a SaaS product. So do people need to download anything or would they just go into a web based IDE and be able to do other coding there or would they have to download anything? Get started.

Nothing to download. As you mentioned Joe, It's a full SaaS product web based. You just need to go get your credentials fully secured. Of course it's an enterprise ready solution and you are getting started with it. Once you got your user password credentials you can go ahead, you just plug the website URL that you want to automate and you get into the recording mechanism, you have your canvas in front of you and you can start creating your scenarios. And oh by the way you can actually even integrate these scenarios from the web environment into your CI CD pipeline. So think about the S that's running Selenium code based testing they can now add to the CI build acceptance testing. Also codeless testing.

Nice. So I want to dive into that a little bit. We've already covered one challenge we've seen in the field and that's people writing tests and maybe not writing the best tests. And the reason why is because the nature of the web and modern web development makes writing tests really difficult and makes your test flaky if you don't develop them correctly. Sounds like TestCraft has some AI that helps you with element locators with which and wait times that it handles behind the scenes that you don't have to worry about as a script developer. So that's one thing we addressed. The second thing I've been seeing is a shift right in a lot of companies of monitoring and also running tests in production. And I noticed I've seen this in testcraft, I hadn't seen it in many other applications. Maybe I missed it is they have a what looks like a unique monitoring kind of solution. Can you talk a little bit about maybe shift right functionality that testcraft might have that you think helps address some of the challenges you've seen in the market as well?

So I would say that testcraft has again the main thing is their functional test automation that is based on Selenium. I mentioned earlier that you can actually shift left as well. So you can actually do some API testing for your website, again using testcraft, but you can also do some what we call test in production using the codeless Selenium and really understand what's going on in the production environment by running a subset of your functional testing against production environment. So this is again APM and monitoring. I think people are still starting with the shift left, the things that are pre production. So testcraft in that regard is looking both left and right and allowing you to do actually what they call end to end continuous monitoring that again runs on different data sets. You can put databases, Excel files and other configurations and run your test against different production solutions, production environments and gain end user experience data that you can obviously use back in your development.

And you did mention shift left, so you mentioned it does handle API testing, which is a big trend that I've been seeing as well. Another one is modeling. Does it help you with anything to do with the modeling your tests before actually creating them?

That's a great question. So it's Testcraft. I wouldn't classify it as a full MBTA like a model based test automation. But since Testcraft is giving the user a canvas to really create different scenarios by dragging and dropping the different business logics and flows within the application, you can think about that as a modeling canvas or modeling environment that you can really kind of design. It's a designing kind of environment for your test scenarios. So in the canvas you can drag and drop different elements from different screens from different user journey flows within the website and this will automatically generate for you the test code or the test scenario in the that regard. So it's a modeling environment, but it's not like a full end to end modeling testing solution like other dedicated tools that you would see in the market. It works in a different way that's all. But again, the canvas is your modeling playground, if you like, for creating your test executions.

Very nice and ran. I think you're one of the first people that was talking about continuous testing as like a phrase or kind of a keyword that more folks were using. So how does testcraft help with continuous testing? Does it have any built in capabilities that you see matches well with the continuous testing type phase a lot of companies find themselves in now?

Oh definitely. I wouldn't say beaten up, but I was definitely challenged by the term continuous testing. People ask me, okay, so what's the difference between continuous testing and test automation? So obviously, and I would say the $1 million question that you just asked. So obviously test automation enables you to have continuous testing, but test automation that brings value is the key for success. And what we have learned is that Selenium cannot really give you 80, 90% automation coverage that is always reliable and consistent. Hence it harms the goal of maximizing test automation coverage from one software iteration to the next. This is why when you combine tools like Test Graph, which are stable, codeless, self healing on top of your good and existing Selenium, Selenium is not going away. You know I'm talking about not one or the other, I'm talking one and the other. So this is why, when going back to your previous question, why Perfecto and testsoft are such a good match because when you combine the testing that is code based with Selenium and Perfecto with testcraft with Imperfecto which is codeless, you're really able to get a maximized test automation coverage that is as stable as you can get. You couldn't get it just doing code based Selenium. We have tried it, we have seen it. It's not really working. So to have a good continuous testing solution that runs one after the next after each pull request, you are able to trigger both Selenium code and TestCraft and get one single dashboard of quality visibility. This is what continuous testing is all about. Question about fitting into the pipeline I think I mentioned it earlier. Testcraft fully integrates with Jenkins and TeamCity. It also comes with a built in scheduler from within the testcraft ui. So you can definitely trigger Testcraft execution either five times a day at a given hour or you can just put a groovy script or connect it to your Jenkins CI pipeline that will complement, as mentioned, the other Selenium scripts that are running from your other pipeline or other environment. So so again it fits perfectly. It doesn't really compete with Selenium it adds on top of Selenium and gives you a really good continuous testing environment for web and by the way in the near future also for mobile.

Very cool. Yeah, I think a lot of times people think anytime you talk about Selenium that you're dogging it if you're saying it doesn't have something. But that's not what we're saying. A lot of these good products out there now are using as almost a wrapper on top of Selenium. So they use utilizing all the great work all the contributors have put in and then not saying there's anything wrong with Selenium, they're just adding extra functionality on top. There is what it sounds like.

Definitely, definitely. And we see in selenium 4 you're also going to see cool features like relative locators and stuff like that that will even make Selenium more valuable and reliable. But still it does need some complementary solutions in the form of codeless or other add ons that are coming on top of Selenium to make, you know, to get the most out of your cross browser test automation.

Right. And absolutely. Selenium is just an API so you could do with it as you will. But you do need to create all the plumbing and all the scaffolding. So these tools also help take care of common things. Boilerplate code that you have to write as well. It sounds like.

Definitely. And the reporting, you know.

Yeah.

Think about the reporting and the analytics, test impact analysis. These kinds of things are not coming baked into Selenium and sometimes not also with their plugins. So with Perfecto and Tester that is baked into this solution, you're getting again the best of both worlds plus a very good reporting dashboards and analytics that will allow you to drill down into each and every execution and really make sense out of all this test data.

So over the course of our conversation you have mentioned CI CD a few times. Pull request Jenkins so when I speak to I've been speaking to a lot of people now that I've been doing this full time with different solutions that are coming out that are AI based and one feature functionality I found really helpful and I'm curious to know if you see is this actually a feature or do you see it being baked in at some point is when someone checks in code, you know, using just straight Selenium sometimes it's hard to know what test to run against that check in. So therefore people say oh just run all your tests and gets all the things and you being a code, a cloud Provider may think that's a great solution as well, but I just find it hard to really, I find it adds a lot of noise. So I'd rather run a set of tests that I know target a particular check in. So is there any functionality that when someone checks in code that testcraft is able to know, okay, they checked in this component I know based on past runs that test test A, B and Z will cover this particular check in. So therefore that's all I need to run. Does that make sense?

Oh yeah, yeah, it makes sense. And I think that the keyword that you mentioned earlier is noise. Okay, so we are dealing today, test engineers and developers are dealing today with way too much noise because they are carrying a lot of test executions from one build to the next, from one pull request validation to the next. And this is a too much time consuming, it's not focused on the real code changes. And in some cases since the code is not really stable, I'm talking about the test code, you're adding too much noise into your pipeline. So tools like testcraft plus Perfecto, when they are combined together, Perfecto has built a reporting layer which is AI based that allows you, using what we call failure analysis, failure root cause analysis, to really look at history, look at labels of the failures, tags and many more and really classify your full test report based on failures, whether there are real bugs, where there are elements not found, pop up network issues or environment issues, backend services issues and stuff like that. So testcraft with Perfecto does give you what we are calling test impact analysis. That can really give you a good data driven scoping for your next iteration. Just by looking at previous failures, previous defects, previous flakiness in the test, you can really scope in and out the exact test cases that you really need to get a stable build, acceptance results or just test regression results in your next iteration. So to your answer, yes, we are providing that.

So that's a great point. Triaging is where a lot of people spend time because you may have even a 5% failure rate if you have a large test suite is going to take you a lot of time to debug. And there's an open source tool called Reportal IO that does this as well. So it sounds like there's like when you test run, you can then go to a portal and see it groups together different error messages so you can help triage quicker as well.

Yes, yes, yes, we're looking at that as Heat map what report portal is doing. And by the way also Sea Lights are doing things that are very, very advanced test impact analysis. So we are definitely doing a lot of the same things by clustering, triaging in an automated fashion based on failure reasons, functional areas that are failing. Even platforms, we have what we call a heat map. So when you look at the heat map, you can filter out all the greens and just see the reds, the ones that fail. And you can see them either by a test name, by a test plat, by an operating system, by a tag that you edit, can be regression or CI can be a server name. You can slice and dice the data as you like because our algorithms will allow you to get at the end of the day an exported view of the specific requirements or query that you ask for. Based on that, obviously you will make the right decision.

So I think another thing that people struggle with is running tests in parallel. So even though you have a cloud based solution where people can just spin up to infinity, I still see people struggling if they don't create that test correctly to begin with, to plan for running in parallel. So does testcraft have anything that helps folks create the test the right way to start with so it is able to scale up fairly easily?

Yes. So testcraft comes with a built in set of virtual machines for web testing in parallel and from CI if you want to expand to a larger scale, can be unlimited. Obviously when you plug Test Graph, connect it and it's baked again into the Perfecto cloud, you have unlimited number of VMs and mobile devices. Think about web. It's not just web, it's a responsive web. So it's both web and mobile. So when you plug test, when you run it standalone, you can run in parallel to a given amount of scalability. When you connect it also to Perfecto, you expand your scalability to both web and mobile and different geographies as well. With 11 data centers that Perfecto has, the sky is the limit. You can run at an infinite scale if you like.

So Ron, you know, I know it's kind of early, you just acquired Testcraft, but you know, you're like me, you're looking at the landscape, the digital landscape, and you see issues that are probably still outstanding. So if there's anything you see on the roadmap or coming down the road you think would be really helpful to be built into Testcraft or the Perfecto solution that you think will help a lot of testers going forward.

Sure, Joe and I keep getting asked this question and it's a valid one. So as mentioned earlier, Testcraft today is focused on codeless selenium for Web and Responsive web testing with Testcraft being part of Perforce and baked into the Perfecto solution in the future this is going to cover both web and mobile native application for codeless testing. So it will actually be be a full codeless digital solution for again mobile applications and cross browser testing based on Selenium and Appium at its core allowing you to really maximize your test automation coverage. Again with a lot of AI and smart algorithms baked into the engines and a smart reporting and analytics that allows you to really get good decision making throughout your pipeline.

Awesome. Okay Ron, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts and what's the best way to find or contact you?

Sure. So I'm the easy part to contact me is either search me on Google, I'm on Twitter ek121268. I have my own personal blog called Continuous Testing Blog. I'm on Facebook and LinkedIn and many other channels. So look me up. Actually just recently created a Slack channel Slack workspace called Test Automation for Web and Mobile. Look it up as well. And one last advice to your audience and to the practitioners out there. Test automation as always people are saying is an art. But today's reality, especially in the digital landscape, this is becoming very very complex and you need to be smart about it. The old rules of executing everything every day, every time upon each code change doesn't really work anymore. You really need to have a good decision making solution that allows you to slice and dice your test data, make better decisions. So you are really focusing on the code that was implement. You are running the right test, the one that are giving you value. So you really get the most out of your test automation. So definitely try being more smart about decision making in test scoping test coverage as an ongoing practice.

Thank you Aran for your automation awesomeness. For links to everything value we covered in this episode, head on over to testguild.com a312 and while you're there make sure to click on the Try it for free Today link under the exclusion Sponsor section to learn all about Sauce Lab's awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.