Accelerate Test Coverage Using TestRigor with Paul Grossman & Artem Golubev

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we've talked with Paul Grossman and Autumn all about how to accelerate your test coverage and eliminate maintenance using Testrigger. Paul, as you know, is the Doc OTS wizard. He's an automation estat who has been developing hybrid test automation framework solutions for nearly two decades, probably as much or if not longer than I have using a host of different frameworks. He's always messing around so uft, leanft, Selenium Test Project, Cypress, Testrigger and more. And he's always working on new automation awesomeness, sleight of hand to upskill his automation techniques. Artem is a software veteran with more than 20 years in software development, building various B2B enterprise and E commerce software solutions for companies like Microsoft and Salesforce. Most recently, he's dedicated his time to automation software testing and is currently working as the CEO at TestRigger. If you want to find a cool new tool that can help you create tests quick and fast, you don't want to miss this episode. Check it out. The Test Guild Automation Podcast is sponsored by the fantastic folks at.

Their cloud based test platform. Helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Paul and Autumn, welcome to the Guild.

Hey Joe, it's a pleasure to be here.

Yeah, pleasure to be here.

Awesome. Great to have you, Paul. I guess before we get into it, Adam, let's start with you. Is there anything I missed in your bio that you want the Guild to know more about?

No, you're good. You're good.

Okay. How about you, Paul? I know you're always doing something new.

I am.

But you cover most everything. The newest things I'm playing around with is test rigor that we'll probably talk more about today. I spoke at a couple of conferences just a few a few weeks ago, so I'm still doing that. So I'm happy.

And Paul, how about a plug for Vivint? I think you're speaking at an upcoming conference with Vivint Worldwide.

Yes, again, Vivid Worldwide, talking a little bit about the magic object model, the idea on that, and also on how Testrigger itself is very close to that model and does a lot of Cool things out there. So I would say, you know, check out that conference, it's going to be a ton of fun out there. Oh, it's virtual by the way, it's VR. So if you've got some Oculus headsets, you could actually sit in a seat in an auditorium virtually and watch the presentations. It's really cool.

Very nice, very cool. So let's dive into the meat of the topic now. How to create quicker automation tests, faster using and less maintenance. So I guess first, before we get into it, why do you think there's a reason to create automation faster nowadays? Aram, you do work for a software company. Have you heard from other companies that there's a need for this type of ability?

Yes. Overall the world is moving faster. Right? It's just faster and faster and faster and faster. And if you're not fast enough, you're just losing it to your competition. And then think about it, it's just the efficiency of building things is improving over time. Whatever used to take us hours and days and weeks and stuff is now taking takes minutes in our case, it takes minutes to build test case on our system versus it takes almost a week in certain cases for Selenium.

So Paul, working hands on with the tools and with automation, do you find over the years that not only is there more companies wanting to have automation, but is there more of a pressure on the software development or the software developer or the software tester to create the automation faster than we had in the past?

The pressure is always there. Obviously. I can tell you something. Oh, about 10 years ago I was working with a manager and other, we were, of course, you know, we're always competing in the field and our competitors would come into a client and say, hey, we're going to do a proof of concept, let's get it. We'll do, take, give us about a month or two and then at the end of that you pay us for all that and if you like what you see, then keep paying us and then we'll continue working on it. My manager would come in there and say, give us three days and we'll show you what we can do. We don't have to, you don't have to pay for it. If you like it, continue on. And that gave us a real big advantage. But nowadays you've got with Testrigger, I can tell you that you could be sitting down at that meeting and say in 30 minutes I'm going to show you something and if you like it, let's continue on. It's Just the whole proof of concept just from the beginning is just getting tighter and tighter to be competitive in the field.

Nice. So Paul, as you know, also a big struggle for people to just have automation is maintenance. So as we move faster and faster, I assume that's becoming more of a challenge. Thoughts around test maintenance and maybe how that's slowing teams down as well?

Yeah, when you get applications being changed from release to release, you've got a bunch of things that are changing. Your element identifiers, your descriptions, your DOM is changing and then you've got to go and try and adjust your, your object repository, your Selenium, if you've got the page object model, you're describing your elements, you know, inside your classes and then if they're not generic enough to handle changes in there that come release to release, you're spending a whole lot of your time doing maintenance. You might also be challenged by, you know, either the existence or removal of iframes that used to be a big hot thing years ago. So that's a lot of where your maintenance comes in. And if you can kind of avoid that as much as possible, that makes your whole end game so much better because most of your time is just writing more and more test scripts or analyzing your results.

Yeah, I'll give you another anecdote about that is let's say you're building a website, small e commerce website, Amazon 2.0 and you have a button where saying buy for product and you did a great job and you built 10,000 functional end to end test cases for your website. Then guess what, you migrated now to a new way of purchasing products on your website. Now you have a shopping cart and now it's not just click Buy, it's click add to cart, click cart and click place order instead. Plus editing some fields. And in our case in our platform it takes seconds to adapt because our underlying language is just one to one with end user actions. So you literally can do things like find and replace.

If you didn't have it extracted in some subroutines. So you'll be able to quickly adapt versus think about it, 10,000 functional end to end tests like it's main months of work to adapt with Selenium. Right.

All right, so I guess another issue I see is obviously the quicker you need to create test is you need people to be able to create these tests. And when I was in a large company of course they used Selenium, which is not a problem, that's fine. But they also chose Java which also required people to Be programmers and a lot of testers may be good testers, but actually if you're being honest, mediocre programmers. And using Java of all the languages seems to be even a harder language to really pick up. So have you seen issues with this approach? And I know there's a lot of down people down on codeless technology. So Autumn, let's start with you. Any thoughts on Codeless or issues that you saw that maybe codeless could solve or things like that?

Yeah, so, well, whatever people call codeless is where you just click through the stuff. It's very, very limiting. And then remember that example we have support where you need to adapt from buy button to completely different flow with add to cart and click cart and those kind of stuff where you have to what are you going to recreate from scratch? 5,000 end to end tests. It's just impossible. So in our case, the closer we giving basically manual QA people an ability to build proper tests. It's just our expression language is that in plain English expressing like end user actions. So this way you can record it using browser plugin, but then it's a code quote unquote in this plain English code. So you can modify it, you can improve it, you can actually maintain it. And that's the key, I believe.

I also say that our tools today, we also probably would have to mention cucumber and gherkin.

I was about to just bring that up.

Yeah, cucumber and gherkin where trying to make the creation of the test cases as easy as possible. I mean you've only got four or five different initial statements given when, then and or but very rarely used over there. But then behind the scenes you have to build all that code. So you do have a balance of people who not necessarily have to be quite as technical in order to build those gherkin test cases, but you also have to have your developer sides on the other side building all the support and whatever new kind of interesting functionality needs to be supported inside of that particular framework.

So Adam, you actually developed a test tool called Testrigger and obviously it's multiple iterations now and it seems really impressive. I'm just curious to know, you know, with all the other tools out there, you know, I was just on a panel yesterday with the folks from Selenium, Cypress, Playwright now is the new darling of the automation scene. Why another test tool? Where do you see your tool that was fitting a need that maybe these other vendors are the tools. Not saying one's better than the other, but what hole do you think your tool fulfills that some of these others may not be a good use case for.

Yeah, great question. So basically what I have seen is that a lot of tools are great for unit testing that people got covered. Like unit test tools right now are so good. It's just hard to come up to anything better than that like API testing all covered or all kinds of tools in there. You don't need to do feel like anything else. Sometimes you can do like even integration testing is fine. However, what we have noticed is that when it comes to end to end testing, especially involving ui, the challenge is that UI often changes so fast it places just insane amount of tax in maintenance on like on people who building that test up to the point where people for the first year maybe they're happy and they're building up tests and this is all great and stuff. Starting from the second year, 50% of the engineers time is spent on test maintenance. Like a tantal huge amount of time. And up to a point where we have seen use cases where maintenance just killed whole automation and companies just switched back to 100% manual just because we couldn't keep up with maintenance. So maintenance is a problem we actually trying to help people to solve. And this is big problem for the UI and this is where we're doing that.

So before we get into a little bit more. So can you use it for API testing as a just a end to end UI functional browser based automation tool.

You can call API as a part of end to end test but we are not pure API testing. Better tools for just pure API testing.

Awesome. And Paul, as I mentioned at the beginning, you've worked with every probably automation tool out there. So you have a bag of tricks with everything. So I saw a few demos that you did with this and I'm just curious to know how real those demos were because you were able to use TestRigger in about under 10 minutes and you automated a site and reran it without doing anything else. So was there any sleight of hand to those videos? Like tell me a little bit about your experience with the tool itself.

Okay, so there is some time compression that. The 10 minute video is about 20 minutes to write out and edit from start to finish. I will tell you that one of the reasons I was really amazed by testrigger was just a few weeks ago Artem reached out to me because I think he saw me talking about this magic objects model where you just describe what I'm looking for, go click on an OK button. And I said let's I've got this, you know, this Candymapper website is a challenge website for anyone to use their tool and just try and accomplish some of the test cases. The initial walkthrough on that is just to go and clear a pop up, enter in a name, submit an email, verify there's a message saying you got to put an email address in there and then click another button, verify that it got submitted. It's like 10 steps. Most tools you play around with, just to get through those 10 steps will take you quite a bit to do. Artem was actually kind of walking me through it, giving you a little bit of help on that. But within about 10 minutes we just basically we did a record on that. We tweaked up some of the text that was inside of the recording and it was basically done. And it goes through and clears everything, hits everything. I don't have to worry about element descriptions or other items. One of the other things I really liked that happens in there, and this is a feature that is specific for the Candymapper website, which is the pop up. When you hit the website the first time, it throws a cookie in there that says this is your first time in here, shows the pop up, right. You go ahead at the second time if you don't clear your cookies. And I know all us automation engineers, we always do that, if you don't, you'll never see that pop up again. That is a, can be a big challenge in automation engineers for where we're saying, well, we might see a pop up, we might not, but don't fail the test if it does pop up there and if it's not there, don't, don't fail if you didn't see it. In this test case demo, we basically say if it exists. So it says if you see it, clear it and if you didn't, move on. But it doesn't fail anything. And that's one of the things that really kind of blew me away. Again, the time it took to create that test was about 10 minutes and that's when I kind of went, wow, this is, this is amazing, this is cool.

And yeah, I think the most interesting part, which I enjoyed when we found that close button, remember, which like was this specific challenge on your side. Want to talk about that? Sure.

So in the demo we've got, in fact it is that pop up that comes up there. We did do a record and playback. And to be, just to be clear, that's just a tool used to learn how to use a tool. You don't want to use it every day, but it is something if you need to learn how to do stuff. It created a line that just said click and I gave it a really weird description. We went down and looked at that line and Artem said you can just describe it and say click the button on the upper right hand corner of Pop up Challenge. And that's all you needed to do. And it figured it out and it's like, okay, it clicks on the button and closes it. So we translated it into plain English and it still worked. So that's another thing that really I found fascinating in that tool.

So that's a new feature in selenium 4 of relative locators. Is that what you're talking about, Autumn, where you say to the left of this or to the right of that, but it knows it all automatically Case.

It'S completely new level. So whatever is introduced in Selenium 4 is just a kind of children's play compared to what our system can do there. In our case we were saying, hey, click on the button on top right. I'm not sure if that is even supported on top right of the header, where there's a header in that pop up. And top right was a close button. So I say close on the first button on the top right. But then you, in our system you can also say, hey, click on the second button on the top right. Or hey, click on a button which is below this and on the right of that and so on and so forth. So like there is just a completely new level compared to Selenium first and second. Again in Selenium you are working with low level stuff. Then you're working with test trigger. You're working with visible elements from end user perspective on the screen.

So I guess because this is an audio and people can't see it, we'll have links to Paul's demos in the show notes. But how do you write a test then? If you're not using code and you're saying it's. There has to be some sort of context that you're running the test in. Like there has to be some sort of syntax, even if it is English. Like so does someone just use record and playback with a Chrome plugin or can they just start writing, click and enter and it automatically will translate that into what it needs to do.

I can answer that one. The answer, the answer. That one is both.

Oh, okay.

Yeah, there's, like I said, there's a recording plugin that you can go and hit everything and it'll Generate stuff for you, but you can also create your own test case and then just start writing out exactly what you want it to do. And then it goes and executes. Grab screen captures, shows you what it hit and goes on from there. Artem, anything else you want to add on to that?

Yeah, so we actually use a combination of natural language processing and the parser and where is the documentation and so on and so forth. So the goal, because the reason why we use NLP in the first place is that the goal of our system is just by describing just two rules, each end user action is one new line. Right. And all parameters are in double quotes to remove ambiguity. That's it. That should be enough for you to be able to express what you want to do. And this is like the exercise which we're doing over and over and over again with new customers, seeing how we do it. And in most of the cases we just are able to come in and do the test without reading even documentation whatsoever. But that's not the point. The point there is that we believe the only true BDD ATDD tool on the market, the only period, because this is the only tool where your program manager can come in and express executable specification for engineers without involving QA engineering to write any code.

So I guess I haven't seen the tool. So what happens when people hear this is they think they're going to have 100 tests and each hundred tests will have their own hundred duplication everywhere. So when you're recording and you have like a login, when someone records their own tests, they're going to have a login as well. Does it realize there's another login or does it not matter? Because the tool will take care of on its own anyway if anything changes. Like how do you make things reusable or does it matter in your particular flow for your tool?

Oh, like no, you can do it like in any other code. You can extract functions, you can extract it post factum if you already recorded a bunch of stuff, you can kind of extract method, extract functions like that. But most importantly, unlike those, no code, record and play stuff. In our case, you can start from a place where you want to start from somewhere in the middle in case if you already have a function to get there so you can record just only the last part, then post factum, add that function to get you to the right state and just add validation and that's it, just like that. You don't have to start from scratch every single time.

Yeah.

And what it's Called is rules, but I kind of refer to them as business rules. So you can take those little bits of the plain English four or five lines exactly as you gave great example, doing the login and then put that off and say, I want the login. We could create, let's say login, admin, login, user, login manager, login, super user. And then each one of those would do the process for specific test. And again, just like modular design, something changes in the way that login occurs, password changes, user account changes, you change it in just one of those components. Every test that references it is now updated.

Nice. So for some reason I always think of this scenario as passing context between browsers, which is difficult in a lot of tools I've used. So with Testrigger, are you able to say like startup second browser and that type of. How complex can your test be? Using Testrigger, it's trivial actually.

You can open a new tabs, you can start up a new browser, or you can run multiple browsers at the same time and then you start up a new browser. They start up in completely brand new, separate instance, completely unrelated to original one. So you can actually test things like chat by logging in with two different users at the same time in the same session, and then test that when user 1 sends a message to user 2, user 2 immediately sees the message. So this is also quite a unique feature on our system.

That's the exact scenario I was thinking of. I was thinking I used to work for a company that radiology software. So you had to be as a radiologist, you had the machine open and as a. I forgot what the other scenario was as a patient or something else, you were viewing it and you had to pass context. It was really difficult. So sounds like it handles it very cool. I guess the next question is Chrome browser extension, does it work with all browsers or do you write your test in Chrome and then it's able to run against any other browser?

So you.

For recording, we only have Chrome extension right now, but then you can run it at the same time on any browser you'd like, including everyone's favorite IE11. And if you're using external infrastructure provider like Browser Stack, you can even enjoy running it on IE6.

And if you're a Mac fan, you can run it on a Macintosh as well.

And just a shout out for my.

Sponsor, yeah, you can run on Safari and all this kind of stuff.

You can run against Sauce Labs, I guess as well. Then if you can run against other.

Cloud providers, you can run On Sauce Labs, Anything that they support? Yeah, go Sauce Labs.

Also a big trend is running in cicd I assume. Is there like a command line option to be able to interact with Testrigger to make it part of your pipeline?

Yeah, absolutely. It's as simple as copy paste of a bash script which will trigger, test trigger and we'll wait until it is done.

So what are some unique features here? I think you told me does something with email validation. Two way factor authentication, I guess. Paul, anything you saw here that you didn't see that you don't see often in other tools that it can handle?

Yeah, the email validation is pretty cool. You have it like from day one, you set it, have it set an email over to the test rigor email account. It validates on that side that it sees what expected text is and then it responds back to you saying that either it did or did not find the expected data inside the email.

Yes. So you can we run our own email so you can generate a new unique email every single time you run the test. Which is absolutely super important feature to test things like sign up flow. Right. Because you cannot test signup flow with the same email every single time. So you must use different emails and in our case you can do it also from unique feature perspective. Our goal is to be end to end testing. Remember I was saying end to end. So the goal is to help people automate anything that a human can do. For example, we can test two factor authentication literally with text messages. We can check your text message and extract the code from text message and use it as a part of login. We can validate that your downloaded file was successful and check the contents of your downloaded file, be it like report, xls, CSV or what have you. We can of course help you to validate that your charts and diagrams look certain way they should and so on and so forth. There are tons of different features. We can even help people to test phone calls and things like audio. For example, if you build your own podcast website where you are yourself running audio, we can help you to test that your audio actually running and the sound is actually playing. And that is exactly the right sound.

Okay, how do you do that? Are you using a third party to do that or is it your own technology?

It's our own tech. So we basically got into this. We have customers in telecom space and basically we build a bunch of stuff for them. And some well known telecom companies are actually using us to test verse stuff.

Right now it almost sounds like an RPA tool. So is this kind of the same model, Paul, or is it, I guess you could use it for not necessarily functional automation, but you can use it for automating things that aren't necessarily a functional test.

Yeah, I think arm's just starting to get into the remote process automation field. The one of the tests that we were looking at putting together was basically searching for a book title off of Amazon, pulling off the title and the price, and then jumping over and logging into Salesforce and then putting in like a request to go and buy this book at this price. With that, I mean you could set up automation of a lot of different existing manual processes and to let them get taken care of. I kind of see like an HR department getting a request for a job. Somebody wants to get a job and then they have to send out an email that says, okay, fill out this information, let an automated process, you know, handle that and look at it and pass it back and maybe even search a little bit through the messaging of an email and tweak on what sort of response the HR department will have to give them and let that be fully automated. I see a whole lot of upside in the RPA field on that.

So Paul, I also saw something in your demo. It looked like it was running on a separate server. You're running your test that was running in the cloud. So just keeping in my mind then, are you able to run tests in parallel? And does it automatically handle running your test in parallel and running in different, almost different Docker instances or however it runs to make your tests run faster, like in parallel?

It does run in parallel, but I'm going to have Artem explain that a little bit more in detail.

Yes, we are like serverless testing for you. Serverless test execution. Right. Usually you would have to deal with all this infrastructure set up, you manage your own servers. Even if you run and all those kind of stuff, it's a pain. And what if your Chrome auto updated and you didn't and when you screwed right, like this is your Chrome driver and all this kind of stuff. In our case it's just completely serverless, running in parallel, automatically out of the box. All you need to do is just write the test, let it go and it will run. That's it.

All right, so I guess the next question is people are in love with open source tools. And so I don't even know the model that's. Is this open source? Is it reasonably priced? Is it more like an enterprise solution? Like who's your target audience? I guess that would make a good fit for your solution?

Well, we are partially open source. We have like this plugin which use browsers open source, there's some bits and pieces which are open source and we are free. We only charge for infrastructure. We don't charge for framework itself or anything like that. You can have unlimited tests, unlimited users, what you would expect from open source. We're not necessarily open source. However, it does not matter because what we help companies to do is we help companies to automate away a lot of manual testing. If there are a lot of manual QA testers which are executing test cases manually, then we help them to basically automate it themselves. And then they need some simplified tools with the UI where we can do it easily. It's not about open source or not, so you can just play with it and all this kind of stuff. It's about effective tool which actually helps companies.

That's cool. Yeah, so. Oh, that reminds me. It sure reminds me then AI is your tool an AI solution? Are you going to slap AI on or is it actually, does it do some sort of machine learning behind the scenes?

We use a huge number of different models. So when you're talking about AI, you need to understand that just AI is a general term. You can use it for anything. You wrote your own hello world program. It's already AI by definition because the definition is so broad. Just you can check it on Wikipedia. But in our case we do use machine learning indeed, because probably most common example would be we want to allow you to click on the cart icon by how it looks. So if it looks like a shopping cart, you should be able to click on it. And the only way to make it work is to assign labels. The only way to make it work is just to use machine learning based models to detect and classify those type of icons and so on and so forth. There are tons of different models which we use specifically in specific cases. Like there are about five or six of them where you literally cannot do anything else other than machine learning. But we're not like.

General AI, so don't mix it with general AI. We're not general AI, we just use machine learning models in the places where we should be used.

Okay, Paul and Arnim, before we go, is there one piece of actionable advice you can give to someone to help them with their automation testing efforts? And what's the best way to find contact you or learn more about test rigor? Let's start with you, Paul, and then we'll end with Arnim.

I'd say always be open to lots of different tools that are available out there. I'm always excited about anything that will make my life easier and faster and remove all the maintenance and let me get things taken care of quicker and faster.

And Paul, the best way to find you and contact you?

Sure.

Well, you can get me on email@thedarkartswizardmail.com I'm on LinkedIn. I'm also on Twitter @dark artswizard and if you want to see some of these cool videos, check out YouTube and just put in Paul Grossman, the Dark Arts wizard and that will take you directly to my cool videos. And by the way, I'm working with Utopia Solutions and my boss Lee Barnes is a really great guy so I got to give him a a heads up, a shout out to my boss.

Awesome. Awesome. Cool. Cool. Nice. Adam.

Okay, so basically I believe that you should use the right tool for the job, right? So whatever is the right tool for the job. Am I not necessarily with the strigo? Who cares? So whatever will help you and make your life happier and you work more efficient. This is what you should use. Like this is. I believe in that.

So Artem, best way to find a contact too.

Yes, feel free to connect me on LinkedIn. You can find me Artem Golubov Coma Test Trigger. You'll be probably hopefully only one Artem Golubov had Test Trigger in there. And yes, please also check it out online. You can go to testtrigger.com and request yourself a free trial. We have 30 day unlimited free trial. Thank you.

Thanks again for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com a331 and while you're there, make sure to click on the Try for free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full Stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full.

Show notes, amazing blog articles and online testing conferences.

Don't forget to subscribe to the Guild to continue your testing journey.