Recheck-web Golden Master-based test framework with Jeremias Rößler

Five, four, three, two, one. Drop that.

Welcome to the Test Guild Automation Podcast. We're all getting together to learn more about software testing and automation with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Jeremias Rustler all about open source AI solutions, including an awesome one that he showed me recently called Recheck Web. Jeremias is the founder and CEO of ReTest who focuses on making AI and test automation a reality. He also is an experienced speaker. I think he's spoken at over 50 conferences and so he really knows this stuff and he really knows AI. So if you want to know how to get started with AI in your automation and love open source solutions, this episode is for you. Listen up.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out. Hey Jeremias, welcome to the guild.

Hey Joe, thanks for having me. I'm very excited to be here.

Awesome. It's great to have you on the show. You actually showed me an awesome demo of this tool called Recheck Web and I thought it'd be a great episode to talk about today. But before we get into it, could you just tell us a little bit more about yourself?

Yeah, sure. So I studied computer science in Germany and then did my PhD in Saarland University. And when doing so, during the PhD I focused on AI based test generation. Actually, the goal was not the generation. In the ivory tower of academia, you can just assume that test generation is a solved problem. So I focused on AI or using the tests that were generated to afterwards.

Fix bugs automatically.

Wow. So how is that even on your radar? Like AI and testing back when you were doing that, what made that be something you wanted to focus in on?

So I've been working in industry for a couple of years and on every project I've been to, the biggest problem always was testing and test automation. And so I figured that this is something or I figured that it would be worthwhile going into that direction. And during my PhD.

I assumed that this was already working technology. And when I was finished with the PhD, I had to, you know, go back to reality where this wasn't a solved problem and so I went on solving it.

What would you consider what the problems were that you saw with automated tests that caused most issues that you wanted to help address.

So in my experience, one of the biggest problems is with software that you change something, one little aspect and then somewhere else entirely different part of the system breaks for apparently no reason or for a reason that's not clear. And then if you dig into it, you see how the things are interconnected and how one change affects another, and therefore how it all came to existence. But this is all a lot of manual test and manual effort to analyze the problem. And it also.

Creates a lot of insecurity about software changes. So people are afraid to change software unless you're, you know, Facebook, where the motto for, for quite some time was go ahead and break things, but with typical projects, you're afraid exactly of that changing something and then something else that you weren't wasn't aware of breaks. And therefore I want to tackle that problem and get people to, you know, have a way to ensure that whatever they change doesn't break existing functionality.

So what are some of the ways you see AI in testing or AI being able to help testing?

I think that AI has a great future overall and especially in testing, but probably in different ways that people assume. So when I talk to people, they tend to think that AI will replace testers and manual testing. And I have a strong opinion on that, and I strongly disagree. I think that AI will replace test automation or manual test automation and will create test automation for you. And we are actually working on that and trying to solve that. And even, like, if you think, like, I don't know, 10 years ahead, then AI will probably have the capabilities to generate, even go as far as to generate code. And if you think about that, the road to getting there, the easiest code you can possibly generate is test code because there are no loops, no if, then else, no branches.

And it's okay if it's spaghetti code. So the easiest code to generate is test code. And if you have enough tests, you can actually go and generate bug fixes and automatically apply them because you have the tests that prove that those changes do not break existing functionality. Actually, if you have covered your whole system with enough tests, you could, in theory, throw away the implementation and just regenerate the functionality, the code that.

Incorporates the system. And so I think this is the road to the future.

Awesome. You actually gave me, I think, a little glimpse of the future the other day in a demo. But I just wanted to touch on one other point that you just mentioned there that I thought was interesting. So in Your vision, you see a lot of people have the opposite vision where they see everyone needs to get into coding and know how to code. As a tester, you're seeing almost the opposite then, right, that we're going to have tools and algorithms that'll be able to generate the code for us and we should be honing in our testing skills in order to make sure that what's being generated is doing what it should be doing. Is that, is that what you, how you see it?

Exactly. That's how I think the future will be. Like, like for. If you have some like even today, software isn't built totally from scratch. You have building blocks like you have a UI framework, you have a database connection, you have various functionalities encoded in tools and frameworks and whatnot. And the main task developers do today is plump these existing kind of components together. And I think that part can be replaced in, not today, but in a couple of years can be replaced by AI generating the code that is kind of the glue code that connects existing components and existing functionality. And then you still need someone to check whether what the AI generated is.

What is actually wanted and what people would like to have and that it works properly from the aspect of the customer. And so I think that eventually testers, manual testing will still be there. Because you know, you have, this is kind of a fuzzy thing, right? You have to talk to customers, you have to talk to stakeholders to, to the product owner, to haver. Whoever to, to get to understand what it's really, what it really is that people want and that you need to build. And then you have to test and verify that this, what has been built is actually meets that requirements. Whereas the other side, generating the code or.

Creating the functionality, I think that's much easier to automate using AI.

One of the things you mentioned that people struggle with is the constant changing of an application that causes a lot of the automation currently to be what they consider flaky. So I believe you came up with a solution and it's open source called Recheck Web at a high level, what is Recheck Web?

So it's a golden master based test framework. So the idea is that when you first run the test, it creates essentially a copy of the website and then in later runs it compares that existing copy to the current version of the website and then it highlights differences.

There's a couple of things that are beneficial. First is.

With typical assertion based test automation, you only check what you write down that you want to check. And most of the time the users care about a lot of different things. So if, for instance, the CSS is completely broken, but the website still kind of displays the correct text, then most of the tests will. Tests will still pass because this is only what they check, that the correct text is displayed and not in which way it's displayed. And this is something that can be addressed with Golden Master testing. And another thing is that a lot of tests tend to be brittle in that if you change, I don't know, the XPath, the ID, the text that the test relies on in order to identify the elements it wants to interact with, then the test breaks. However, Gold Master break based approach has a benefit that if the test breaks, then we still have the Golden Master, where that text ID whatever is yet unchanged. So what we do is we have a look, peek into the Golden Master, find the element that we originally wanted to interact with, and then go to the current version of the website and find the element that has the highest match in terms of all the remaining attributes like XPath, name, text and whatnot, and then return this element. So it means that the Golden Master test based approach is much more resilient to change because it can use all the redundancies that are inside the website. And we even go one step further, when we create that copy of the website, we introduce into that copy, we introduce another attribute that does not exist on the live website. And because it doesn't exist on a live website, it can never be affected by change. So this means that you now have a virtual constant identifying attribute which we call the retest ID that you can use in order to reference the element that the test has to interact with. And then again, using all the various attributes of the past of the persisted, the Golden Master website.

And comparing them to the current version of the website, we can find again the highest match and return that and interact with that specific element, making your tests very robust, very resilient to change.

A lot of problems people have is, well, I don't have a unique id, they have to go beg the developers to add something. This almost sounds like you're injecting that ID for them automatically. Is that correct?

Yeah, right. That is also one of the issues that this approach addresses, that you can have an ID that is independent of the live website, so you can give it any semantic meaning that you want and just reference the element within the copy. And then you don't have to have an ID at all on the live website. And, you know, some people have to resolve to using, you know, complex XPath expressions or complex CSS expressions to reference a specific element that they want to interact with, and using that approach, you ingest.

Instead, you can just inject that ID into the element in the copy of the website and then use that ID and reference that element.

It also kind of reminds me of an old school windrunner Quick test professional. We had something called an object repository, and so it used more than one attribute to identify an element. So there was like, you could use five attributes, and if only found four of it, it still would work. So is this something similar where it uses multiple attributes then to also drill down and find what the test needs in order to identify that element?

Yeah, in a way you could say it's an. It's an object repository. Instead, you don't have to create it manually, it's just conveniently created for you on the fly. But yeah, it's very much comparable to that.

And I don't mean to keep bringing up all the tools to compare it again. So I'm just trying to get people to envision it. Another one that's kind of like. I'm not saying it is, it's totally different, but it's kind of like is applitools has this feature called Check Eyes, and it's just like you do one check eyes check, and it's able to automatically do a visual validation for the whole page and then be able to tell differences. I think you have something similar here. So to give people an idea, like how would this work? I think you have something called a cap test and just a matter of using that one line. And that's what actually does the Golden Master comparison.

Yeah, actually it's very comparable to applitools in that regard. So at some point, applitools was our.

Role model, so to say, when we designed the API. So you can have explicit checks where you say, now you want to create a golden master and compare against that, or you can have implicit checks where you just kind of wrap the existing driver, whether it's a Firefox Edge or Chrome driver, inside the recheck driver. And then whenever an action occurs, like you open a URL, you send text, or you click on a button or something, whenever an action occurs, then we do an implicit check, but under the hood, essentially what happens is really that we have a call to recheck, create a Gold master and compare to that Gold master and then show you and highlight all the differences. The difference in contrast to applitools is that we do this on a semantic level, meaning that we do not pixel comparison so we don't create a screenshot and compare the screenshot to an older screenshot, but instead what we do is we compare the elements and their respective CSS attributes after rendering. And this gives you a lot of power because this means that you can. One of the problems with Gold Master based testing is the noise that you have. So sometimes there are things that do change and they are okay to change. Like you have, I don't know, IDs depending on your framework that are randomly generated or you have some part of the website where you have advertisement or stuff like that. So these things do change and it's okay for them to change and Golden Master based testing has to cope with that. So what we created is kind of a possibility to define what you want to ignore, but because it's not pixel based, you can define that on a semantic level. So you can say ignore every, I don't know iframe, ignore every data attribute in the HTML, ignore for instance the color attributes because I don't care about that. Stuff like that. It's a very powerful kit in that regard was our role model. It's a very powerful way to define what you want to ignore and.

Still care about the rest. And in that regard you can for instance, ignore the font of a text, but not the text itself, which is not possible, for instance, with applitools or other pixel based comparison tools, because there you only can ignore parts of the image and whatever is contained in that part of the image, you ignore it completely.

Cool. So what are some other benefits you think someone would get out of using this approach?

I think another very important aspect is that you can't write an assertion for things you don't expect. So if something unexpected like there's a very funny example from Google where they use some sort of unicorn or dancing pony to mark elements that have been changed. So if you're a tester at Google, then you and you find a dancing pony on the screen, then this element where the pony is dancing should be very deeply tested because that has been changed recently. However, at some point they didn't remove these markings and those things went out into production. So the actual customers had dancing ponies or unicorns on their screens apparently, because they didn't write a test for that. So they didn't check, you know, they didn't write an assertion. Is there a dancing pony on the screen? Obviously you should do that. And what I mean is that if you write assertions, you have to write an assertion for every possible change that you don't want. So every possible thing that could potentially happen you have to write an assertion for or else you won't be notified about it. And with a golden master test based testing, it's kind of the other way around. It's like you write an allow list for changes that can occur and you don't care about them. However, if there's an unexpected change, something that you didn't think of in advance, then you will still be notified about it. And then you can decide whether you care about that because it's, you know, a dancing pony that you don't want to escape to the customers, or whether you don't care about it because it's, you know, something that that regularly changes that you just don't care about because it's an irrelevant frequent change. But that's one of the major Also one of the major benefits is that it's the side of error.

That gives you more security because it guards against unexpected changes.

I guess another question folks would have is it sounds great, but they need to test against cross browser different browsers. Does it work against different browsers? I know a lot of new technologies only work against Chrome.

No, it actually does work against different browsers. We are right now still working on i11. That one we haven't covered yet. But you can do both mobile testing and cross device and cross browser testing with Recheck Web and it will show you again, not only it will show you that there are differences, it will not only highlight, okay, there's now something different, but it will also tell you exactly in what way it differs. For instance, you have a different font installed, you have a Times New Roman vs. A Times Roman font, for instance. Or you have a CSS difference in a CSS attribute. However, as I said, it does not do pixel comparison. So what it guards against is if browsers or JavaScript frameworks or anything comes out differently in terms of CSS. If the same CSS comes out differently. This is something that this approach does not guard against. But we feel, and we see currently that these differences kind of get less and less over time. Like 10 years ago, using the same CSS rules, you had very different outcomes. And now there are only the edge cases where there are differences remaining. And we feel that these edge cases will be addressed also in the not too far future as browsers kind of merge in their way they understand css.

If something does go wrong or it does find a difference, how does it report? Is there a report that's generated? Or how could someone dive into what actually caused a Validation error.

So the first thing that happens, these tests are usually Junit tests. Right now it's implemented for Java and one of the next things we're going to tackle is JavaScript implementation. We're right now looking into Cypress or something like that, but right now it's implemented for Java with Junit. And then if a difference occurs, what this does is it makes the test fail. And in the error message of the failure, it shows exactly where the differences or what differences there were. And also it generates a report. And that report then contains screenshots both of the original version that the Gold Master was created with and with the current version of the website. And then those two are used to kind of highlight the differences for the user. So you can use that report and either open it with the CLI and apply the differences to the Golden Master using the cli, or you can use that report, download it and open it with a gui. And then within the GUI you see the two screenshots of the old and the current version and the marked differences and the textual description of the difference. For instance, something's moved or something's the color changed or the font size changed or whatever happened.

Cool. So in the show notes, I'll definitely have a diagram, but you show me the diagram of the pieces that make kind of up your solution and I guess you had rechecked cli, recheck Web and a Chrome extension. Are those the three main pieces that make up the open source piece or side of the solution?

Right. So this is something that we aren't aware of any solution that is comparable to that. And so we figured that this is something that we want to share with the community. So we created a full stack, full fledged open source version that is completely usable, that doesn't lack in any functionality.

That comprises of essentially two parts, which is the API part that can be used to hook into Selenium and generate the difference report and then the difference report that can be opened with a CLI and can be used to both show what has been different and to maintain both the Golden Masters and the Ignore file. And then the Chrome extension, which is also open source, which can be used if you do manual testing, like if you want to manually compare, if you don't want to, you know, go all the way, creating a Selenium based kind of project where you execute your tests, you can use the Chrome extension and can manually compare websites, which makes it easy to spot differences if you're, I don't know, web developer, or if you're just a Casual, you know, user trying to find differences between the old version and the new version of a website.

I believe you have a paid piece also that people could upgrade to, but it's not, like I said, mandatory. Is this Open source is completely end to end open source and available free. But if someone wanted extra functionality, what would they get if they wanted to pay to get even better insight into what's going on.

So one of the convenience features that is within the paid functionality is that the reports are uploaded. So if you're for instance, running your tests on the ICD environment, say Jenkins, and the tests are being run on different nodes of Jenkins, then it can be quite cumbersome to access the reports. So in that case you can just upload them to our service and conveniently download them again to access them and have some team functionality and something like that. And you can use the GUI as a pure viewer. So that's free. But if you want to use the GUI to maintain the test cases, to apply changes, or to maintain the ignore file, then this is part of the paid functionality.

Does this work with CI cd? And if it does, I guess, does it work with the Recheck CLI piece where it sends commands to the API using your Jenkins or something in the pipeline?

Yeah, so this just integrates with your typical.

CICD stack. So this is essentially an extension to Selenium. So wherever you can execute Selenium, which is typical Junit tests or Cucumber or any other test framework that you use, you can just exchange driver that you have with a retech driver and then use retech instead and have your reports generated there so it integrates nicely. So if you, for instance, some people ask us about reporting functionality, well, we haven't built specific reporting functionality because this is already baked into junit. Right. So if you run your Junit tests, you see how many tests executed, how many failed, and how they did over time. And so we didn't create specific reporting functionality for that part because this is already baked into existing solutions.

So how hard would it be for someone who has an existing framework to get started using Recheck? Because it sounds like it would pretty much be seamless, but how hard would it be, you think, for someone to implement it in an existing framework?

So we hope that this is almost no additional effort. So this is our goal at least. So all you have to do is add the dependency to the project and swap your or wrap your existing driver. As I said, Firefox, Chrome, Edge. So all of the major drivers should work except ie, you Just wrap your driver in the retake driver and essentially then you're good to go. So this should be it already.

Cool. And like you mentioned, because it's almost like an add on for Selenium, it's able to run in Sauce Labs and cloud based providers, it sounds like.

Right. So you can use all the Sauce Labs and other cloud based providers to execute your tests and run this on like real devices in the cloud, real, I don't know, even mobile devices, and any, you know, combination of operating system and browser that you can come up with.

So the only red flag that comes to my mind is I don't even know if this is an issue. You said something creates a master copy of your whole website. So one thing is performance. Because you're scanning a whole application, does that impact the how fast your test run in any way?

So what is actually a bottleneck is screenshot creation. So what we do in the next release, the creation of the screenshot is not mandatory anymore. So you can opt out of that. However, the rest is pretty fast. So what we learned so far is that firing up the browser and doing the communication with the browser takes so much time that loading the local Gold master and analyzing and doing the assignment of old to new components is not adding very much time on top of what is already happening. So I mean, you know, end to end tests tend to be slow because of all the overhead you have. And so far what we've seen is that reject does not add much to that, except as I said, for the screenshot creation which will in the next release be not mandatory anymore.

Cool. And I guess one of the last objections someone might have is they're usually afraid to use libraries because they don't know if it's going to be supported or how invested a company is into it because it is open source. So.

How long has the solution been around and do you foresee your company maintaining it throughout, you know, the next few years coming up?

Yeah, so we've been, we've been venture capital funded by the end of 2018 and one of the things that we actually aim for, which is then the paid or commercial solution is AI based test generation. So what we, what we really want to do is use AI to generate tests so that you don't have to manually create test automation anymore. So you can just give us an URL and we will generate the Selenium tests for you or the Cypress tests or what have you. And, and this is our kind of our main focus and our main product. And this technology that we've made open source is the technology that we rely on. So we need this to work in that way because AI doesn't know what the correct outcome should be. So if you, if you have a, I don't know, tax calculation software and then the AI can't know whether if the result is $1,000, whether this is the correct outcome or not unless it understands tax software, tax rules, which is quite hard. And then again it can generate the software, it doesn't need to test it. So instead we created that framework to replace traditional assertions. And since this is kind of the foundation, the underlying technology that our premium AI based test generation software bases on, this is pretty much supported by us. So this is what we think is groundbreaking new approach and, and yeah, we will support us as long as we exist, which hopefully will be.

A bit longer.

Okay Jeremias, before we go, is there one piece of actionable advice you can give someone to help improve their AI automation testing efforts and let us know the best way to find contact you or learn more about retest.

So the best way to find us is, I mean I try to give a lot of talks, so I try to do as much conferences and meetups and things that I possibly can in order to spread the word and get people to recognize us and know us. And so the best thing one can do is probably either listen to one of my talks that's been recorded on YouTube or contact us directly via the website which has a chat feature built in, and I'd be happy to connect to anyone that's interested in that topic. So we're very open about everything we do and we are very excited to show this to as many people as possible. So if you're interested in that, please go ahead and contact us and we'll happily show you everything we can and answer any questions you might have.

Thank you Jeremias for your automation awesomeness. For links to everything of value we covered in this episode, head on over to test guild.com aalecautomation277 and while you're there, make sure to click on the Try it for Free Today link under the Exclusive Sponsor section to learn all about Source Labs, all awesome products and services. Also, just a heads up, I'm about to Release Automation Guild 2020. It's the fourth annual online conference dedicated 100% to helping you succeed with Automation testing. As far as I know, it is the biggest online automation conference in the world. We had over 700 folks join us last year and I'm hoping to have even more. Learn how to create better, more reliable tests in 2020. You can learn more about that@guildconferences.com so that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Gilbert Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.