Fitnesse & Visual Regression Tracker with Surat Das

Welcome to the Test Guild Automation Podcast where we all get together to learn.

More about automation and software testing with your host, Joe Calantonio.

Hey it's Joe. And welcome to another episode of the Test Guild Automation Podcast and today we'll be talking with Seurat all about visual regression tracker fitness and automation testing in general. He's been doing Test automation since 2002, so we're going to cover a lot of things here. He's worked with many companies, from startups to Fortune 500 companies across various domains. Currently he is an automation architect working in the San Francisco Bay Area. He has also lots of experience in developing in house automation frameworks based on open source tools and he advocates product experts like QA and BAS to write, execute and maintain their own automated cases, thereby reducing dependency on automation engineers to achieve automation goals for each team. I'm sure we'll dive into that as well. And he oversees automation engineers writing generic framework code, finding solutions for challenges, and building infrastructure to help support the team. So this is going to be a full stack pipeline interview. You don't want to miss it. He also contributes to a lot of open source tools as well that we'll touch upon and he promotes automation and tests, build, process or any task that can be automated, which I am 100% behind. I think you're really going to dig this episode. You don't want to miss it. Check it out. The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testskill.comSourceLabs and click on the exclusive Sponsors section to try it for free today. Check it out.

Hey Seurat, welcome to the Guild.

Thank you Joe. Thank you for having me.

Awesome to have you. You have just the right experience of what I'm looking for for an automation engineer. So I'm really excited to have this conversation with you. Before we get into though, anything I missed in your bio that you want the Guild to know more about?

No, I think you covered most of what I have been doing so far. I mean I have been advocating the open source tools and there has been build versus Buy conversation for quite some time now. I belong to the crowd where I actually build and when I say build it does not mean you right scratch, right from scratch for everything. You get a lot of open source tools, you just integrate them and then you code only when it's needed and that way you get the flexibility and that's what I have been doing. Yeah, I think you covered most of it.

Cool. So already I'm off tracks here. But when you say build versus buy, is there any time you think people should buy something though? Are there any situations where you see a company where you're like, I wouldn't trust them to build anything, so they probably better off buying something?

I would say unless there's a compelling reason we should never buy it. The reason is there are so many solutions out there and if you buy today, you will have only your goals or your requirement at this point of time. But going forward, the requirements always change. Your the evolving needs are changing all the time. So the tool that you bought may not be able to scale up with the requirement that we have. But if on the other hand, if you have a good automation engineers and they are very good in coding, you can always scale it up based on our need or we can always change based on our requirement.

So one of the tools I was surprised to see which I haven't seen in a while, is fitness. And you seem to be an expert in fitness. It's an older tool. Maybe I'm behind times I haven't kept up to speed with its development. Is it still being updated? Why do you use fitness? I guess my first question what is fitness? For people that don't even know.

Okay, sure. So I will give a little background on why we decided to go with fitness. So let's look at a typical workflow of automation, how we automate test cases. QA normally gets a requirement. He or she writes the test cases. He automates them one by one using one of the testing framework like TestNG, Junit or any of the other unit test framework. And then to write those test cases we also have to write the code using some other libraries like Selenium, APM or REST issued and so on. So let's say if we can segregate this part one writing the test cases using the methods that we have written already. And the other part is not the test classes but other classes where we have the logic implemented. If you write the test case you will probably write just like a. You know how end user will write the steps, sequence of steps that the test cases I would say. But to perform those steps one by one we had to write the code behind which goes in a. In a separate class probably we'll write the separate class so that our test classes are as simple as possible. Just the sequence of steps and assertions. So if we have this Kind of segregation. One person who is expert in product features, they write only the test cases and the other person or other team who are really good in framework, they write the all other code behind. If you can think of these as a two separate aspects. One is experts writing the test cases alone. They don't care much about the internal logic which is abstracted away within the framework code and the framework team writes the framework code. So if you can think about these two segregation, one is the test code versus framework code. So now in order to write the test cases you still have to have person knowing the test framework, they have to know the coding, they have to know the version control and so on. You have to set up correctly with the right Java version or if it's other framework, you have to know the right version install on their machine, you have to have admin rights and so on. So this is where fitness comes in nicely with the Fitness. What we can do is instead of writing the methods in testng, we can write the methods in a clear plain text format in a website. If we can write that and we fit in the framework code that I talked about a little bit earlier to Fitness. So fitness knows which method to invoke, which classes to invoke and that will do for us. So, so that way we can completely get away from having to install anything on a machine, having to know coding and having to compile and build and version control. All this stuff we can completely do away with. So that in mind, if you have Fitness now, just to give a little background for the listeners on what fitness is, it's an acceptance test driven framework and it is also a wiki server. It's an open source tool which you can download as a JAR file from the fitness website. You can run it as Java JAR and the JAR file on any other port and then it is available for everybody to use it as a web page or as a wiki server. So because it's accessible to everybody, each tester or each QA member doesn't have to set up their own local machine. They don't have to worry about testng, they don't have to worry about compiling the code, version control and so on. So that's how we started with the fitness. We can get up and you know, we can hit the road quickly. We can easily write the test cases, we can execute them, we can see the report right and there and we know this test case is really automated. I have run it, I've seen the report and I'm good. And they can move On Cool.

So I think people might be more familiar with keyword driven approaches and cucumber, but this sounds like the server piece is the unique piece because a lot of times even with the cucumber and those other things, you need to install it, you need to have all the Java versions, you need to have all the Selenium bindings, someone needs to know how to use it. So it seems like the wiki is used almost like, you know, English, like syntax that's written and behind the scenes you have layers and that layer then calls a more technical step that maybe an automation engineer had implemented with the coding. So the front end user then no longer needs to really know code in an ideal world. Am I understanding that that's right?

That's absolutely right, yes.

And so you've been using since for a while now, so I assume it's working well for you. How big is your team and has this any challenges you've seen using this approach?

Yeah, so there are a few. Now before I go to the challenges, I'll just briefly talk about how it works in our organization. So what we do here is now I belong to the framework team where we actually take care of the framework, where we write the generic framework. Now again, when I say generic, it means let's take example of a REST API testing. If you go to any REST API, it has the standards, you have an endpoint, you have a request body, headers and so on, you get the response, you get something out of the response, you do the assertion based on the response, probably you will call the next API. So this never changes for any team, it's the same for all. So we write that part of code and any team based on their endpoint, based on their request body, any team will be able to write their own test cases, regardless of how it is. Because I don't need to know what the test case is. QA will need to know. They'll write their own test case based on their logic. In fact, we can write it even before our product is ready. Because we have the specification, we know how the request should look like, we know how the response should look like. If we have this specification body, we can write in the fitness test skill right there and we can test it. We can do a pure test driven development with fitness. Now I'm not saying we cannot do with test NG, we can do test NG or Gemini 2 for this kind of thing. So there are I know at least four teams who have been doing, but then again, they are maintaining their own test case. I Do not go and monitor their test cases regularly because they have the confidence now to write their own test cases. And they come to Framework Team only if there is a new requirement. But I will give a new requirement. For example, let's say I need to be able to connect to two databases, one on the cloud, one in the on premise databases. Now initially we did not have that capability, so we as a Framework team go there and implement it. Okay, you enable connecting to two databases, for example. And another example is like when we do not have the access to direct database connectivity because of security issue. So how can we solve this problem? We need to like, for example, there's some coded PIN which we have to get it from databases because we cannot read from SMS or something as part of automation. So that's where Framework Team comes in and solves these type of problems. So yeah, I would say for API there are at least four teams who have been using. Now, in terms of test cases, I can give a rough count, but I will not be able to give you the exact count because they maintain their own test cases. And similarly, this framework we have integrated with the mobile and we also have integrated with the web. And like I said, API and database already are integrated. So that's with the structure that we have here. And we as part of the same test case, we should be able to do mobile, web, API, database, all as part of the same test cases. We can mix and match in the same test case. That's the capability that we brought in. Now again, this is nothing specific to fitness. It's against the framework.

So fitness, because it's acting like an interface, the user doesn't care what technology is behind the scenes. So I assume you can switch out. Say for some reason you were using Selenium, like, oh, let's go to Playwright, some functions I guess. Could you just easily swap that out with newer technology? And none of your test cases change because it's just the bottom layer.

Right.

So cool.

You gave the exact example that we did. We actually tried with Playwright and test cases did not change that. Did not know that we are doing Playwright. The reports even report, for example, we change the reporting from x10 to report portal without them having to know anything. Okay, what's going on? I don't know. So but these are things that we can work behind the scenes. So you can think of it like we removed testng, replaced it with fitness. When I say testng, just the test class, not the entire thing, just the test class. So that's coming back to your other Question on what are the disadvantages? Like I said, in Fitness, in the page you can write plain text. So when you write plain tags, it's very easy to make mistakes. Like if you would say spelling mistake, it's gone. It'll not find the method name, it'll just give up, throwing exception on the fitness page, saying, okay, I cannot find this method. So that's one thing that we really have to be very careful. So to solve that, I can think of two approaches. One is you maintain one Excel where you have the dropdown of the method names. So you can only pick and choose from those list of implemented methods. You can use those. And the one that we implemented here is we as a framework team built one angular website. So you just type in whatever you want, like verify. You type verify. It'll give you the list of the methods implemented for verification. See pick and choose from there and copy and paste it there instead of blindly writing, you know, a plain text. So that's one the other things that I can think of is it doesn't support parallel run like testng or other framework where you actually can run multiple test cases in parallel. So that is not supported by Fitness.

All right, how do you get around that? That's a huge one, I would think, for teams.

All right, so what we can do is we can have Fitness server running on multiple machines, so multiple servers and we can split the test cases manually. Okay, this fitness server will run only this suite, maybe subsuite the other one, the subsuite. And that's how we do it. And moreover, we rotate like so there are two things. One is parallel. One is the distribution, the parallel write all the test cases in all the browsers. If we normally don't do that because we know our product, we know what has changed. So it doesn't make sense to run all tests in all browsers. It is more for just feel good factor. Like it doesn't have logic all the time saying, okay, why are you doing that? So what we do is in Sprint, wise, one Sprint let's do in Chrome, the other Sprint let's do in Firefox, the next Sprint, we do some other browser. That way we have coverage over a period of time. We'll have coverage in all the browsers. So later point of time, if UI has not changed, probably we don't have to test in all the browsers. So that's another thing that we do. The next problem, okay, so we talked about, it's easy to make mistake because it's plain text. And second, we Talked about parallel run. And the other problem that we have is when if you have only one fitness server, remember it is accessible by everybody who can access the server. It means let's say you are running the test, some other person is running the test at the same time. Both of them will work, but it is limited by let's say you are executing on an emulator for example, or on a real device. Let's say you have only one device and both of triggered a mobile test case which will try to hit the same device. Then one person will not know the other person that he is executing. That's another problem that we may run into this issue. So for this the solution is run through Jenkins only so that it can queues up. It queues up. The other solution is let's have your fitness server for your team alone. So running fitness server is very easy because you don't have to do any setup. It's just a Java jar and jar and the top.

Can you dockerize it?

Yeah, you can dockerize it too. And the last problem that I can think of is it doesn't happen very often. Like let's say if you refactor a method name. Remember all method names are written in a plain text in the text case, right? Yep. So if we rename the method name, our test cases will break because it will not know which method to call because it's simply plaintime. So again we can use some, you know, shell command to modify all the test cases with a single command. We can do that.

All right. So how often does happen, Is that a common occurrence?

No, no, it doesn't happen. A few examples, it happens mostly with the UI layer. For example, you have login. So initially let's say we just wrote login with username and password. That's a method name, for example. Now let's say our test cases grow and we increase in coverage. So there can be login in the mobile, there can be login in the website. So fitness will get confused which login to call. Is it for mobile? Is it for web? So let's say we decide to change the name login in web with this username password logging in mobile with this username password that this is one example when we change the method names.

Cool. So you know, I'm just thinking of this approach pros and cons. One pro is I worked with the team Enterprise team that uses BDD and we said we're going to empower everyone to do automation and everyone's going to be able to Code. And the problem with that is each Sprint team, even though there was an existing function called login, they write their own login because they didn't want to touch anyone else's code. Seems like the way you wrote this you have separation where they can't write any code. They actually need to go to the framework team to implement whatever they're trying to do. And you wait a minute, we already have a login. Use this login so you avoid code duplication. It sounds using your approach, is that correct?

Yes, that's correct. And like if it's a big team, we can have our own class for the team. Like we write a dedicated class for their team alone. So in fitness we say okay, for this set of suit this set of test cases, use only this class, not everywhere, don't look everywhere else. We can do that too. Like for example, when we do the framework code right, we just make a jar out of it and we give the jar location to fitness so that fitness know which classes to look for. So for a specific team we can have a specific jar file if it's needed so that fitness will not get confused which method to look for.

You know you talked about, you said it integrates already with CI and Jenkins. I'm also thinking of bdd. What was good is we could use tags. So say for some reason we didn't want to run certain tests because we had tags in it. Even though it's plain text. We could say in a, we could pass a parameter like D filter at runtime. Don't run this tag or just run these tags. Do you have that type of functionality as well with fit?

Sure. So currently what we do is we write the test cases. You can tag, you can add multiple tags. For a test case, you can have priority one, priority two, for example tags or you can have, okay, this is disconnects to database. And if for some reason you want to avoid running the database related test cases like access issues and so on, we can decide not to run database related test cases. So you can tag your test cases based on what you what it supports or what it when you don't want to run it. And then the only way to execute those test cases for a specific tag is by giving the query parameter in the URL. Ultimately it's a web server, so you have to hit the URL and with the query parameter you append the tags. So run this tag.

Okay, okay, okay. Because it's a server, you just in Jenkins do a command with the URL with the parameters you Want to append to it and it knows what to do. That's cool.

Yes.

So I know a lot of times people once again use BDD and tools like serenity is really good reporting. How's the reporting? Is there anything you need to do special with fitness to reporting to work for you?

So when you execute a test, we have the test written in a web page and you can click the test button, it tests and it will show you the default fitness report. Now most of the times like for example, if we go to Junit report, right, we have Junit report as an xml, we can publish it, but we don't normally publish it for wider audience default Junit xml we normally come up with report portal extent or ELU and so on. So these all can also be done with the fitness code because it is not fitness by itself doesn't run any code. It runs the code that we feed into it. So normally we integrate this type of reporting with our framework code. So when you you are done with the fitness test, it creates that report for us and we can publish through Jenkins. If we're running through Jenkins, we can publish it or in fitness you can give link to that Excel to that report HTML file so it can show in the fitness itself. Other cool thing about fitness is that we have talked about only test case writing and test execution, but these because it's a wiki, we can use it for documentation as well. So in the same test case we can write it. Okay. This test case does this and this as a plain text, as a formatted plain text or like for example, we can place some files like xls or some special files we can download or download it through fitness by giving links. You can do all those stuff.

Nice. Yeah. I used to work once again for a large healthcare company and if you ever audited, you have to show them your test cases. And this way they prove to me what you test and what you're testing. Because it's English language, you just point them on a wiki. It must be very easy to to do something like that.

Yeah. And the other problem is like testng for example, Right. If we are writing test case, we have to commit ourselves. Like every QA member should know git, they should know how to resolve conflicts and so on.

Right? Yeah.

So with fitness you don't have to worry about this.

Nice. So I guess the next question I would have is where are the test cases stored? I guess it's a server. So is it also stored in fitness on the fitness server?

Yeah. So what do we do with every test case that we write, it creates a dot wiki, it's kind of text file with extension of dot wiki. So all these text files, we upload it to our git periodically. Like you can set it to upload all test cases on daily basis, on neat basis. So they all go in our git repository. So when you set up a new server, we download all the test cases from git and feed into the fitness. And Fitness will be able to read all those test cases.

So it's also version control then too as well. Nice.

Yes it is version control. Yeah, Fitness has its own version control, like through zip files and like through git as well. Like every change you do, it'll commit to git. But then we don't do it because there are too many commits. So we decide to do periodic commits or need basis commit.

Cool. So as you mentioned, fitness is like a UI interface and behind the scenes as layers. And you can use any technology. One popular technology nowadays is visual regression testing or visual validation testing. In that pre show you talked about actually using a tool called Visual Regression Tracker. So so maybe we jump into that really quick, maybe give people a little idea of what is Visual Regression tracker.

Sure. So many of the times you may think we should. Okay, so our tests are good for functional testing, but we also have to do the visual testing to make sure that the alignment is right, the font color and everything is right on the web page on a mobile app. So in order to do that we have a commercial solution like applitools where you can do that. But there is an open source tool available called Visual Regression Tracker. If you go to GitHub and look for this, you will get it. This is, this was created and maintained by Babel Strunking and this is something that we are using to meet our visual regression goals. So what we do here is it's a simple docker compose file which you can download and just run as a docker container. So in the container you will have the ui, the services layer, the postgres databases, everything in the docker container and you will be given one ui. Just like I would not say pre tools but just to give you analogy, this is how it'll look like. You'll have login, you can get your API key and with those API key you'll be able to write the code needed to send the images to the Visual Regression tracker. Once it is sent to the VRT tool, you can either approve or reject the changes. If it's first time there is no baseline. Obviously you either you have to approve or reject, but from the next run onwards it will show you the changes that has happened since last baseline. You can also apply the ignore areas. If there's any dynamic part like something that you keeps changing animations or all those things, you can ignore those or you can specify just this area to be compared and don't ignore everything else, but compare only this area. You can also have your regression tracker compared branch wise, if you have multiple branches for your project. So you can do that. And if you're happy with one branch, you can merge the baselines onto another branch. You can do that too. And that's in brief what it does.

How reliable is it? Like this goes back to the build versus buy and you said usually there's open source tools. A lot of people don't know there's an. Actually I've heard about this a while ago, but I don't know how many people actually aware that there's a visual validation tool that's open source and free. But you also have like you said, big players that have vested lots of money in this technology as well. So a lot of knocks that people say is oh, with visual testing is you're going to get pixel to pixel differences. And so therefore that's why you want to buy something versus use an open source technology. How has your experience been with this tool though? From a reliability point of view?

Sure, it uses the JavaScript framework and in JavaScript you have, at least in this tool we have three different libraries to compare. One is, you know, pixel to pixel comparison. The other one is, it's called looks same. That's a library. It doesn't do pixel to pixel but it does for example if you have a carrot like blinking. So it'll ignore that part. So it has those kind of logic. And there is another, I forgot the name of another library, but these are the three different ways you can do it. So you can, you know, play with one of them and see which one you like it. And even for the pixel to pixel exact match, you can define the tolerance. Okay. For example, I'm happy with 1% tolerance. So you know, please go ahead and approve even if there's a difference of 1% difference. So you can define this tolerance level. And so far we have been, we, you know, once in a while it happens like it's a. Because it's a very pixel to pixel comparison. Especially if you're taking your images, for example, right? You have to take the images from your test cases at a Very specific time. If it's too early or too late then you may see some pixel differences. So I think it's a problem with same problem will be same for any tools you use. It depends on specific time of your screenshot. Taking for example Selenium, right. If you take a little bit early maybe it has not loaded completely so you'll see a few pixels off. So that's, that's one of the challenge that we are seeing other than from the now again I would say it's not a framework related thing. It's more of a our test case that we may have to either we have to do wait for something or we have to do sleep so that it settles down before we take the screenshot. And speaking of that, for a mobile it's very, it's very reliable if we use the native frameworks X UI test and Espresso. I've seen it's very reliable than apm, especially the screen comparison part.

Good to know. So what I like about you is you lead by example so you highly promote open source and the only way I think open source is successful is people actually contributing to it. And so I think you mentioned the pre show that not only did you start using this tool but you actually contributed. So you can just give a little story around maybe people that are listening that may be using open source tools why you decide to contribute some of your free time to an open source project.

Okay, so I'll give two examples. One of the not two, I will give three examples. One of the Fitness itself. So my boss came and said okay, I know you can see the test cases in Fitness, but I don't know how many test cases are there in this suite. It's not visible in the UI until I run it. Okay, so I said okay, yeah, I get what you mean. So by looking at the at the top level you want to know how many test cases are inside it. So I went ahead and implemented and merged the code with Fitness itself. So that's one example where you really get the benefit of open source tooling. If you have something you know how to implement, you will get it immediately. The other is a fitness plugin for example to Jenkins, right. Initially it did not support authenticated fitness. So if there's an authenticated fitness it will fail. So I went ahead and added those logic so that even if I have authenticated fitness I'll be able to give username and password in Jenkins it will take it. The other example is like Fitness was not generating unit JUnit style of XML file. So I went ahead and gave JUnit XML because I have the JUnit XML generated as part of fitness, I can feed into and integrate with other system like testrail Test link. I can feed the results with the code onto those test management system. And other example with visual regression tester is like it's still developing, it's still new, it's just one and a half year old. So they're adding a lot of features. So I thought, okay, there are a lot of features that I personally need it. So I went ahead and added those feature into this visual regression tracker. And like for example, log analysis, right? If somebody deletes a build, somebody approves a baseline, for example, I need to know who did that or when it was done those example. Another example is like did not have a help and nobody reads documentation. If there's a help in the web page, it's very easy. They will click. Probably they will click and see it. So I implemented that so that others know what is a build, what is the variation, how do I approve the baseline? What does it mean to integrate with a specific language framework like Java JavaScript? How do I do it? Here are the examples. I did all of those and I also did on the backend side, for example, the approve. All right, initially when it started, you had to go each image approve next image approved. So it's a click, click, click, click hundreds of times if you have so many pages, so many sorry images. So I added the approve all button. So in the first run you can approve all if you're happy with it. So these are some of the examples that we went ahead and done.

Oh, I guess. Where did you get the time? Were you able to do it during work? Because your company uses open source, they also realize, okay, we're going to have to give back. Are you doing it on the weekends? Like how does that work out?

Yes. Yeah, yeah, yeah. So most of the features that I added is something that I also needed it. It's not right. It's not with the intention of giving back. It's like, okay, it's a good tool, I like it. I need these. Probably other people also need it. And I also went through their issue list in the GitHub. I say, okay, some people is requesting for that. That's interesting. I went ahead and added. So it was useful for them. It was useful for me too. And yeah, of course you with open source you really have to spend time during your personal time to be able to. And it's a learning too. It's actually Challenging and fun too. You'll enjoy once you start doing it.

And so just a quick when did you hear about regression tracker? Just curious to know.

Sure. Last year, around December time. I don't know if I read one of your blog or I read one of your podcast. I saw mention of visual regression tracker. Okay, wait, in the past I was doing with Image Magic, that's another library. And yeah, and then you had to do a lot of coding for that. Okay, so okay, if you really want to do that, there must be some better way. And when the moment I heard from you, I went ahead and looked it up in GitHub, found it and started using it right away. And because it's docker based, you don't have to do a lot of setup. Just download the Docker compose and start using it. And thank you for that. I mean it all came from this and your test Build podcast. Awesome.

I appreciate that. I'm glad that the community is benefiting from it. So this is awesome. So thank you for contributing and help spreading the automation awesomeness around. So that's awesome. Okay, so before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts in general, or with fitness or with visual tracker and what's the best way to find or contact you?

Yes. So the advice that I would see, I would give to everybody is like whatever solutions we are using, let's try to improve upon it because automation field is changing very, very fast. So whatever I'm saying today, after one year, maybe you will have a different and better solution. Maybe. So we have to constantly upgrade ourselves, upskill ourselves so that we get the best out of it. And we cannot rely on automation engines to have all the test pieces automated. We have to have everybody involved in the automation to be successful in the automation journey. And in my experience I found fitness is a very big stepping stone for everybody so that they get into this automation journey together. And you know, I am available at my email address which is surat s u r a t.dasmail.com that's the email that I can be contacted at.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to test guild.com a373 and while you're there, make sure to click on the Try for free today link under the exclusive sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in iTunes. Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is help you succeed with creating end to end full stack automation awesomeness. As always, test everything them and keep the good. Cheers. Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full.

Show notes, amazing blog articles and online testing conferences.

Don't forget to subscribe to the Guild.

To continue your testing journey.