Next Generation Functional Visual Test Automation with Tobias Müller

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Tobias Muller, all about next generation functional visual test automation, continuous validation and much more if you don't know. Tobias is the CTO at Progle and a seasoned tester. His goal is to help shape modern development methods to drive software quality. With his distinctive expertise, high quality standards and drive, he actually led a team who developed a really cool testing platform called Test Results IO that I've been using for for the past few weeks and running a review on that I'll be releasing this week that you definitely want to check out. So really excited to have him on the show. He has a lot of experience, you don't want to miss it, check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment. For Every framework, browser, OS, mobile device and API. Get a free trial. Visit test guild.comSourceLabs and click on the exclusive sponsor section to try it for free today. Check it out.

Hey Tobias, welcome to the Guild.

Hey, thanks for having me.

Awesome to have you. I guess before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

No, no, it was perfectly fine, just better than I could ever express it.

Very nice. So I guess before we get into the main topic, you know, we had a few conversations before this and you mentioned something about dashboarding that I thought was kind of cool that a green dashboard is bad. I think it's kind of counterintuitive. So I just wanted to get your explanation why you said that or what you meant by that.

Sure. The thing is, I learned that from Alex from one of our speakers. And the reason is actually that the green dashboard means like everything is okay for 90% of the management at least. And the red dashboard says something is not okay. And I actually had it with a customer this week where they had like 120 test cases and 21 test cases were failed. So there was a little red pie in the top right corner and they actually got nervous and testing got the attention. So they actually said like, yeah, there has to be something broken, stuff like that. But Nobody's actually looking into the details. And that is why we say like a dashboard is not enough. If you look at a dashboard and it's green or red, it doesn't mean anything because it just means, yeah, none of the test cases failed or none of the test cases passed, that's fine. Because as soon as there's something reddish on the dashboard, people screaming and they start to get interested in testing stuff and they always usually say like, hey, we need to get that fixed. Even so actually don't need to fix it because it's just a minor finding somewhere. And that is why you seem to get a lot of insights from a dashboard. But that's actually not true.

Yeah, absolutely, I totally agree with you. So I think this is one of the problems, not problems, one of the side effects of DevOps trying to release quicker and faster. So, you know, I know you have a lot of experience, especially what I love about UL is you have a lot of experience in heavy regulated markets, enterprises. So you're not just testing simple websites. So I'm just curious what other type of issues your customers you've seen have been facing within DevOps.

Yeah, some of the customers actually are not even adopting DevOps yet. So you need to think about that. In regulated environments you need to validate everything. So if you change from one version of your build server to another one, you need to revalidate your whole build environment. It's not something somebody is changing. Like, yeah, I have an update on my laptop and I can just use it. And you push that somewhere in the cloud and can run it. Everything needs to be validated. It goes like up to the framework version and up to the operating system version that you use to build. The system actually need to prove that the system that you build is equal to the one that you built previously if it has the same version number and stuff like that. So some of our customers are not yet in the DevOps world because of that. For the ones that are actually the problem is that most of the tools out there don't reflect on regulations. Most of the time it's like an add on. So you add something like electronical signatures in the process. But the problem is traceability can also be handled by some of those tools. But the biggest pain is really tool validation. So whenever you change something, what I just mentioned is like you need to validate it in the chain again. And that is where you need to come up with clever ways to say like, the surface that you need to validate has to be as small as Possible. And that is where it's another architectural term that I do not like. But it's microservices where you say like I put all of the required stuff into a small service and that is fixed. We call that a frozen solution so it doesn't change over time. And all of the additional features are just added around it so that you don't need to validate it again because the core functionality stays the same. So like you can say the execution engine stays the same. So they struggle with that. Actually they struggle Having like DevOps means lightning fast and always being on the latest tool chain. So that is what developers expect from that. Like if there is an update, we get new NuGet packages and they just get applied to the solution. And the same is true for the buildchain like CI CD pipeline. You always keep that up to date. You are in the cloud, you actually get updates all three months anyway. Like for example in TFS, in DevOps Server for Microsoft. And that is what they actually struggle with also by the way, with Windows 10 as an operating system in like we do for customers with a blood typing robot actually. So if you go to the hospital and they do blood typing, they actually will use that Robot at least 90% of the hospitals worldwide. And they actually have the problem with the Windows 10 version that is on it because Windows is usually updated and in those environments you can not updated in the usual frequency because it needs to be revalidated which takes a lot of time and the update frequency is faster than the validation frequency can actually be. So they struggle a lot of. They actually struggle with the frequency.

No, I used to work in a regulated environment and used to have to validate each tool and it was like a big project just to validate one tool. And like I said, you just can't pull things randomly create a solution and ship it out there. So definitely an issue.

Yeah. And think about most of the modern test tools is actually in the web.

Yes.

So you don't even know if they are updated. And that is the real problem for them.

Yeah. So people didn't realize that. That's one of the reasons why we use a vendor tool when we started because the tool was actually validated. So it just helped where didn't have to validate an open source tool. Not that open source was bad, just in this environment. It just didn't make sense. So you bring up a good point though too that it's not just browser. Right. Because if you're working in one of these markets, I assume you know Like Medtech and Farmer and all these type of environments that they have more than just web apps. So do you find that a lot of people struggle with automation because maybe they're. They chose a tool that was popular, but that tool just happened to be just for browsers. And then like you said, they have to run against all these environments that didn't meet their needs.

Yeah, I guess the biggest pain point is actually that they don't have a tool that then can support all of their different environments. Because most of the time like companies start with Windows, then they have problems with Windows 10, now they move over to a different operating system like in Unix space, because that is controlled and you have much more control of it. And their current tool is lacking functionality of that. Now they have two tools in the set that they need to validate. So validation just got double DF of validation just got doubled and they struggle more with basic stuff like that. And then yeah, naturally, I mean they are updating to an HTML based interface which is still hosted locally because again, it needs to be under version control. But that's another new technology and most of the tools actually support one or two technologies or most of them actually only one today, like web applications. But if you have a bunch of different technologies on the same analyzer and you need to have one single solution, that is actually where they struggle.

So this is why I love talking to vendors, is that you know, you have a lot of customers, you see things in the real world. So I assume the solution should come up with address of real need you've been seeing that a lot of people are struggling with. And so I believe Test Results IO is one of those solutions that address this because it does something a little different than some other tools where it uses more like a visual validation type of approach or image based approach rather than looking at the code. And I guess that has multiple benefits. The first one I think we talked about, I'd like to get your opinion on is we got into this discussion is if you're using certain tools and you're going at the code level, where you're writing loops and everything to try to interact with an application, at what point do you stop testing the actual applications, the user, and you start just testing the code, I thought maybe we'd start off that as a jumping point. Did I understand that correctly, that conversation that we had?

Yeah, that's the point actually. I mean there are multiple benefits of doing it visually, so doing. But perhaps we need to clarify that first. Like visual testing. Everybody knows visual testing, but Visual testing in the end is just screenshot comparison. So you compare what you had in the past to the current one and you either accept the changes or you don't. And based on that you can actually then file defects against the software. That is not what people are looking for in those scenarios because what they want to have is like a unified approach, a unified access to all of the different technologies that are used on the analyzer. And what we are using is like we are grabbing the screen and also interacting based on visuals. And that's a bit different. So there is functional and visual testing all combined in one so that you can access all of the different technologies the same way and that solves the problem for them. But nevertheless, what is also a problem is in most of those test automation tools. That's the interesting part is like you do a lot of automation, like enter text, do this compare here and press the button. But you never check the individual interactions. So at some point in time the test case fails at step 2022, for example, and says like, okay, I couldn't enter the text or the result was wrong, but the problem happened already in step number two. So what we also changed for those regulated environment is what they actually do as a human is like they verify every single interaction with the system. And that is also what makes a difference in those markets actually.

So, you know, image based automation is not new. So I just want to get why this slightly different. I'm not going to name vendors, but I know there's an open source tool called Sakooli that could do this, but it was kind of flaky because if a pixel was off, you know, it just made it really hard to make reliable. And like I said, there's other solutions that are more commercial based. So how did you get around that? Did you have a different type of approach to get around those type of issues that you may have seen?

Yeah, we work around that by using convolutions. So most of the people know that from convolutional neural networks, like those artificial intelligence stuff. We use something similar to that actually to make sure that we can interact with the elements and also find the elements on the screen, but they can be different. So it's more like customized feature vectors where we can still be able to identify an element and tell you like this is the element with the following probability. And that is then good enough for testing because we can say like there needs to be an 100% probability. And that is what we did with you also with those tests is like we Said no, we want to identify is the site shown in the popup or is the site shown on the full screen? And you can also say like, no, I'm accepting changes like that. And that is where we act a bit differently. Because if you do a pixel by pixel comparison, the chance is extremely high that it will only run once or twice. So you run it on a different PC and it will not work anymore.

Yeah, and you actually show me a demo where you have a field that has a text above it. And so you're identifying it based on the text, but it doesn't matter where that use some sort of algorithm where even if the screen gets shorter and that text may move, it's still able to find the field. Is that the type of technology you just explained?

Yeah, that's also. That's the technology on top of that is actually to identify that. Because most of the image based approach is actually. And you see that in those demonstration videos. Most of the time if you look at it closely, if you know how it works, it's like most of the time they select the label and the text field and say I have an image now and now I can enter some text. That is true. But actually if, if you resize the web page or resize the application, if it's responsive, the field will be somewhere else after the resize operation. And that is why we use actually spatial algorithms to identify the relation of the text field to the label. And the only thing that we need to know is actually your reading direction. So if we know that you're reading from the left top to the bottom right, you can actually identify the text field that you as a human would associate to the specific label. And that is how it works underneath.

So I know a lot of companies where I used to struggle with is they had a build a version that was US based, so it had us text and then they had to do a localization that ran in different countries with different text. How would you be able to get over that using this type of approach? If you're using like a visual.

Yeah, it's for stuff like that. So if you're a text based actually using ocr, but it's not a typical kind of OCR because you know that from your history as well. If you just use plain OCR most of the time you are looking for single words or double words and the failure rate is 99.99. The corrective rate is 99.999%. But that means on a thousand letters, on a. A4 paper, which is black text on white background. And for screens, usually if you do OCR on those small single line elements, you get wrong characters. Like the typical one is an O replaced by a 0, an small L replaced by a 1, and stuff like that. So we do a reverse OCR. That's another, another algorithm invented by us actually where you know in advance what you're looking for. So if I know that I'm looking for the label and I get back from the OCR like one, then I know exactly that that is actually the label based on some distance calculations that we do. And that goes back to the spatial computing that we use for the image detection as well. So what we do is like we always use the test first approach. It's like we don't have an option which says give me the text and then I need to find the information somewhere in this text. It's like, hey, you are looking for this text, so tell us what you're looking for. And we make sure if that text is on the screen and we give you the context of that position on the screen.

Nice. So you also have something, I guess it's like a model based type of approach that's kind of unique. Like over time you have all these screenshots already in your history. You may not have tested a specific flow, but you already have the screenshots in order to create that test case. Can you talk a little bit about that approach? I believe it's like a model.

Sure. The idea behind that is actually that you don't need to change all of. So the typical approach, you have test scripts. And as I'm not allowed to name any vendor, I guess at some point in time in the past, somebody came up with a clever idea to say like, hey, having individual test script doesn't make much sense. Because if you need to change something because the application changed, we need to change all of the different test scripts. So we remove the interaction logic. So how do I interact with the application into a model and I just have my scripts working against that model. And that was like 20 years, 30 years ago, I don't know. And we use the exact same approach. The only benefit that we have is actually we are capturing screenshots all the time and you tell us which area are of interest for the user. So based on that information, what we generate in the background is a visual twin. So that means you have a model that you can use actually to access the application in a normal fashion. Or you can actually define a test case based on a visual twin. That means like in the web browser, you have your full application available even without having the application available. So you're working on screenshots and interactive elements of your screenshots. And you just write a test case based on going through your application as if it would be the real application. And that is how we brought the model based testing to the next level.

All right, just something that just came to my mind was interesting. If you work like in a regulated environment, sometimes getting these environments up and running are difficult. So if you're onboarding someone new, they may not know how to get the application up and running on the machine or anything. But it sounds like if they use this approach, could they get started or getting their feet wet? Understanding the application because it's working on the model, does that make sense or the image?

Exactly. Yeah, that's the point. Because for example, for the blood typing robot that I just mentioned beforehand, you need to put in some blood samples, you need to put in reagents, you need to put in additional consumables and stuff like that. So till you know how to start takes roughly two to three weeks actually. And then you're able to start all of that stuff and have everything set up and can start the software. And with that model, you just start instantly because you don't need to take care of that.

So as I mentioned in the intro, I'm actually checking out this solution. I've been messing around with it for a few weeks. I'm going to be dropping a review this week on it. What's interesting as well is you have two approaches which I like. You're not like a dictator where you have to do it this way. So you have like a code based approach where if someone's familiar with Visual Studio, they can start coding up these images, but not with a lot of code. It's just, it's like a code based approach. I guess I'll get your opinion on, explain it better than I am. And then you also have like a no code type of approach. So can you explain both approaches and how maybe your code approach is different than what people may think when they hear a code based approach?

Yeah, that's a good point. Actually, Joe, the thing is what we decide, we definitely decide for low code approach because I don't believe in no code because I've seen a lot of different tools and usually if you have no code either, you don't have all the capabilities that you need and somebody has some box where it says like you can put custom code in this box and what ends up in a year of test automation is like most of the testing is then in this little box in there because you can do everything in that little box. And that actually breaks the whole no code idea, which gives you maintainability and stuff like that. Because nobody can do something specific. And I haven't seen a tool yet that doesn't have this. You can put any kind of code in this little box which is then misused. And on the other side, like take Selenium, that's also open source tool. I mean, it takes some programming knowledge to actually be able to use Selenium in a fashion that is actually reliable. Yeah, for sure. You can get your first elements within minutes, that's no problem. But to make it reliable, you actually need to understand loops and whiles and stuff like that. And also different coding techniques. And you can also see that actually testers giving introductions to how to code. So they have coding courses meanwhile, where the profession is actually testing. That is brilliant. But you see that there's a lot of additional effort required if you do have a full coding environment. So what we decided is we give you the full coding environment, but we only give you low code access initially. So we do have something that's called codified expertise. That is where all our expertise was put into code. So if you think about visual testing, it's like if you open a drop down, it might drop down to the bottom, it might drop down to the top, it might also drop out from the middle. And all of that is just a visual representation of it. So you need to understand like what is the difference to the previous screen? How do I actually detect that dropdown? How do I handle it? How do I scroll within that? Because you only have pixels, so there's no additional meta information on it. And that is, that is what we did for like the last 15 years with and that is why we came up with codified expertise. So you can easily use it, like for scrolling. You don't need to KK at all. It just scrolls out of the box. I think that's something that you've seen in the demonstration as well.

Absolutely. What I kind of like about this approach is it goes back to the basics of testing. I think a tester is a tester. They want to test the application. They don't necessarily want to be a developer. Right. It's two different mindsets, but it's almost went out of vogue. And it's like, well, testers are developers too, so they need to learn about all these crazy development concepts that I guess it's good, but it's almost like almost lost sight of testing. Right. So this kind of unifies it now back to where it was.

Yeah, that's another brilliant point actually. Because what we want to do is actually, that is actually my mission is like I want to combine development know how and testing know how. So we do have developers and we do have testers and we know in today's world everyone should be able to do everything, but it doesn't work out. But what I want to be able to is like the developer can add additional functionality because they can code and the testers can afterwards use them in the model. So the testers have a low code approach. They create all of their dedicated test cases against that model provided by the developers. And then the magic appears because you can now use those tests that were written by the testers and actually put that in your CI CD pipeline because it's actually part of a development tool stack. And that is now you really see the benefits of having testers in there doing all of the tests, writing the test automation and then bringing it back to the pipeline. So for every iteration now, their test will be executed. And that's one of the typical problems that you see for testers. Like, hey, I tested all of that stuff, but nobody cares. It's like if I put something red on the dashboard, everybody cares. But if I don't put something red on it or not enough red, then nobody takes care of what I'm actually doing, of seeing the benefits of what I'm doing. And if I can bring back my stuff to the developers and they use that as test as like they use their unit tests today. Like I have full automated end to end tests or like the full system is tested based on the test of the testers during the CI CD pipelines. That is actually what we, what our goal is because that saves a lot of time in the end.

So I know another thing your tool help addresses is not only does someone need to be a developer, they almost need to be like an infrastructure expert nowadays and know about AWS and configure it to run in parallel, all this crazy stuff. So I believe you have kind of a workaround where you're able to scale up a lot of VMs automatically for folks. So maybe talk a little bit about how you achieve that or what that technology is like.

Yeah, so that is actually what we needed also for one of our customers in the beginning is like we need to be able to scale. Because the problem is like if you do have automated tests, you Also want to run them. That is the interesting part because most of the test automation tools actually trick you in. You have a few design license and afterwards you need to buy a lot of runtime licenses because otherwise you cannot run them overnight. And that's the interesting part. And what we said is like hey, why don't we just make it simple and say like hey, you say I want to run that in the cloud. And you just run all of your tests automatically simultaneously in the cloud and we just bring those VMS up, run the tests and bring those VMS down for you. And that saves a lot of time and money to be honest. And you don't need to have a kind of device farm anymore because it's just spun up if you request it and gets down afterwards. That is actually not how cloud providers want us to use their cloud. That is also something very had some discussions on it, but that is brilliant for testing.

So another thing in regulated environment that I saw a gap in, I'm not picking on open source tools, it just takes a lot of effort to get it right. Is traceability and dashboard. Not dashboard, but having like a portal where someone can go in and know what test map to what and if an auditor came in, they can give them just to the portal. So can you talk a little bit about the portal feature and how it helps maybe especially testers in regulated markets.

Yeah, that's also point is the traceability in that. That's true. Usually you do have some and they all work honestly, all of those tools like Selenium, Appium, all of those tools, they work brilliantly and use them. You just don't need them anymore if you use test results IO. That's all. And the problem is all of them is like they generate log files or they upload the reports somewhere or you actually have to build around stuff. And that's the problem is like in regulated environments if you build stuff around, you need to validate that stuff around. So you need to think about like a complete system that needs to be validated. And that's the point if you want to have some traceability, which means like you can trace from the requirement to the specification over the software detail design down to the test case. Then you have different test reports which mean you have executed test cases. And all of those executions then can still be traced via the tracing chain to the requirement actually on the top. And you need to imagine that you need to create like trace tables in regulated environments where you actually trace from the bot tile from the top to the bottom and from the bottom to the top in both directions to see there's nothing missing in all of those change. And that is actually what we address with all of this traceability in the portal. So if you put a test case into the portal, it is automatically versionized, so you have a version on it. If you change it afterwards, you get a new version. If you execute, you get a new revision of the execution. And you link all of that to the environment in which it was executed and the software version that it was executed against. And that means you have a full traceability of your test case without adding any additional stuff on that. So that is like part of the central core of it that is already validated.

You know, once again, regulated environment. We had to save our environment, freeze it in the code and the tests that we ran on for the validation phase, so a verification phase. So if anyone came in to do an audit, they want to see the exact environment. And it was a nightmare to get together. So it sounds like this helps overcome that type of issue.

Yeah, that's the point. For actually uses VHTs and ISOs and you just archive them like you would usually do. And if it comes back to an audit, you actually just can spin them up and that's it. The software knows exactly which version of the ISO VHD file was used. It knows exactly on which cloud provider it was instantiated and knows exactly which test case revision was used, which software version. And also the software binaries are also in the portal. So we can regenerate the state that was present during the initial execution at any point in time.

All right, so I just think it's important once again to stress that with this approach, the main benefit would be you write it once and you could actually literally run it on a Windows machine, a Unix machine, you know, a browser, a web app. Am I understanding that correctly? Like, what is that? Like the main selling point, you run at once and then you can just run against all these other other types of environments and OSes and things like that.

You can do that, but usually everybody claims that and most of the time it doesn't work. So what we actually claim is like, based on what you mentioned before, and it's like this model driven approach, so you generate models for the different environments where you want to run the test case on, and then you can just run this, then you actually can run the same test case against those different environments. And that is true because you are actually interested in a test case and not in how to interact with the software on a technical level.

Absolutely great for Tobias. Before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts? And what's the best way to find contact you or learn more about test.

Results IO so the one advice is actually before you automate, think about what needs to be automated and where is the most value in it. Where can you actually save the most time in automation? Just automating for the fact of automating. That is fun, but that is what developers do. You shouldn't do that. And actually, where can you find me? You go to www.testresultsio or you write me an email tobias.muellerrogilech or you actually connect on LinkedIn. It's tobias.mueller.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a424 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.