Model-based Testing vs. Recording—Which is best? Matthias Rapp & Shawn Jaques

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. And today you're in for a special treat when we talking all about model based testing and record and configuration based testing, two hot topics I hear a lot about from experts on why it really saves a lot of time. So we're going to really go in deep with this. So we're going to go over why they're different, what are they, what would you use versus one another? Configuration testing may not be as familiar with you as a model based testing. How can they work together? And so we have two experts joining us to go over that. We have Sean, who is the director of product marketing at Tricentis which is focused on SaaS based test automation solutions. Tricentis through Testim. I love Testim, so great acquisition with that. He has over 20 years of experience, so really great to have him on the show. We also have Matthias, who is a test automation expert and Tricentis veteran. He implemented, deployed and sold test automation solutions internationally for many companies over the past 15 years. He recently was GM of the popular Tricentis sponsored open source project SpecFlow, another favorite here at the Test Guild. And he's currently designing and overseeing creation of the next generation product yet to be revealed as VP of Product Management. Really excited to have them both on the show. So let's get into it. Hey guys, welcome to the Guild.

Hey, thanks for having us.

Thanks for having us, Joe.

Great, great. So I always ask this is anything I botched in your bio that you want the Guild to know more about?

I wouldn't say just that we try to have a lot of fun at work and you know, laughing and you know, interjecting some comedy is definitely part of the daily work.

So yeah, plus very international, very international work. So that's one of the things that I enjoy actually a lot to, to talk to many different cultures. We have offices in so many places these days. So that's really enjoyable to me.

Great. So I'm already starting kind of off the rails already. Sure. I'm just curious to know, you worked at Testum, you were acquired by Tricentis. A lot of companies are acquired by others in this space. Any tips you give to people on how to really make a smooth transition so you get the best of the acquisition and the customers really. See, that's a good question.

I must say that, you know, going from a small company to a large company, I'd worked at large companies before and while Tricentis isn't as large as IBM where I was, it certainly was larger than Testim. And I just said, you know, you just gotta embrace the process and just be a disruptor. Just do the things that you think are the right things to do and the processes will work themselves out.

Absolutely. All right, so I guess let's get into the meat of the topic here. You know, we're going to talk about model based testing, but I think a lot of folks may be not really familiar with recording configuration based testing. So I thought we set the stage, maybe set a definition for we're talking about here. So Matthias, any questions on, you know, our thoughts on what is how you would define model based testing?

Yeah, sure, sure, let's, let's talk about that. So model based testing, to me at least, right, is in its essence is the idea of producing reusable artifacts of automation, of different automation bits and pieces. So that's the essence of it. So whether it's, you know, it's a reusable function that you want to program, whether it is, you know, a technical layer that represents how you're identifying controls on your application, then how you're parameterizing this and what logic you give it, and then, you know, what data you're driving it with, what parameters you have and variables you have with it. These are all layers and bits and pieces that you have to think about and go through as you assemble a well thought through test case. And model based test automation really segments all of these and puts these into specific bits and pieces you can plug and play with. And that's the heart of it. So that's what model based testing really is.

Now I know a lot of people know a little bit about model based testing, but I think there's some next generation model based testing where before you had to come up with the model and make sure you had belief in what the requirements were and what everything was doing. Now it sounds like almost you can use technology to sniff production and things like that and to actually make the model for you. Is that something you've been seeing as like a next generation type of shift?

Yeah, really good question, Joe. So that's definitely where we currently see the trend going and also where we are going. So we are actively working in that area that we can basically generate some of these models from actual User inputs versus, you know, tediously assessing it after the fact. So definitely where everything is going. AI is actually enabling this and driving it, making it possible these days. So I think there are some exciting opportunities ahead of us in exactly that area.

Absolutely. So talking about AI, Sean, you worked at Testim, so a great segue there. But not AI based question. I'm just curious to know how you would describe what is record based testing?

Yeah, so it sounds pretty simple. So you basically record what the user is doing within the UI of the test that you're, of the application that you're testing. And so it's, you know, you turn on the recorder, you start to record a flow. If you think of an E Commerce app, you might, you know, log into the app or search for a particular product. You click on that product, you add it to cart, it kind of records all those actions and then puts it into the. To an editor that you can, that you can edit. Now there's two ways that those tools do it. One is that kind of do the editing after the recording, and the other is kind of do the editing while you're recording. There's pros and cons of both approaches, but Testim is more on the edit after you do the recording. But I just say that, you know, there are kind of record and playback kind of 1.0 tests that were pretty static. They're almost like a video. And then there are kind of where the market has moved in this record playback area where everything that gets recorded is this unique separate object. Then you can take action on or you can share across different tests, or you can delete it and remove it and group it and share it. There's just lots of different flexibility that has happened as the tools have evolved over the last several years.

Could you just explain a little bit more? It's not record and playback. It sounds like you're recording the app. It's doing maybe screenshots and it's creating a model for you. Then after the test is done, can you go to the image then and create a test based on those that model it created? Is that off base?

Yeah, it's a little less like model based testing in that you're not really seeing necessarily a flowchart of the way that a user would progress through the app. But what you are doing is following the steps that you click on. So think about a simple login process. You know, you click the login button and then you enter your username and you enter your password and you click login. And each one of Those steps is recorded. Each one of those tasks that you're executing in the UI is recorded as a separate step within a test case. And then you can add different things to that test case. So if you say, well, I want to add, you know, data to it, I don't want to just use the username and password that you have. You can configure it to, you know, use the data set that you have. You can also say, I'm going to save this little group of steps and reuse that in other tests that I do. You can decide that, you know, maybe I had a tab in there between the, you know, password and the user ID and the password part and I want to delete the tab because I don't need that thing. So there's a lot of like manipulation you can do afterwards. So once you get that recording down and it's in your editor, then you can do a lot of manipulation of it.

Nice. Matthias, how would you describe then how model based testing may be different from recording configure based testing?

I think the big difference is so, you know, if we do what you, what you were hinting on Joe before, that we are basically auto generating some of these test cases from observing users, for example, I think that's where it's blending a lot. It's also blending, I think, in the refactoring space, those two approaches. So essentially what we are emphasizing with model based testing is to have these bits and pieces and every single concern that a tester may have is a bit that you can use and the same sample into a sequence. And the reason why you want to do this is because you want to be, you want to go from one test case to many, right? Reusing all these things. And you also want to be able to maintain and update these things easily. That's the reason why you're doing it. But a smart, you know, record, record based tool like testim is, is actually doing a lot of that bits and pieces creation as you record, right. So that's why the worlds are blending. So basically it generates some of these reusable bits and pieces that you then can assemble from use case 1 to the next use case that you want to do with it. It might not do all of it. You know, there is some complexity still in generating that model of the business logic, especially when you come to more complicated long process chains. There would be very sophisticated AI you need for that. So I think, you know, while we are trying to spearhead in that direction, you know, industry is not quite there yet, but this is where things are blending nice.

So, you know, obviously you want to be able to maintain and make your tests more maintainable. Someone's hearing about a model, does it make tests more maintainable, less maintainable, the same? Maybe we could talk a little bit more about maintainability. Does model based testing help with reuse or that type of issue?

Yeah, sure, very good question. So I think it depends a lot on the length of the test case to some extent what path you want to go down. So if you have. And also how often you want to use it in slightly different flavor. So if you're in an environment like that where you are going to assemble very long test cases where you may have a lot of dependencies, if it's not your small agile team that has a single application they're responsible for. But maybe it's more the direction of, hey, I have many different apps. Some of them are even in a different technology. You know, some one is written in Java, it's a desktop application, the Other one is JavaScript and HTML and it's a web application. If you're in a world like that, it makes sense to, you know, to emphasize these bits and pieces more, you know, creating these models, extracting it some more, allowing the tester to automate these different applications the same way, you know, not having to learn JavaScript and having to learn Java respectively to automate against those, but abstracting that away with a model and also creating these bits and pieces. So in those environments I think that makes a lot of sense. But if you are in a more, less dependency rich environment, I would say more contained to your team. It's a very specific tech stack. You can predict better how you're going to use it going forward. I think you would be probably better off with just a very swift creation experience that emphasizes bestly aided by AI, which would be a record based approach.

And Sean, any thoughts on reusability or maybe the record based, same type of.

Yeah, I think both of the approaches have a lot of aspects of reusability, you know, so depending on how the tool is implemented, you can share every test step, every group of tests, or even every test case can be reused in different test cases. So there's definitely reusability in both of them. I think the key difference or the way that Matias has set it up is a lot of times when you think about model based testing, it's decoupled from the underlying infrastructure. So if you are moving from a Java application to a JavaScript application, the Essential, the model that Is, you know, following the different elements and the test cases that they're going to. They're basically the elements and the steps they go through to create a test case that can still work in either one of those two environments as long as you connect it to the underlying technology. Whereas Record Base is really focused on whatever technology recorded in. You're kind of limited to that technology.

Nice. I've always heard people that use model based testing. A lot of times it's easy to make changes because you just make it in the model and it automatically will update all the tests for you rather than having to go through each test finding all the page objects and all that. Is that one of the benefits as well? I assume certainly.

So a single point of maintenance is what comes to mind when you say something like that. So that's the goal. One place where you propagate all these changes from.

Awesome. So a question I get asked all the time is there's so many tools out there, how do I decide? Now not only are there a lot of tools out there, you know, huge companies like Tricentis, which are experts in this space, have multiple tools for multiple scenarios. So I believe you also have Tosca, which I think is more of the model based type testing approach. And then you also acquired testim, which is AI, mostly browser testing, last I checked. So you know, what are some test cases that would drive maybe using both these products or how do people know when to use one over the other? Do you have any rules of thumb that you try to help customers decide or what do I use?

So yeah, this is one of the first things that we tried to tackle when we came into Tricentis is to figure out how do we explain it to our sellers and explain it to our customers. The difference between when somebody would want one versus the other and it really comes down to the applications that you're testing. So let's start with the technology. If you're testing just web based application, then you know, that kind of fits in that test and model. Tosca does web based applications, but they also do, you know, 160 other technologies. So it is, you know, it's truly a robust tool that spreads across all the apps in your enterprise. So if you're thinking about, you know, what team might only be concerned with one application versus, you know, all these applications, we think of that Agile, you know, development team is building this customer facing application, you know, that is probably iterating a lot and changing a lot. And they really like that record based testing because they can quickly come in record a test for their feature that they're building and have that serve as a gate to their CI. And that is, you know, really quick and easy to learn and doesn't require a lot of, you know, expertise in the tool. Whereas, you know, the Tosca users or those model based testing that are going across these long end to end business flows that might start in a web app but then, you know, go into SAP to get inventory, or go into Salesforce to create an account, or go into netsuite to do the financials. Those kind of business flows are much better served with a tool like Tosca that a QA team that sits within it can model that application business flow across all those different apps.

Sounds like enterprise kind of grade, but I don't want to say Testim is an enterprise grade. Is that the wrong way to.

Yeah, we think of testim is kind of sitting within that agile dev department within an enterprise, but really not being the kind of end to end across that long business process flow within an enterprise.

So once again, I don't mean to keep bringing up testim, but when I hear Testim, I think of AI. It's one of the early solutions out there that incorporate AI. And so I'm just curious to know, you know, we touched a little bit on AI, how maybe it can help with the model. But I know AI machine learning are kind of thrown out there nowadays, it's just buzzwords. But can they be applied to maybe model based testing to make it better or record based testing? Matthias, any thoughts?

Yeah, it certainly can and it should and it must if you ask me. So I think where you know, these record based test automation solutions that are out there right now, and especially what testimony spearheading is how to really produce, you know, stable automation from the get go as part of the recording by leveraging AI. Right. And that's great because that takes a lot of the headache away from unstable recordings. Right. If we look back in history, we know that this was the reason the recording tools failed in the first place. It was because they recorded based on parameters that were by default unstable. First recording tools went for screen coordinates, as we all know, 30 years ago, and that was never stable because if you run it on a different machine or even a different day on your own machine, it won't work anymore. So testimony is doing a great job at doing this and leveraging AI for that. And I think that no matter what automation tool you are these days, you need to incorporate some of that to have more stable locators or identifiers the other thing that, you know, future automation solutions have to do, in my opinion, is to help with giving users more guidance as they're producing their automation artifacts. And Testim is yet another great example of doing this because Testim is actually looking at, okay, you know, I saw two different users creating, recording a flow right. Of the application and Testim then figures out, well, parts of that flow are actually the same. So it's automatically merging those automated sequences for the users, making them reusable, making them a single source of truth by detecting that pattern of similarity. So great solution for that problem. And you know, we see that that's definitely, you know, driving the market forward in that direction. And similarly, you know, we should and have to apply the same patterns for model based solutions. So when we have issues with object detection, we and locators, we should use probability and redundant locators where we make decisions based on that AI aid to figure out what's the best one in this situation and therefore stabilize the test. And the same goes for pattern detection in the actual test flows. So I think this is really where we need to be. But then even take it further with making even more suggestions to users on how they could assemble a better and more suitable test case going forward. This is really where the power lies.

I love that, bubbling up insights to say, hey, just, just so you know, 90% of these do something. I love that.

Exactly. You know, 90% of the market or 90% of your peers are doing it that way. Why are you doing it? Right, that would be most ideal. Exactly.

How about analysis? A lot of times I see people spending all their time looking at test results and going, and then the next runs off and then it's too late to even debug. I would think machine learning would be perfect for this. Do either these solutions, either TOSCA or Testim have something to help with the analysis of? Maybe if you had a lot of failures.

So yes. So one of the ways that Testim helps with that is we kind of aggregate like failures. So if you are seeing kind of recurring failures, it'll suggest to say, hey, this, you know, you're not finding this particular element, you know, you go fix that element and then your tests start to work again, all of them. And so it does kind of help you triage a little bit. You know, those results. We also will suggest what the root cause is of a particular test, a failure. So if there's a failure that, you know, we've seen before, or we believe we know what it is, we suggest that failure and then you can tag those failures and that gives you that history of, you know, where are my failures coming from? Are they coming from bugs in the app? Are they coming from, you know, flaky locators? Are they coming from some environmental issue? You know, if it's something like an environmental issue or a network issue, then perhaps you can fix that systemically in your test environment. And then, you know, all your tests start to improve. So there's some things that we think kind of facilitate that continuous learning and improvement through the analytics.

Nice. So you know, we mentioned Tosca model based testim is a little more record based. Do they ever talk to one another? Would you ever use them in the same flow? I mean eventually I'm sure once you would be, they'll be integrated, but I'm just not. That's not just a prediction. I don't know. Nothing but. Any thoughts on. I guess the bigger question is if people don't have these tools, if someone has a model based testing tool and someone has a record based testing tool, would they ever need to talk to one another or would they ever help each other out?

Joe, My comment would be that they certainly should because I think the context is a bit different. As we've pointed out before, when you would use one or the other and where they come together is actually when the scope increases. Scope increases basically for where you use the record based tool. So anytime I would say when you're a user of a record based tool and then suddenly you want to reuse that test artifact, you the test case, maybe bits of the test case in a larger context. So for example, to Sean's point from earlier, you have a custom built web application, but that application is integrated with an SAP system and a Salesforce system and maybe some other systems downstream or another custom built application and you're starting to end up with that process flow. This is when these tools actually would come together, right? Because you would want to use for the two custom built applications in that flow, you would probably want to use the record based test cases, but they would feed into the other test cases that are taking care of the rest of the workflow and there needs to be an orchestrator of this whole thing. And this is how we currently think about this. So this is where model based also becomes potentially the orchestrator of, you know, the individual bits and pieces where you, where you have used record based.

The thing I would just add to that is that there's a lot of organizations out there that are using some sort of test management tool. And that spreads not just in the UI test, but across kind of all the different kinds of testing that you're deciding to use. And you certainly see both these tools fitting into a test management strategy that would say we're going to use this tool for this part of the testing and this other tool for the other part and kind of coordinate it all at that level.

Very cool. So I was actually at Starwest a few months ago. I know Tricentis was there and there was some rumors, like low whispers about that you're all working on a new model based type of record based automation tool. I don't know if we're able to talk about it here, but it was Some sort of SaaS like model based testing solution. Anything you can reveal here about what that is or what that's going to be all about and when people can try it.

Yeah, we can. We actually can. Joe, just for you obviously, but. And for your audience.

No, so we are actually. This new product is actually at this point in open beta, so anybody can try it out. That's curious. So it's not a full on secret at this point anymore. The name is Transientis Test Automation and it's a brand new SaaS platform that is bringing model based test automation to the cloud. And we're very excited about it.

So the cloud, everyone talks about the cloud. What's the benefit of this, of having something in the cloud? If people are like, it's just another tool, why is this beneficial?

So it's a very good question. So it's beneficial because one of the main reasons that customers have basically asked us, you know, look, I love your model based solutions. They provide tremendous value to me. Right. I'm operating a central QA team, I'm responsible for many different applications. But I really need a lightweight solution that I can manage where I get the benefit of the cloud.

Right?

The benefit of the cloud for most of our customers is that they can scale their executions. That's really what everybody has in mind. You know, get infinite cloud resources, right. And you can run as many tests in parallel as you want. You don't necessarily have to operate a test lab anymore, which you do if you have something not in the cloud. Right. So those were all reasons for us to completely reimagine model based test automation and leverage cloud as actually an advantage, providing that benefit to our users.

I don't think a lot of people think about scale when they're starting off. So if you start with a cloud based solution, it's something. Just another thing you don't have to worry about because it's kind of built in almost for you from the start. Sounds like.

Exactly.

It also helps with those remote employees or the different teams in different locations that all can contribute to testing and they can share objects in a central location that's global.

And this may sound silly, you won't imagine how many people I've talked to that have problems just installing software sometimes. Like getting Selenium. Not there's anything wrong with Selenium, but getting Selenium on the machine locally for a developer and up and running. It's like a big tutorial I guess with a cloud based solution. Just log in, there you go.

Right. That's the ultimate goal, Joe goal I guess of any, of any cloud product.

Yeah, all the cloud products have some sort of software that kind of ends up on a machine. It could be simple as a chrome extension or a little, you know, plug in or small software. But yeah, that once you've got that installed then you're just off and running and, and it does make it a lot easier. And the other thing is, you know all those updates that you would have if you had an on prem solution, you can just avoid those because they just get pushed out to the SaaS automatically.

That's a big feature for sure. Absolutely. So you know, we talked a lot about model based testing, recorder based testing. I think a lot of people may be doing just traditional automation testing or automation scripts. Is there any advice you'd give to someone if they want to start moving towards a model based type of approach or a cloud based approach? They need a switch up tools or is it just a different kind of mindset?

I think the best way to get started with this, it's not a different mindset. I think a lot of, you know, even any experienced test automation practitioner probably knows about the concepts of a keyword driven framework or a data driven framework or things like that. Right. And even if you use Selenium at some point you run into, into these thoughts because you have to really be mindful how you, you know, build a well structured project in your code base that you can, that you can maintain and manage. So if you have that mindset, you basically get an off the shelf solution that takes care of these concerns of this good automation architecture in your model based product. Plus you get the benefit of abstracting it so that it's readable for business users and you get a sense for what is actually being tested without having to debug or read through the code. So I think if you come with that mindset, this is really where a model base feels somewhat familiar to you. If you're completely new, I think to test automation, you're probably better off with and have no prior experience, you're probably better off getting started with a record based solution and just record a simple use case, you know, see how it goes, expand from there. Right. And that's basically how you can get going.

Yeah, I think, you know, if you are using, you know, a framework like Selenium for sure, you know, you have that kind of experience of, you know, starting it at one point in an application and you know, going, you know, writing scripts that basically get you to another point in the application so that it's fairly similar. When you come to a record based tool, you're starting at one place in the application, you're clicking through to another place, you're creating your assertions and you know, that's how you get a test case. Model based testing does require a little bit more, I would call it crafting the test case or maybe thoughtful planning for how you want this test case. Because you're going to think of, you know, what are the objects within the UI that I'm going to interact with. You say, okay, well I'm going to capture all those objects and then you're going to start, you know, putting them together in a palette. You know, I'm going to like the login scenario. I've said, you know, you've scanned all those little objects and now you start to put them together in an editor that would represent how somebody would go through that web case. So it does take a little bit more thoughtful consideration. I would just say when you're going through that part of it and a little bit more of understanding the tools so you understand how these individual piece parts or modules then, you know, get added together to create a test case. So it does require a little bit more planning and. But you know, by the same token, when you do come up with that use case, you do get the benefit of, you know, the reusability and the, you know, the modularity that you can then use across different applications. And it really does help, you know, kind of set you up for longer term success.

Great advice. Okay guys, before we go, is there one piece of actionable advice you can give to someone to help them with their automation testing efforts and what's the best way to find, contact you or learn all about Trisense's awesome test automation projects and solutions?

I guess I give two points of advice. Number one, automation is a really exciting Discipline, I hope everybody knows that is listening to your podcast, Joe. So, you know, keep going. It's a really exciting discipline and stretch the boundaries of what's possible and what's doable. And the second one is that, you know, stay tuned on what Tricentis is doing because we are really trying to give you guys the right tools to be successful with test automation. And I hope you guys are going to check out Tricentis test Automation once it's available.

Yeah, I'll just add that I think that there's a tremendous amount of benefit from test automation. And you know, if you're currently manually testing, you know, whether you're going to jump into record based testing or into model based testing, I think, you know, it's just get out there and give it a try. And you know, within testim or within Tricentis, there's ways that you can try the test and product in just a matter of minutes. You can be into the product. Same thing with Tosca. And soon with our new product, you'll be able to try them out very quickly. And in fact, if you guys are interested, if anybody's interested in trying out the beta for this new technology that we're working on, it's actually out there on the website. You can get into the beta really quickly. Just go to tricentis.comttabata and that'll get you into the beta.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode, head on over to testguild.com a427 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space, or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.