What is Reverse Mutation Testing with Leonardo Lanni

Hey, what is mutation testing? What's Reverse Mutation testing? That's what we're talking all about today with our featured expert, Leonardo. If you don't know, Leonardo is a globally recognized quality engineer. He has Expertise with over 18 years of experience from E health to E Commerce, fintech to mobility. He has built and transformed QA teams across startups and enterprises in the us, Europe and beyond. Also, he is the founder of QA Roots from really excited to have him on the show. I caught this LinkedIn post about reverse mutation testing. We never had anything on the show about this, so really excited to have him on. You don't want to miss it. Check it out. Hey, let's face it. Mobile app testing is broken in more ways than one. QA teams and developers are drowning in manual visual reviews, flaky emulator tests and false positives that waste hours testing across real world devices, often out of reach. And while functional tests may pass, you still end up shipping misaligned UIs that break trust with your users. And that's where App Percy by BrowserStack steps in. App Percy is a purpose built visual testing solution for mobile apps designed to fix the pain points that frustrate modern teams. With a single line of code, you can run visual tests on over 35,000 real devices. No simulators capturing only meaningful UI changes thanks to App Persy's Visual AI. It handles dynamic content, layout changes and anti aliasing noise so your team can review confidently and move fast. Plus it works seamlessly with your cicd, git and testing frameworks whether you're on Espresso X UI test or Appium. For teams aiming to ship aesthetic stable mobile apps faster, AppParsey delivers where others can't. It cuts review time scales effortlessly so you can release your mobile apps with confidence. Visual testing isn't optional, it's the experience and App Percy makes sure you nail it every time. And even better, App Percy's base plan is absolutely free to use. So try for yourself and enhance your end user experience drastically with AI powered visual testing. And you can check it out using that special link down below. Hey Leonardo, welcome to the Guild.

Thank you very much. No, no issue though. I'm very honored to to be in the Guild. So happy to be here. Thank you for hosting me.

Great to have you. I guess. Before we get into it, I always am curious to know, how did you get into testing?

A very good question actually, because back in my young days at IBM I was a software developer at IBM. What happened? Long story short, I think it was around 2009.

Performance and scalability testing was, you know, the new hot topic back in the days. So IBM was building up a team of, you know, people that would do this. Automation basically consisted in testing systems where you needed to create a lot of agents. Back in the days there were not many, many tools. So the idea was to build up a team of experts that would start doing performance and scalability testing. And the idea was to pick people from developers because, you know, those guys have a little bit of, probably a little bit of know, how about, you know, building scripts and programming and so on and so forth. So there was a possibility of this pilot and decided to join it because I thought, you know, it was a very cool idea to start doing something that was a little bit edgy on the frontier. And that was basically the beginning of my long term adventure and relationship with, let's say in testing.

Love it. So I started off as a performance testing as well. When did you start getting more into like higher level QA topics and why? I guess you started a company that actually helps people with this as well. Why pivot that way as well?

Well, there was, let's say a chain of elements. So once I move into performance tested that really opens a new world for me. My claim, my perspective was more into the building things as a developer. But then that really opens an amazing world about quality and quality engineering. And you know, the more I was working in that, the more I thought, you know, there is so much room to help companies and to, you know, make sure that the products, the services that we all love as customers, as client, they get delivered on the proper standard. So throughout my journey I step by step I move from, you know.

Starting from an individual contributor in a team and helping with the performance testing. Then slowly, slowly I start getting roles where I was, you know, the, the responsible, the head of QA or the QA lead of the person that, you know, would work on the quality engineering, on the strategy. So most of the times I was collaborating with, you know, head of engineering, ctos and I really like that because, you know, beside the test automation, which is clearly an important part of the process, there is a lot of strategy and a lot of consideration to be done in quality engineering. And I really like also to try and keep my standpoint as the final client. So the final goal is eventually delivering quality and delivering business value. So throughout different roles and different companies, startups and in other countries, I ended up having more and more responsibilities until I told, probably for me it's the time to do this as my main mission, my main job, to help companies deliver great quality. And, and that's what I'm doing now. And I'm very happy with. It's very, you know, motivating, rewarding and helping companies deliver great products. I think it's, it's, it's great.

Love it, love it. So, you know, you've, and I guess the answer is probably depends, but I don't know. You work all over the world, you work with all kinds of different companies. Is there any commonalities you see people struggling with, with testing or is it, is it just different depending on the vertical they're in or the country they're in?

I would say both answers are correct. So there is a common basis into doing testing. One thing that I see in a lot of projects and companies, they all try to follow the testing pyramid. You know, doing a lot of unit tests, a little bit of integration and as you go on the top, just a few end to end and you know, exploratory testing. So there are some generic guidelines that I more or less I can see almost in all the projects. But that's, I would say the boundary in what is common. Then it's quite interesting that every single company does quality engineering in a completely different way. Their company, they really make an effort to deliver a great job. They create book of standard, they try to make their own way to deliver a great application and they invest a lot into creating center of excellences or you know, if you need to build an app, don't start from scratch but rather download something a little bit. I would say battery in clouded so you know, contains a lot of things that are done under the quality of the company. And then on the other side of the spectrum you got also companies where there is basically nothing. It's just, you know, the quality relies on the ability of developers and knowledge on developers. But you know, in these companies that's where the magic happens and the fun and you know, try and build something that help deliver great quality. So it really varies depending on the company, but probably the testing pyramid is something that everybody more or less agrees on.

Awesome. Awesome. All right, so as I mentioned, the meat of the show I want to cover on reverse mutation testing. I saw this on LinkedIn on my, on my feed and I covered it in my new show and I think it was one of the top clicked on news items. So I guess before we get into reverse mutation testing, it may be a term that people aren't familiar with even mutation testing. So let's define what is Mutation testing. And then we'll dive into what is reverse mutation testing.

Yeah, yeah, that's the right way. Let's start with. Okay, so mutation testing, for me, the first time I heard that, that's a great idea. So let's say, let's simplify with an example. So you have your code and you have some, let's say, unit test, which is covering the code that you're building. The main problem is that. Or the challenge, I would say, is that how do we make sure that those unit tests are really effective. A lot of times people talk about coverage as a good metric to evaluate unit testing. But, you know, covering all the code is definitely a good starting point. But is that really enough? My personal opinion is no, because you could write just, you know, as a matter of proving that is not enough dumb test cases that just, you know, asserting that true equals true or absurdly absurd in the obvious. So test coverage, covering all your code through unit test is a great idea, but it's not enough. So there is a need for me to understand, are those test cases built in a way that if there is a bug due to some code modifications, they will be able to catch the bugs? So the need is measuring and verifying the quality of those test cases. Are they valid? They really test what is important.

That whatever that fails could break things and could fail deliver business value. So they could put the entire application under risk. So the mutation test comes in a very good, cool idea. So the idea would be, what if we have a programmatic, let's say, automated way to modify our target test code. Imagine there is a tool that takes an input the code that we are testing, that we're producing our application code and modifies it, mutates it, if we want to use the right terminology. What does that mean? It basically means modifying, for instance, the condition in an if then flow. So if the then flow is executed with a major sign, you just flip it and then you create a modified and mutated version of these code and then you execute all your test cases against this mutated version of your code. So why would you do something that's, you know, the first glancing may say a little bit, that's way. Why would I do something like that? Well, the idea is very smart, actually. If you run a test case against a modified version of your code, what would be the expectation? The expectation would be that your test is failing. Because by modifying the code and by having the test case still testing the original version, you would expect this mutation make the test case. Now this is Something that I would say applies in general, that we could always find cases where this doesn't apply. But in general, mutating the code and running the automated, the unit test against this code should make the test fail. So the mutation testing tools, all they do is they take your code, they apply some mutations, and mutating the code could be done in so many different ways. So also the challenge is finding the right way to mutate the code. And this is also a computational challenge, because if you imagine there could be billion of ways of mutating the code, you want to pick the right ones. Because at the end of the day, this mutation testing is something that you normally would add on top of your testing pipeline. And this is something that must have a time budget. If mutation testing will take 20 more hours, it's not worth it because nobody can wait extra 20 hours to get test results. But imagine this can be done in five minutes. And at the end of this automated activity, you have a report that would tell you, hey, hold on, because there are a couple of test cases that are not failing against this mutated code. So all you have to do is say, that sounds a little bit weird. So you just go and analyze those test cases manually and you try and understand why they, they didn't fail. Now there are two options. The option is in general that we can say there is a reason why they're not failing and then life is good. Then you have validated your test cases to be good. But the worst case scenario is, haha, you may have found a test case that it's not doing a right job. So basically it's asserting something that it's meaningless, there could be an error, there could be something wrong in a way that if you mutate the code and your test case is still passing. And honestly Joe, this is the worst situation ever. Because we all rely on our test cases. We all assume that whoever wrote those test cases, and they could have been written many, many years ago, they were reviewed, they're correct, and life is good. And what is even worse, technical debt is already big anomaly in companies. So do we really have the time to go and review the test cases? And do we really want to review test cases which are testing? And this goes a little bit into, you know, psychology. When I first, if I see green stuff on my screen, happy days, you know, I feel like popping the champagne and celebrating. So the little thing that says, hey, are you sure those test cases are right? Maybe they're passing, but they're not doing the right thing. I may Say, well it, it would be great to validate that. But it's effort, right? But if I had a tool that would do that automatically and would point out to me a couple of test cases of maybe hundreds that could be suspicious. Happy days. It's well spent time because if you find the wrong test cases, you fix them, then you have increased the quality of your test automation and you sleep much better in the night knowing that your test cases are really effective, they're delivering quality.

So there's a few things there. I guess everyone is always like, oh, we need a green build, we need a green build, we need it, you know, but you're right if it's always green.

And like how do you know it's really passing? I don't know if that makes sense. Like yeah, it's green, but is it really green? Is that the problem? And like and because. So I guess that's the first thing you want to build trust in your, your, your, your test. So when they fail, people pay attention, but you don't, you can't really trust them if you actually never sell them. Fail I guess is the main issue.

Okay, this is exactly, this is one of my actually mantra that is do not ever trust a test case that you never seen failing at some point in the past because it's too good. And you make a very good point. We want green pipelines, we want everything working fine. But at the end of the day those pipelines, those tests, automation, they are our guardians. They make sure that if there are problems, they find them and they fail. So in an ideal world, a pipeline fails upon problems. In reality you can have cases in which a pipeline fails because the test case is not maintained and that can affect the reliability and the fact that the test case teams are relying on those test cases because if they start failing not over a problem, but over maintenance issue, then over time teams start to, you know, ah, that's, that's, that's rare but you know, it's not really reliable. They don't even go and look if there is a real problem. So they become useless. And this is something that maintenance help to prevent the situation on the other end. And this is actually even.

Such a corner case that if it's always green, but we also want to rely that green means green, green means the test cases are passed and we normally don't go and check if that something showing green is really green. And that's actually the catch. And by not doing that, which I think it's the human nature, if the testing is Telling us everything is good. Happy days, let's just move on. But companies lose millions by relying on test cases that are not doing the right job. And mutation testing and reverse mutation testing are, you know, relatively quick wins that can help diagnosticate. Do a checkup on your test cases, find the wrong ones and fix them. And you know, it's much better if we find problems rather than the clients find problems due to leaky test cases, let's call them like that.

And it's pretty dangerous because you're giving someone a false sense of security if you can't honestly say yes, it's green, but is it really doing what we think it's doing? It sounds like yeah. All right, so is this a one and done activity? Like would you just write a suite of mutation tests and you run it once and say okay, I've seen a fail, so do I need to worry about that again? Or is this something you run in the pipeline ever so often? Like how does it work?

So it's a very good question. So in reality I would say that my approach, I mean would be it's great to integrate the mutation testing in a pipeline, but maybe it's a little bit over engineered. Unless you run mutation testing only on new test cases. Because once you validated your old test cases to fail upon mutations and therefore that is telling you they're working fine unless you modify those test cases. And then again you should ask yourself, are the new versions of those test cases working? This is something that you can delegate to new test cases. So it's great to have mutation testing in a pipeline given that you tune the mutation testing so that it doesn't take too long. So mutate the test cases in the right way. There are tools that do mutation testing offering you the right way and also make sure they take care to run mutation testing only on the new test cases because once you validated the old one, as long as you don't touch them anymore, you can rely on them. So that's. But I would say, you know, that could be the to be ideal situation if a company starts also to say hey, let's from time to time run mutation testing even out of the pipeline just to verify our test cases. Sort of a manual or semi automated process. I would say it's a great starting point because if you validate your test cases from time to time and you know they deliver quality, they're testing the right thing even if it's not in the pipeline, it's better than not doing that at all. I would say 100%.

So what's reverse mutation testing? Are these terms interchangeable?

Yeah. So basically, reverse mutation testing, it's something that. It's a sort of an evolutionary. I would say it's a complementary technique to mutation testing. You can apply both of them. The idea that I had comes again from this mantra. Do not ever trust a test case that you never seen failing in the past. So I think most of QA people, when you're in a stage that you're writing automation unit, but it could be any type of testing. As you're designing, you're implementing those test cases. You want to make sure that they're working fine, so you run them and they pass. So life is good. But how do you really know that they're working fine?

The easiest way possible that I may think of is just programmatically make them fail. So a very simple example, you're testing that a module that is summing two numbers. If it takes two and five as an input, you're writing a very simple test case that the output should be 7, and this test case should pass as long as the code is really working. Now, to be 100% sure that this test case is working, what I can do is really simple. Let me mutate the output from 7 to 3 or say anything that would be wrong. And now if I run a test case that is checking that two plus five equals three, I'm expecting the test to fail. If this happens, Happy days. It's probably showing me that, you know, the test case is well designed. The challenge is the problem is if the test case doesn't fail, because I will say, hey, hold on, this test case keeps on passing, even if I manipulate it temporarily by breaking it. And why is it passing now? Obviously, on a very simple example like this one, it's not going to happen. But there are complicated test cases where you're manipulating objects, you're checking API responses, where the chances of writing a test case that it looks like it's working, but actually as a mistake, at some point it's quite consistent. And a great way is just modify the test, break it, run it, and expect the test to fail. But this is an activity that it's relatively fast. But now scale it up to all your test cases. It's great when it becomes part of your methodology as writing test cases. But again, what happens if you. In every test cases that were built in the past by some other people, how do you trust them? Therefore, I thought, I can put all those pieces together. And I thought, hey, why? If instead of mutating the target code that we're testing. And this is the mutation testing. Why don't we mutate the test case? Exactly. Automating what I would do as a tester manually. So I'm breaking my test case in a purpose way. So really knowing what I'm doing, but instead of doing myself, I let this reverse mutation testing tool doing that. And again, when I run my test automation, the mutated version against the not mutated code, I would expect the test case to fail because I broke the test case by mutating them, and now I'm expecting them to fail. So having a tool that does all of these automatically and comes to you with a report, finding the little nasty test cases that are still passing upon the mutation, help you identify problematic test cases. And we end up in the same scenario. As with the classic mutation testing, There may be reasons why they don't fail. And then life is good. Happy days. There is no action item to do, but you may identify test cases that oops. Those test cases are. By analyzing them, you realize there was an error. You fix the error, you run again your reverse mutation testing. And this time they fail upon the mutation. And then you know that all your test cases are reliable. Now having a tool that delivers that in minutes and plus your extra effort just to modify. Imagine hundreds of test cases, a couple of them are not failing on the mutation of the testing. You fix them and now you run the reverse mutation again. Now they're all passing, and then you know that your test cases are reliable. So that's basically the idea of reverse mutation test, just flipping the angle. Don't mutate the target code, do mutate the test cases.

All right, so this might be a real dumb question. Is it always the test case that's broken or does it actually find real issues with the code? Does that make sense? This is just an activity to test your test code, not actually to test the code that you're testing. I don't know.

Yeah, it's a very good question. Actually. The thing is that when you do reverse mutation testing while also mutation testing, the ideal state is that you're starting in a situation where everything is working. So your test cases are passing. So let's go into the reverse mutation testing. If you start in a situation that your test cases are passing, mutating them, they should not pass. So ideally it means your code is correct.

The scenario of those test cases instead failing because the the new version of the code changed and this broke the test cases. This could always happen. But I Think there are two orthogonal things here? I would say the focus is more to validate the quality of your test cases in the meaning of catching, in an automated way, faulty test cases because a faulty test case delivers nothing. And of course, by running them, you can always fall in the situation that they spot, they identify a real problem. And happy days, because at the end of the day, this is what test cases should do. But I see them as two orthogonal things.

Gotcha. So how's this different than like tdd? But you're using TDD like with bdd. I used to do TDD with the BDD before I implemented each. You know, given when I make sure to fail before I implemented the next next team. Is this the same or is that different?

Well, TDD is, as you say, test driven development is a methodology where, you know, start writing a test case before the code implementation so the test fails and then all you have to do is deliver the minimum code that make that test case pass. So this is tdd. Reverse mutation testing is something that you can put on top of TDD in the way in the meaning of, okay, now I've written my test case through the TDD methodology. It's failing. Now I wrote the code that make the test case pass, but am I sure that the test case that I've wrote since I wrote the test case manually, am I sure that this test case is really correct and there is no problem? Hey, let me apply the reverse mutation testing. Let me break the test case and make sure that upon breaking the test case, it will not pass. When this happens, I know that my test is likely correct. That doesn't happen. A I may have written accidentally a test case that always passes so delivers no value. So the short answer is that reverse mutation testing can be done on top of TDD and they're complementary and both deliver value. TDD has a methodology and reverse mutation testing as a way to validate your TDD generated test cases, I would say.

All right, so you mentioned tooling a few times and I forgot to follow up. Is there a tool for this? What is the tool for this? I thought this was being more of a manual, just you fat fingering things to try to make them fail. What's the tool you use for this?

So I clearly did a little bit of search about reverse mutation testing and tooling and I found nothing. Basically the best practice of breaking your test cases in the process of developing that. So I actually decided to run Ryle myself, what I would call a POC or proof of concept of a Reverse mutation testing tool.

Just you know, to, to create something that can be easily used. But obviously the idea would be let's, you know, create something which is an MVP or really like showing the concept. And. And then I created one is you can find on my GitHub is freely downloadable, you can check it out. Actually I had already a pull request by a nice person I think from India that already found out some improvements and I'm very happy because you know, that's the. Eventually that's the goal of having GitHub projects make them available to the community and the dream is having the community contribute to them. So I created it. It's quite basic because basically what it does in this example, it takes existing test cases, it creates a version, he writes it into the file system and the mutated version. So it creates this mutated version and he run the mutated test cases against some toy code that I've created and he creates a report identifying the test cases that don't fail over the mutations. So the suspicious ones, of course, real mutation testing tools, they don't create physical copies.

Of the code in this case, but they would rather use everything in line in memory. So there is a gap to implement a much better version of reverse mutation testing where the mutations of the test cases can be created at runtime in memory and run them. But you know, the idea is that let's build some foundation and then if the community sees some interest, we can, you know, build a better vision and more optimize and start piloting in some projects, I would say.

Nice. So I know you're a big proponent of lean QA and shift left practices. Does this slow things down? Like how do you know what you should everything be mutated testing, used for mutated testing, reverse mutation testing, and how much data do you need to validate whether or not it actually is passing that at that point?

Yeah, it's a very good question because at the end of the day, is it worth doing mutation testing and reverse mutation testing? My answer is yes, as long as you have a timed budget, as long as this doesn't affect the pipeline execution. If I need 20 hours to mutate my code or my test cases, it's clearly not worth it. But again, if we find, if we create a tool which is optimized and very fast into mutating the test cases and we do a proper mutation.

You can mutate the test cases in different ways, you can change the assertion, you can change the result.

The condition, and so on and so forth. So by implementing and this clearly has a room to discover how this can be done. But by finding the right way of mutating test cases and implementing a tool which is optimized, we can make sure that the time budget for running reverse mutation testing is under control. So personally, if somebody tells me, hey, would you pay five more minutes in your pipeline? But the return on investment is you identify potentially broken test cases the next time you fix them and life is good, I would say definitely, because if this prevents the situation of a green pipeline, life is good, but the test is broken. This prevents the possibility of a bug to escape and to go to production and eventually being formed by clients. And you know, we all know that through bugs, companies lose reputation. The damage can potentially make the company lose million of dollars of euros. So my answer, if we can time budget, time limit the execution of mutation testing and reverse mutation testing, I see a lot of return on investment.

All right, I'm surprised we went this far without talking about AI, but obviously everyone is talking about AI. Since you've been around for a bit, has AI really changed QA and testing? And where do you see the role of testers going in the future? Because a lot of people are saying it's going to diminish or some saying it's going to grow. You know, I hear all these companies, founders of AI saying we're all doomed. Where do you stand?

So I'd say a couple of points. So I think embracing AI is great as long as, you know, you don't deliver everything to AI. But you know.

Ask AI to, you know, you can divide some tasks in ASAI to implement some automation for you. As long as you understand it, you review it, you own it, you even ask AI to explain.

How it implement a certain solution when you can clearly see that's something I would not be able to do that, but definitely embrace it because it increases your productivity. I actually think, you know, yeah, a lot of companies are probably thinking that everything can be tested through AI, but at the end of the day, I think you still need somebody to validate it. So I see a shift in the role. You can delegate to AI a lot of work, but you have to be in control that it doesn't hallucinate and it delivers a great job. The other side of the medal is like, I see still a challenge on how to test AI based software. For instance, how to test non deterministic systems. How am I testing? Like a chatbot unit testing? Like when I ask a question, the answer should match my expected one. Clearly it doesn't work when we move from, you know, from determinism into the world of semantics where, you know, a question can be answered in many different ways with the same semantics. So that's actually where I still see a challenge of human beings needing to find a way to automate non deterministic system. So I still think that real humans can and must deliver a lot of work. In the age of AI, it's just as every big revolution, there is just a shift of the world but that we definitely need to continue working on that. So that's my take.

I forgot about one last question. A little plug for your company QA Roots. What do you all do if someone's listening, who's a good fit, maybe that they can contact you to maybe get some assistance from you?

Yeah, absolutely. So I created QA Roots with the metaphor of establishing quality assurance and engineering as the rules, the foundation of your company to deliver an amazing product. The three so the philosophy behind the company is really to help companies deliver great quality through.

Best practices and quality engineering. So I'm super happy to discuss with any company in the need to create quality engineering or even to fine tune their existing practices. So I'm absolutely happy to support them through my knowledge and feel free to reach out to me and you know, sometimes quick fixes, quick wins, can deliver incredible quality. Doesn't need to be necessarily things that last for for a long time. But even just starting in a conversation, I'm happy to help deliver to help companies deliver better, even better products.

Absolutely. All right, Leonardo, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts and what's the best way to find or contact you?

So definitely I think my suggestion for testes is those were the days where you can do black box testing and remain at the surface. Try to shift left. Also in the testing, try to be adventurous. Understand the code that you're testing even if you end up doing just end to end tests. Talk with the developers, establish a good relationship with them. Analyze also the way the unit tests are written, considering normally they're done by developers. Be involved into the code reviews of pull requests, merge requests and analyze the way they're testing. Provide your suggestions. Don't be afraid of the code. You can even create a little project on yourself where you start writing some code and understand also the challenges from the development standpoint. But really be curious and get as close as possible to the core. This is probably the best way to become an excellent tester to answer your other question. The best way probably to reach out to me is either through LinkedIn, I'm very happy when people contact me and I try to reply to whoever writes to me, or you can write to infua-roots.com and clearly I'm absolutely happy to reply and I monitor the email every day so those are the best ways to reach out to me.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode head on over to testguild.com a548 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 4 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences. Together, let's push the boundaries of what we can achieve.

With hoots and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing us secrets to behold.