The Value of Using Native Test Frameworks with Jaswanth Manigundan

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast and today we'll be speaking with Jaswant Metagondhan all about the value of using native test frameworks. Jazz is currently a lead quality coach at Culture Amp and specializes in test automation and testability. He's worked in the mobile app development space for almost a decade where he's seen mobile devices and operating systems get better and better by the day. His fascination with mobile devices has made him start his professional career as a mobile app developer, but interestingly, his desire to build better apps and write code that is much more testable led him to transition into test automation engineering. So he specialized in mobile testing. He has a ton of experience with a wide variety of test automation frameworks and has consulted for some of the leading mobile app teams in Australia to build effective test automation suites for mobile apps. So if you have anything to do with test automation or mobile app testing, this is the right place to be because Jazz is about to drop some knowledge on. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testskill.comSourceLabs and click on the exclusive Sponsor section to try it for free today. Check it out.

Hey Joss, welcome to the show.

Thanks Lerjo, how are you doing?

Great to have you. Really excited. Before we jump into it though, I always tend to botch people's bios, so is there anything I missed in your bio that you want the Guild to know more about?

No, nothing much. I'm currently working with an awesome team at Culture Amp led by the legendary Anne Marie Sharrett. So yeah, I've been working there for almost a year now. So yeah.

What's Culture Ramp?

As a consultant now, Culture Ramp is a product company. It's a people experience company. So it's all about your end to end people experience. Like starting from one on ones to feedback to performance management and yeah, more around like trying to develop employees in the company. So it's been there for almost 10 years now and the company has grown a lot and for the past two years we've scaled quite a bit. And yeah, it's very interesting to see the journey Especially currently we are going through a phase from just hands on testing into more quality coaching and quality enablement. So really enjoying the ride right now.

Awesome. So speaking of a journey, as I mentioned, you started your career as a developer, mobile app developer. So what made you get addicted to testing or wanting to make things more testable? Was it the struggle to release software and realize there were a lot of bugs in it or just like what would push you to make, to get into testing in general?

General, yeah, great question. So I actually came to Australia to do my master's in software engineering. So I've been preparing for, to become a software engineer for quite a while, since my early age as well. So. But when I did my first app in a professional environment and I pushed my first production release, you wouldn't believe it. It had a bug in the signup page. So the sign up, the signup page had a, a bug where you can't enter in if you enter the right details. So there was an exclamation mark that I added. It changed the whole dynamics of the whole page. So that was a very, very, very good learning experience. And since then I had to, I was searching for like, how do you write better code? How do you stop these things from happening? How do you have like checks happening as you develop something and not just at the end? So that's been my, my major turning point in my career. So I was still developing for almost five years since that point. But I worked in a consultancy and I had a really great test lead in the company I worked. Kelly was, worked with her, she pretty much taught quite a lot of things to me about what testing is. She's been in the industry for almost 10, 15 years. So I quite learned a lot. But at that point test automation was kind of booming as an industry at the same time as the tools as well. Like the tools like Espresso, Appium X UI test, everything was just booming at that point. So I kind of dug into it and then I couldn't keep it as a side project. I had to like dive into deeper and learn quite a bit about it. So that's where all the fascination started about oh okay, it's, there's something that you can utilize to stop pushing buggy code into production, especially like the critical one. So yeah, that's, that's the journey and the turning point in my career.

Cool. So when you were a developer and you learned more about testing and automation, what did you start with? Did you start with unit testing or did you go directly to you Know just like a front end type of testing solution.

Yeah. So it was quite hard for me to understand testing at that point. I was, I was able to do very simple testing in the unit testing space about how would you test the function, what type of values would you provide, boundary value testing and all that stuff. So that was all part of my unicourse as well. So it was quite okay for to understand. But the problem was when it came to mobile app development and how the whole framework worked in terms of spinning up a screen or how would you test the screen as a unit. So those things were still confusing for me at that point because you need to write really good code to actually achieve that, not even just able to test it locally. And in that unit testing space, your code has to be written pretty well, structured pretty well. So I think that's the journey. So you write better code to eventually lead you into writing good tests. So that's the thing I was confused about. But that's the starting point as well.

Right. So I didn't think I was going to dive into this as much. But I just curious to know, because you have a development background, if there's any developers listening. You said you had to write tests better or you had to write code better in order to make it testable. I guess so. What does that mean? Do you have any tips or things that you think people don't do to make it testable at the coding stage that they could to make it testable later on down the line?

Yeah, there's a lot of principles and patterns that's been coming up, especially in the mobile app space since I started that allows you to do this kind of testing. But to give you an example, when you are spinning up a screen, you kind of like put every business logic into that and that stops you from testing everything together because everything is in one place, you can't modularize it. So everything comes from trying to keep things separate and simple. So there's different patterns like open, close, principle and trying to modularize how to write code and stuff. So this, there's a few things that you can follow, but the very simplest thing would be to try to modularize as much as you can and keep things separate so that way you don't have to spin up the whole app to test just one workflow. You can test it in small bits and pieces and you can kind of combine everything together to achieve that coverage of testing.

Nice. So are you able to use those same principles used for developing mobile apps to actually developing mobile testing frameworks that test mobile apps.

Yeah, there's still a few patterns. It's not exactly similar but you still have, you can follow patterns. I think this is pretty common. The page object models or the newly not new but screenwriter pattern seems to be quite popular right now. So I think whichever works for you, if you choose one and kind of stick with it, I think that should be great. But there was a different dynamics when it comes to test automation and things specifically to automation because I realized that you don't need to over engineer your tests. It has to be as simple and as dumb as it could possibly be because it should allow anyone who's jumping into the code to understand what your code does as in your actual feature code does at the same time contribute to writing more tests as well. So in order to achieve that you can't have layers of abstraction which takes people like days to figure out what a small test actually does. So I feel you can, you need to keep it as testable, as simple as possible. So it's what I think changing point in terms of like the patterns.

So that's interesting. It's almost like if you over engineer it, it makes it almost unreadable which then makes not good. So it's fine line for what you know how to over engineer compared to make it readable.

Yeah, I kind of do some quality coaching with engineers right now and that's the problem I always counter. So when I present them with the very simple automation framework, they tend to go and add more complexity to make fancy changes to it. But I go and say like that's how it's devised to do things and not done there because it was hard to. But you need to convince them to say hey, tests are meant to be simple because like the overhead it presents. Yeah, even with code itself I think it's quite similar in coding as well because I've been reading like the clean code books since the beginning of times and what they mentioned was try to keep your abstraction levels to just one layer or maximum two layers possible. So I think that's how you also try to keep code. But sometimes it just gets over complicated and that's when tech devices and hard to get onboarding people, new people into it. So all everything comes because of the fundamentals of how you actually define and modularize your code. I guess.

So does this philosophy influence you then on which mobile frameworks you use? Because it seems like Appian maybe has more abstraction because it needs to handle multiple devices where a native solution I think is closer to the code. I Don't know. So, like, is that what made you. That's the whole, that this whole podcast was based on was native mobile frameworks. So what made you get into mobile frameworks as opposed to like something like an Appium solution?

I think everything, every framework has a place because everything depends on the team. You are in the skill set that the team has, the skillset that the tester has. But the whole reason that I try to recommend native frameworks is because of the ecosystem. You kind of work hand in hand with the developers, you reuse some of the aspects they've done for unit testing, the more faster the things that you can try to get into because the performance aspect is quite better with the native testing tools. But that said, I've done Appium testing as well, so everything was just like the situation you are kind of in. So I would recommend people to choose the right one for that situation rather than, I'm not going to say this is the best framework that would work for everything, rather I would say there are criteria in each and every framework. Just make sure you pick the right one which suits you and your team, so you're set up for success.

So what are some of those criteria then? Say someone's working with the developer, they're doing mobile app development. Let's start maybe with when Appium may be a good choice and then maybe when native mobile apps. And like you said, not one's better than the other, but in that particular context one may be more suited.

Yeah. So I would categorize it into three buckets. So let's say Appium is one of the frameworks and I would say the second bucket would be espresso and X UI. So Appium can do both Android and iOS, so you need to find equivalence for that in native. So that means that you need to get into Espresso for Android or XUI Test. And the third bucket that's recently been coming up is if you have React native apps, there's a framework called Detox, so it's built just specifically for React native. So based on these three buckets, you can kind of choose which one suits you. And for example, let's start with Appium. As you said, Appium would be a great choice when you just entered automation and you have a Selenium background. So you're coming from a web automation space, you're jumping into mobile because your team has started doing mobile app development and that's the easiest entry point and also gives you like the end to end coverage of everything you need from CI CD space to understanding how automation works and everything with mobile. So that's the best suited for starters and people who have a web background. And it also makes if you are just comfortable with a specific language already, which is, let's say Python for example, you can't use it with the native frameworks. It has to be either Java Kotlin for Android or Swift for iOS. So if you are comfortable with other languages in that case as well, Appium is much more suitable for you at that point.

Awesome. So that's when Appium is a pro. What would be a con then?

Maybe using Appium, the one that I actually personally face, I can't talk about in general, but the ones that I've faced in the journey that I've been through is mostly the performance aspects. Because when you want to run a test, the time it takes to actually set up your whole test and get to a point where you can test it seems to be very complex when it comes to Appium. Whereas I found the native test to be a little bit more performant and easy to configure, especially like if you're new to it and if you're trying to start and if you get stuck with something it might be harder to debug because Appium is quite large and as you mentioned because of the abstraction, sometimes it might get harder to get started. But yeah, I think the performance aspect is one of the pros when it comes to choosing your native automation frameworks.

So is Appium tends to be slower or is that just maybe how someone may write their mobile tests that makes it slower?

So to understand a little bit, Appium actually internally calls the native framework. So that's how it does automation. So you just bypass that by straight away digging into your native frameworks. So that's pretty much what the concept is. And the other thing that the biggest thing that I've actually made me the two jump into native awesome automation frameworks is how you can set up your tests. So for example, you have more control over your test environment when it comes to native automation frameworks. Say if I want to pass custom test data for one page, so I want to say fire up this page, pass this test data and do my tests. So that kind of things you can do with native frameworks because the whole environment is set up for that. Whereas Appium is more black box, so you can't dig deeper into the apps, you just need to look at it from a black box and find other ways to do your mocking. Whereas these mocking can happen in A native way within your mobile app, your native mobile app frameworks.

Thanks. So X UI tests not versus Expresso, but I guess X UI test is for Apple and Espresso is for Google Type oss or Android for Android.

Yeah, that's right. So, yeah, in terms of languages, Android obviously supports Kotlin and java, whereas with iOS right now it's Swift. So that's the learning curve when it comes to native automation frameworks is they're more peculiar, they are mobile based. Yeah. It's not like Python or JavaScript, which is more widespread.

So what happens if you need to test, if you want to use native mobile frameworks and you want to test in an Apple app and then you need to test an Android app, do you then have two different frameworks or is there any tips you have to get around that that Appium was usually used for.

That's right. I think that's the biggest selling point of Appium is that when you're starting, you don't have to spend too much time in maintaining two different test suits. But that said, it's not 100% just like out of the box that works. You still need to tweak quite a bit. So maybe 80% or 70% of the code can work in both iOS and Android. You still need to tweak 30% of the code in my experience to actually get it to work separately for Android and iOS because the layouts are different, there needs to be some custom things that should happen to get it working. So in order to counter that in the native frameworks, the way I've managed it is to I have one layer of abstraction which is the tests would read exactly the same when it's for Android, iOS doesn't matter. Say, for example, all the functions that I write, say when I click this button is what my function name is. So I try to keep the functions pretty straightforward and readable in pretty much plain English. And those functions would go and do different native stuff within it. So my abstraction is how I try to manage the way that the tests work. So if you combine it with page object model, which is a quite known pattern, so you still can get away with writing similar tests, but you have different page objects for Android and for iOS and you try to achieve what Appium is trying to achieve. But with native frameworks, still a learning curve to get started with and have your initial setup. But once you get started with one app, you can pretty much follow on with other apps. Or if you, yeah, you can keep digging better if you're Just working in one app, then you can keep improving on what you've done till now.

Nice. So are there any gutches for either one of these? So X UI test, is there anything people maybe do wrong that make it more difficult than I guess the same for Espresso. Is there any things that you found if you do this, it makes it easier to create Espresso Mobile test?

Yeah, I think the common, I think it's across any mobile app platform is the emulator setup or the environment. Because most of the tests become flaky because of the way that you set up your emulators. So if you need to make sure, especially if things run on CI cd, you make the best effort to build out your pipeline in a way that these emulators are set up with good devices and good CPU and et cetera. So that way that you don't have to be finicky with the environment, your tests, if your tests fail, you know that it's because of something strong with the app and not with your setup with your pipeline. So that's one of the gotchas when you work. Yeah, when you work especially for Android, I would say, because iOS simulators are really well constructed. It's just that for some reason Android still hasn't caught up that well when it comes to emulator setup. So, yeah, get that right. Make sure this memory and everything makes sense for that configuration that you have and that should work well.

Cool. How about Detox? Like you said you would only use it for React mobile testing, is that correct?

That's right. So only for React native apps, you would use Detox. And again, it sits really well alongside your React native apps. It sits within the same package. It's just another library that you add to your React native code. And you can test for both Android and iOS because it's the same code essentially. So that is still not as mature as the other ones. But if you want to have a smoke test suit that just you just want to run to make sure your core workflows are working fine, you can definitely prefer using Detox.

Nice. So when I was in software development, I haven't been in the field for a while, is that React was just coming up and it was like the hot new framework, but now I've been hearing more and more about Flutter. So what do you use then to test Flutter? Have you ever done anything with Flutter? Is that something you've been seeing?

Yeah, I haven't had much experience with Flutter and even within the top companies that I've worked here Flutter doesn't seem to be the choice, right? Not yet. I feel React Native is the go to one. If people are very developing something like more pox or more small scale apps, people tend to follow React. It's just my experience. I didn't get any chance to work with Flutter. But to be good to. Yeah. See what's out there. I haven't had any insights on what test frameworks were working for that as well.

So you did mention simulators and emulators as something to be aware of. So. And you talked about Pipeline. So I guess along the pipeline when does it make sense to use simulators and emulators? It sounds like when you're doing development and then when you put it in your cicd does it go off to like a. An environment where you have real devices? How does that normally work for you?

Yeah, that's. That would be my recommendation. So there's a few vendors right now who are available. I think this source lab has one offering. I know that Firebase is one of the offering. I know AWS has its own offering. So I would really recommend to go and figure out the right one which works for you in terms of both cost and efficiency. In terms of like less finickiness. So there's also one called bitbar. There's a few vendors.

Oh yeah, a Smart bureau and spit bar.

That's right. So there's a few vendors. But I would recommend people to choose the right one to to make sure they run their tests in these ones if they want to get rid of the emulator setup. Because trying to set up emulator on your own in the CI CD pipeline is the most hardest challenge I face. And especially if you're trying to do with Appium, I think I would just say forget it because the way Appium works is that you would need to actually bundle your Appium code and then try to get it run with your app on top of a thread. So those things were what actually made me push into more. Yeah, trying to find different solutions like using a real environment to test your code and stuff. So the emulator one with CICD pipelines is something that I would really recommend to look into and see if there are any offerings that would suit your budget to make sure you want to run these ones.

You know, when we think about mobile testing, it always seems to be functional testing. I know with like Selenium and things like that, that's what it was built for. But you could do other types of testing with other tools like performance testing and things. So with These native mobile apps, are they just for functional automation or do you also recommend after you get your functional automation you also focus in on maybe especially with mobile devices like visual testing. Is that something that you focus in on or anything like that?

That's right. So one of the projects that I work required for some visual testing. So there's a couple of options there. You could go applitools, they've got really good support right now. But the same time there's also open source stuff as well. There's native libraries available in both iOS and Android that allows you to take screenshots and manage your own visual testing suits. On top of that, one thing I would really recommend is to test for accessibility as well. Because that you get out of the box with not out of the box. There are a couple of libraries for Android. It's quite native. There's a native library that would allow you to do it for iOS. There's a third party open source library that that's available. So all you need to do is to try to encrypt incorporate this as part of your UI test. So every page it opens, it makes sure that it's checking for some basic accessibility components and giving you that feedback every time you run it. And you can fail the test if it pass, if it fails the activity threshold and you can set up to things like that. But it's a really good tool which allow you to see, at least see what's wrong with your code. So if you have UI test which is already running, it's, it's a bonus for you to be honest. It's a single line of code in some of the libraries. So I would really highly recommend to people to start looking into it if they see have this kind of an experience.

And you keep always talking about testability, which I think is really important. So when someone's developing software in order to make it testable and especially to make it so it's testable for accessibility, like you said, it's just one line of code. Is there anything else they need to do in order to make it so that is easier to test accessibility on it?

So the one line solution I mentioned was for just basic level automation, more visual aspects of it. So that's still really recommended. If you don't have any accessibility testing, I would really recommend to look into that. But when you develop for mobile apps and when you want to do activity testing, the thing that actually worked for us in the past is to push it to the left. So as soon as you write your stories. You go through a list of all the items that are there and you make sure what's more important for you in an accessibility level and start developing it at that point. Because if you want to go back and write for accessibility, it's, it's total. Sometimes it's going to be total rework because of the nature of how accessibility works. Because you need to make sure these components sit within the whole like a scroll view, say for example, that's one of the hardest ones. You need to make sure that all your components within the scroll view are properly aligned and doesn't overlap on each other when you scroll it. You need to be able to actually move the screen up when, when you have activity turned on. Some people totally miss this aspect and if you have a list, view similar aspects as well. So these things can't be tweaked after you've finished your whole code. Because that's what usually companies do is that let's just push this MVP out and we'll come, come back to accessibility later. If you take that approach with, it might be easier for web, but if for iOS and Android it's, it's not that that straightforward. Yeah, I've dealt with a lot of apps and every app when someone says that we'll come to accessibility at the last, it's yeah, it's going to cost you more when you do it at the end.

Is there anything you can automate on a mobile app? I know sometimes people say they'll automate like a gesture, but then when someone actually physically does it, they find a bug. Is that common or is there anything else like that that you've seen?

I think in the early days it used to be quite a bit of a problem because the emulators that you test with, especially with in any type of frameworks, espresso or xei, it seems to not provide the exact same like gesture replicated. But yeah, I think recently they've become much better and they've been pushing really good updates over the last few years with both Espresso and XUI tests. So it's become really close to what a user, real user interaction would be.

Cool. So I don't know where I saw this hermetic test pattern. I know you've been around for a while, so you've been sharing with the community for a while. So thank you for that. I think one article you wrote was on their hermetic test pattern. So really quickly, what is that? Is that something you created or is that an existing pattern?

So when I was working on the automations on mobile automation and I was trying to do research on how other people are doing it. And one really interesting aspect I found with what Slack does, even though they're not native mobile app frameworks, they use, I think electron, but I really like the pattern in which they've done it because they devised something called hermetic test pattern, which is the way you test your app is much more contained within the mobile app. Say for example, I was talking about this earlier as well. So the way you can do tests is that you open up your screen, you don't even need to hit the app API or anything. Or all you need to do is just to mock the data that you want and push it into the test. So as soon as you open that screen, you would see this configured data that you want and you test for that particular flow. So the beauty about these tests is that for every test you write, it can have custom data. You don't need to tweak your APIs, you don't need to tweak your mocks. It's just like whatever data you pass in, it comes up in the app to in the specific way that you want. And you can pretty much test your scenarios from there. So this way you can pretty much navigate every nook and corner of your screen. Because some screens come up only when there is an error scenario, Some screens only come up in this particular data set. So you can test for all these with very less pain because all you need to do is just to give a different JSON file to it. So every time you give it that kind of a JSON file, so it would present you with different scenarios. So a few people have done this with debug menus, similar concept. So you have a debug menu in your app and for every button you click and you configuration you have, you do this. But this is within the test itself. So you can pretty much configure everything within your test. It doesn't have to be only a set of data sets, you can configure it from top to bottom. So say things that I've done in the past is you can skip login. Sometimes that can be like an overhead because every time you need to go through the login flow. So that's one simple one. But the other ones I've done is the list views. Say for example, if it's with zero elements, with if there's just one element, 100 elements. So how you can test your scroll, how your scroll works, how your pagination works and everything. So these kind of things you can pretty easily test if you configure it with hermetic testing in mind. So hermetic means just like trying to do it in a contained manner. And just an additional point to that is that the testability that it improves, like it really improves the way you write code because everything has to be modularized. As I was mentioning in the early times, that's the biggest pro of actually having something like this. And that's why I really like testing, especially when it comes to automation, because it drives you to write better code and more cleaner code.

Love it. So if folks really want to follow along, I'll have a link to that in the show notes. So it's a detailed article I think has code samples as well. Right, Josh, before we go, is there one piece of actual advice you can give to someone to help them their mobile automation testing efforts and what's the best way to find or contact you?

They can find me on LinkedIn. Yeah, you can search in my name, Jashwan Manigandan in there. But I would like to leave with a very small advice to everyone in coming into the automation journey is that it's a journey, you can't pretty much reach the end state or the ideal state just by jumping into it. Sometimes it's a journey you need to go through. So in my journey I went through Appium, I learned the pros and cons about it and I found out that something else was actually working for me, which is native testing. And then at one point you would actually pass the UI testing and you would just go and dig deeper into more component level or controller level testing and you'll be comfortable with it without having to worry about UI testing. So this everyone has a journey. I would recommend everyone to go through that and find the right balance that works for your team, whether it is fully or UI testing. Or you can skip UI testing and do more controller level testing, whatever suits for you. It's the journey that you need to go through and your team needs to go through to arrive at that point. So I would recommend people to research about it and get to that quicker. Yeah, so that's. That's the last bit of advice I would like to give to everyone who listening.

Thanks again for your automation awesomeness. The links, everything of value we covered in this episode, hand on over to test guild.com forward/a411. And while you're there, make sure to click on the try for free today link under the exclusive sponsor section to learn all about Sauce Labs, awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to test guild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.