Appium React Native Testing with Wim Selles

Five, four, three, two, one.

Drop that.

Welcome to the Test Guild Automation Podcast. We're all getting together to learn more about software testing and automation with your host, Joe Calantonio.

Hey, it's Joe.

And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking all about Appium and visual testing with WIM cells. Wim is a solution architect for Source Labs based in the Netherlands. He has a ton of knowledge in the automation space and has assisted hundreds of SAUCE customers with solving automation challenges in their organizations. Wim is also known for his extensive experience using Appium for automating hybrid and React native apps. So listen up and discover Appium hacks for your mobile automation efforts. You don't want to miss this episode.

Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure.

Your favorite mobile apps and websites work.

Flawlessly on every browser, operating system and device.

Get a free trial.

Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days.

Check it out.

Hey Wim, welcome to the show.

Hi Joe. Well, thanks for having me also on your show.

It's great to have you. Before we get into wimp, can you just tell us a little bit more about yourself?

Yeah, well, I'm Wim, I'm based in the Netherlands. I've been in industry for, what is it, 12 years now or something. Been a consultant. Did a lot of different assignments with web and one of my latest assignments was with mobile apps, React Native and it was a really awesome one. Also give a few talks about the assignment and all the things I did there. And I really love mobile automation. I love JavaScript and I also try to help a lot of people in the open source community, especially protector, webdriver, IO and yeah, because I love JavaScript and open source and help testers, I also created some modules to help people do proper automation. Creative boilerplates. Yeah, that's basically what I do and who I am.

That's awesome. So believe it or not, one of the questions I get asked a lot is what do I use to test React native apps? Is there a specific technology used to automate those or would you say what would you normally use to test a React app?

Well, it depends on the team. I would never say you should use tool X, Y or Z. I would always say, okay, just do an investigation, for example, all the tools that are available. And I also did an investigation when I started at TELO2 for react native. I also gave a talk about it during Appium Conference 2018 in London. And there I investigated what was it, I think six or seven tools. And in the end I came up with Appium because we had specific requirements that we also needed to test on real devices. And yeah, basically only AppKim was the one that could provide us a cross platform solution, but also a solution that was able to test on real devices. So in the end I think if people would do their investigation, they would even test it with Appium and hopefully they would also use Appium. But yeah, as I said, it's up to the people, it's up to the teams what I want to use.

Very cool. And I guess along the same lines, like you said, I agree this is. I don't believe in one tool or one technology that rules them all. It depends on your team. But you deal with this more often than I do. I also get asked a lot, what do I use for my Android mobile apps or my Apple apps? And a lot of times they say they don't want to use something like Appium, they want to use the native tools. Are there any pros and cons to that approach or would you recommend that they use one Appium more often than the others? What would you say for that?

It depends. In my opinion, it depends on.

What they want to prove with their tests and the amount of risks they want to take. And I'm not saying that if you would choose for example Espresso that you're taking a lot of risk, but if you're doing your automation, hopefully you will also do that based on risk. Some tests you do automate and sometimes you don't automate. And I think if you would look for example at the native frameworks, you might have some limitations. In a real engine end to end flow.

You can really do kind of like proper gray box testing where you have access to the source code. You can, for example, for Android you can say I want to verify that a specific intent is loaded. For example, if you click on a link then your Gmail should be opened. You can easily test that with Espresso. So that's a very good use case to use Espresso, for example. But if you really need to do an end to end flow with more.

Applications connected, or if you need to automate webviews, then I think Appian would be the best pick. But again, it depends on the risks you want to take, the business value you want to get out of your tests and use that to select your framework.

Very cool. So I think at the heart of a lot of these questions is, and you told me this was one of the things you've encountered earlier, was that a lot of people don't understand Appium. So when I mentioned, hey, did you try Appium? They come up, say, oh, it's slow, or they come up with all these weird reasons why they want to use Appium. So I guess, what did you mean by when you told me not a lot of people understand how Appium works? What are some misconceptions you think folks have right off the bat when they hear, hey, try Appium for that?

Well, I think first of all, the infrastructure of Appium itself, it's quite complex, I must say. I've been working now with Appium for four or five years, and especially the last few years, I've been diving more into some details of Appium, and then you would see that there's not a lot of documentation out there about Appium. There is documentation about how to set up your test cases. But, for example, it's very hard to find the differences between all the drivers that Appium is using. And one of the things that I see a lot, and especially when you need to automate a native application, is that there's a lot of differences between the UiAutomator 2 driver for Android and the XUI test driver for iOS. And I think if you would kind of like compare them with each other, then there's only one thing. And I think that a lot of people don't know it, but there's only one thing they have in common, is that they both work based on views, so they don't work based on xml. And I don't know if you ever heard about it, but basically there's no XML when you're using Appium and you want to retrieve or you want to look for, for example, elements in your application. Have you, have you ever encountered that part or have you ever looked into that part, Joe or.

No, I haven't. That's the first I've heard it, actually.

Okay, well, I think this is, this is one of the things that a lot of people know, and basically what happens if you're using Appium is that Appium kind of like translates all the views into an XML tree, and that XML tree is built in two occasions. The first occasion is when you ask for the page source. So, for example, if you're using Appium Desktop, then it will Translate all the UI views from Android or from iOS into an XML tree, if you then want to interact with them and especially if you're using, for example, your XPath selector is one of the most used selectors with appium. Then the XPath needs to query the XML tree, and then the matched element needs to be deserialized back to matching elements and given back to Appium and especially to your web driver, to your driver agent. And that will take some time. And that will also get back to the point you were mentioning, is that a lot of people say, okay, AppSym is slow and especially with iOS, because if you would also look at the source code, kind of like the views you have on iOS and the views you would have on Android, you would see that Android is only giving you back the elements that are in the view. So if you would open the apps on desktop, you would open a screen of a native application, you would only get back the elements that are in the view. But if you would do that on iOS, you would get back all the elements that are in the application. And just take a simple example, you have a menu in your application and one of the menu items will bring you to a page that will hold all the countries in the world. Then you would see all those elements in your source. Then if you would then go back to what Appium is doing with XML, and especially when you're using XPath, it needs to serialize all the views into XML, use the query, then kind of like translate it back from the matched XML elements to a matching UI element. Well, and that will take a lot of time and that's something that a lot of people don't know. And yeah, hopefully also with this interview and also the talks I'm going to give at Selenium conference beginning of October, I hope I will give some more insight about how that part will work within Appium.

When I hear that, it's like, why would I use Appium then? Because it sounds like it's inherently part of the way Appium works, that would make it slower. Are there ways to work around that? If someone understands that that's how it's behaving?

Yes, yes, there are ways to work around it. If you, for example, would look at Android, as said, the views are, you will only get back the elements you would see in your.

So Android is already much faster in comparison to iOS, and that's mainly to do with the fact they have a useful dump window hierarchy, so that's easily for them to translate to XML. And if you would then look, for example, at iOS, there are some ways to kind of like speed up finding elements. And one of the Things is that if you would look into the documentation, and as I said, it's well hidden somewhere in the git repos of Appian, but you would see that you have different selectors for iOS which help you pinpoint the elements in the views instead of doing a translation by using XPath. One of the things is for example, the iOS predicate string. It's like a SQL query on the views. I did an investigation a few weeks ago and I found out that if you would have your script, if you have a simple app where you need to buy something and you need to do a checkout, and if I would create that based on XPath, it would take for example, 50 seconds on an iOS device, 16 seconds for the same flow on Android and iOS was 50 seconds based on XPath. If I would translate that to predicate streams, then I brought down my time to 23 seconds, meaning it's almost half. And the advantage of using a predicate string and you can even use IOSJ is that you're kind of like pinpointing to the right elements in the UI view. So you're not asking Appium to translate the views into xml, do the search, then deserialize it back and give you back an element because that's. That's really time consuming.

Very cool. One of the reasons why I actually do this podcast is because I learned too. So this is something I'm learning right now. So this is awesome. So I've heard of XPath, and when I talked to Jonathan Lips, he mentioned a way to get around this was to use accessibility IDs, but I've never heard of a predicate string approach. Is this just for JavaScript or is this new?

It's something you can use. You can use it with Java, Ruby, Python or whatsoever. Just to get back also to your remark about what Jonathan said, yes, accessibility IDs is the best thing to use if you're talking about native applications and also to create something to create a script that is cross platform. Basically, when you're using an Accessibility id, you're also pinpointing directly into the UI views. It's kind of like the same as what you do with predicate string or within a class chain of. For iOS, the only disadvantage you could have with for example an accessibility ID is that they are not added by the developers. And so then you're limited in finding your elements. And with a predicate string, for example, you can just look for a type of an element, for example an XML other element and xui view other element for iOS, but you can also verify if that element has certain attributes, like if it's visible or if it has a name or if it has a text or what. It's kind of like doing a simple query in a database only based on a single element in the UI view.

Awesome. And you're right. A lot of times people say, well, my developers don't add accessibility IDs. I say, well, tell them to ask them, but it's not always that easy. So why do you think people like maybe people do know about it, but I didn't know about it. Know about predicate string like you said, is it buried in the documentation or is it hard to implement?

No, I think it's buried in the documentation. When I did my investigation, I think it was two, three months just before I went on holiday.

And I also placed it on LinkedIn. I said, okay, for someone who's using Appium for five years and now did proper investigation in the documentation and I found out that I didn't know anything and about Appian because the documentation is there for, what is it, three or four years already that you can use predicate strings or class chains and there's even a preferred order of which one you should use and I can send you the link afterwards. You can also add it to the, to the podcast, but it's really useful. But it's quite old already and I think this is one of the. And don't get me wrong, one of the downsides of open source software is that there's so much out there in code that we don't always have the time to write proper documentation or proper blog post or whatsoever. That's what I also like about, for example, the Appian Pro newsletter of Jonathan and Jona is that they kind of like highlight some stuff each time from Appium that it's still in the documentation but nobody ever took the time to really write it down how to use it.

Absolutely. I definitely agree with you there. I'll also put in the show notes the image of your tweet that you have comparing these different approaches just so it's hard to translate audio ly, but I think you said XPath in your little example took about 59 seconds compared to the predicate string which took about 20 seconds. So you know, that may not seem like much, but over time those kind of time savings is huge, especially if you have a lot, a lot of tests. So it seems like it's going to be a big time saving approach for people if they're not using it. Already.

Exactly, exactly. And there's only one thing that the predicate strings in the class chain are very strict with. Using proper quotes, single quotes, double quotes.

So that might be a little bit harder for people, but just kind of like try it and make errors, because you will learn from the errors and see and debug your errors. But then if you're going to use it, you will really see the benefit of it. And I said the time between iOS and Android will be shorter, the difference in execution time.

Awesome. So I think this is a huge tip. So use predicate strings. Besides this, are there anything else in Appium you think that folks don't know that you think they should or should be more popular than what it is?

Yeah, I think one of the things they should. First of all, I would always advise people to go to the Appium documentation page. It's the Appium IO out of my head, Appium IO docs URL. Because there you can find which command will work with which version of Appium, but also if it's available for iOS and for Android. Because a lot of people think that if I would use, for example, Fingerprint, that it will be that you can use it for iOS and for Android, if you want to use, for example, sending a text message, then you can use it for both. But if you would look at the documentation there, you would already find that, for example, sending a text message can only be used for Android. And secondly, I think one of the that was one thing I found when I was doing the development of my image compare module for native applications is that there's a Settings API. And with the Settings API in Appium, you can kind of like influence the behavior of the Appium server. And also, for example, Android, as I said in the beginning, you don't get back all the elements you might need, you only get back the elements you see. And for my image compare module, I wanted to kind of like automatically determine the height of the status bar and of the toolbar in Android to kind of like block them out during comparison. Because especially in your status bar, you would have time, you would have icons, you would have your WI FI or your battery, and those can influence kind of like the image comparison. And during my research, it kind of like costed me a lot of time to figure it out. But then I found the settings API and then there's a option to tell Appian to give you back almost all the elements. And when I enabled that, during my execution, I got back the status bar element. I could determine the height of it and I could automatically block out the status bar, even though it was higher on device A and smaller on device B, it would do it automatically. So again, reading the documentation, especially on the Appium IO Docs page, it's really useful.

Very cool. So Wim, just to switch things up a little bit, you didn't mention image comparison. I think like I mentioned earlier is one of your latest sessions you've been giving is how you created a module for visual testing. So I thought we'd talk a little bit about that. When people hear about visual testing, a lot of times they think of, oh, I can only use a vendor solution for that and don't necessarily know there's libraries out there that maybe they could use for quick and dirty visual testing. So is that true? Is there open source solutions that people can use for visual testing? And if so, what are they?

There are tons of solutions for them. And if I would just look at what I've done, just look at my own work. I created a module for web applications for protector that can be used. I created a module for WebDriver that you can use to do image comparison and I also did it for WebDriver IE for native applications.

And I think if you would just look in the market, there are tons of open source libraries. Some need extra environmental dependencies, some don't need them. There's only one.

Big con about all the open source stuff and even about my own modules, is that the comparison you will get is only a pixel by pixel comparison. So it will give you a good insight if something changed. But it could also cause some flakiness, especially when there's a pixel movement, you cannot detect that or it will be harder with the open source tools, you need to write better software, better software engines or whatsoever to kind of like detect pixel movements or even text changes. But the open source tools will give you a very good impression if stuff change. If I just look at my own work and also refactoring the demo app we have at Sauce Labs.

I needed to migrate it from react native 57 to react native 60. And even though the version numbers are not that big, there were a lot of changes. And even in the UI my functional tests were passing. But when I just use my module, I could exactly see which component changed, resize got different margins or paddings. And it made it easier for me just during my development to just fix those issues before I release the new version. Maybe use it for that instead of detecting it always. Because then you might have some false positives.

Absolutely. So Are there any other use cases? Do you think this would be a good approach for a quick and dirty visual testing? Not necessarily like an enterprise wide kind of solution because like you mentioned, they can be pretty flaky.

I think other use cases, like I said, during development I think it will be very useful because if you then would need to go to a dashboard, verify all the things, I think that would be time consuming, especially during development process.

Yeah, and I can't think of any other use case you might be able to use it within enterprises. I think there are even some customers that still use open source libraries in their pipelines.

But the only thing they might be doing is kind of like setting a threshold and saying okay, if I'm allowed to have 1% difference. But then yeah, what, what do you accept as a difference? Because you only get back the mismatch percentage.

Very cool. And I think the two that you mentioned in your session was Pixel Diff and Resemble Js and it sounded like you went with Resemble JS for your your implementation.

Yeah, yeah, that's true.

Cool. So you also mentioned there are like a few components every visual testing module should have.

Do you remember what those are? I thought it'd be interesting to let people know what you know, how you approached it, and what modules of functionality you think is helpful when you're doing visual testing or creating a visual testing module.

Yeah, I think if you create a visual module testing module yourself, basically you don't start coding. That's always my advice. Just kind of like write down what you think you might need to have. And basically if you would look at that, you would need to have a safe screen and a compare screen. Sometimes it depends on what you make. If you do it for native applications, then you should. You might also want to have check elements and save an element. But if you do it, for example, for web applications, you can also create something like a full page screenshot and basically you create a screenshot, determine a position automatically scroll to the next point, create a screenshot and in the end if you created two, three or five screenshots, stitch them together, that could also be something. But yeah, it's not doable with native applications, but you can do it easily with web applications. I think those are like the four till six components you at least need to have in a visual comparison module.

I guess another thing is we talked about there are a lot of open source solutions, but there are also a lot of vendor based solutions. So there's two I could think of. Applitools is probably the Best known. There's also Percy, I think. Did Source Labs recently acquire a visual validation solution?

Yes, we acquired Screener.

Interesting. Does that work with Appium as well?

Currently it's only supporting web applications, but we have it on the roadmap to also support native applications and mobile applications.

Very cool. So it sounds like anytime I see companies acquiring other companies especially, it makes me realize the approach. Visual testing, it's been around. No, it seems to be gaining more and more popularity and people are gaining benefits from it. So I think it's something that if someone hasn't done yet, they definitely should look into because it does help you with a lot of things.

Exactly, exactly. And especially if we just. For example, it's some of the things I think. A few years ago I was in Romania during a Romania testing conference and I gave a workshop about testing beyond the default click paths. And I used this scenario from production with one of my customers where we kind of like, we had our pipeline, we had a flow, everything was working, the pipeline was green, we deployed the stuff to production and in the end we were called by DevOps and saying, okay, guys, there's one problem, the page we don't have a style. And then we found out that all our tests, all our Selenium tests were only checking functional flows.

And basically we were able to click on elements in the form, for example, even though it didn't have a style. And that's basically also what Selenium is doing. So I think the visual validation will really give you good value and it can do some extra assertions even for you, with only one command that you cannot do or can hardly do with Selenium or appcun?

Okay, Wim, before we go, is there one piece of actual advice you would give someone to help improve the Appium or automation testing in general testing efforts and let us know the best way to find or contact you.

If they want to contact me, they can do that on Twitter, on WS web creation, they can find me on LinkedIn, they can find me on Stack Overflow, they can find me on Gitter, they're tons of ways get up also to find me. I think the best advice I would give people is.

Try to be a tester and not try to be an automator. Try not to be too technical because I think all the people from all the people I've seen, the people that started as a QA engineer, manual QA engineer and started their career in that way, they, they have an advantage in comparison to people that directly start with automation because what I've seen is that people that are then too much focused on getting things done instead of looking at what is the business value of automating this yes or no? Because in the end if you have 100 test cases automated and you only have 10 test cases that give you real value, then you wait a long time for your build to succeed. And if you kind of like done that with the kind of like in the back of your mind, what you would do as a manual tester, you would only get those 10 or 20 test cases and you have a fast built and really valuable test cases. So yeah, don't dive into automation directly. First learn to do proper manual testing.

Thank you Wim for your automation awesomeness.

Links to everything of value we covered in this episode. Head on over to test guild.com alike automation a271 and while you there, make.

Sure to click on the Try it.

For Free Today link under the Exclusive Sponsor section to learn all about Source.

Labs awesome products and services. So that's it for this episode of.

The Test Guild Automation Podcast. I'm Joe and my mission is help you succeed with all your end to end automation awesomeness.

As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to test guild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.