Performance Testing iOS apps Using Appium and XCode Instruments with Sandeep Dinesh

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be talking with Sandeep Dinesh all about performance testing iOS apps with with Appium and Xcode Instruments. Sandeep has over or close to 17 years of experience as a software developer in tests, build and release and DevOps and he's really passionate about continuous testing and writing code to make software testing more effective and efficient. Really exciting to have him on the show because he's a tester, but he also does performance testing. So we're going to dive into a little bit why he thinks performance testing is important and how he incorporates it into his normal day to day automation testing plan. So really excited to have him on the show today. You don't want to miss this episode. Check it out.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SAS UI load testing tool, and LoadUI Pro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions performancetesting to learn more.

Hey Sandeep, welcome to the Guild.

Hi Joe, thanks for having me.

Awesome to have you on the show today. I saw a post on LinkedIn that grabbed my attention about you doing performance testing when mobile apps. Before we get into it though, is there anything I missed in your bio that you want the Guild to know more about?

Oh no, no. We are good. Okay, thanks for. I mean this was like a great introduction about me. Thank you so much.

Cool, awesome. So Cindeep, Cool, awesome. So Cindy, I guess you have a lot of experience with testing in general. How did you get into performance testing? Because a lot of people sometimes completely ignore it.

I've been mainly working as a software developer in test and most of the time the focus of most of the companies where I was working in they were like okay, we want to get to this much of test coverage. We to make sure we have a CI CD working fine. We want to know how the software is behaving functionally, be it the backend servers or even the clients Maybe like the browser client or maybe even the mobile clients. So I've been working in my current organization for the almost like one, one and a half years. We have a very small mobile app which is both in iOS and Android and so we have got to a point that we are reasonably stable in terms of the test automation functional coverage and that's where we really thought okay hey, what's the next thing which we can do product and the next thing obviously is that how best the app is performing in live systems. So and the biggest challenge is that now that we have iOS and people who have worked in iOS automation would know that it's really tough maybe going through things like APM or you know, all these open source tools because the platform in general is sort of, you know, limiting you in a lot of way not compared to comparable to something like an ADB or all the programmer friendly APIs and tools which the Android provides. So I was working on this feature for almost three, four months now and just to figure out hey, how to get my existing automation tests to also trap the performance matrices as well. It's been long time since I wrote something and I really wanted to make sure I knew for sure that this definitely going to be useful for people. That's why I wrote this piece of material.

Very nice. And I assume the reason why you wrote as well is probably wasn't a lot of information out there on how to do this. So was that one of the challenges?

Yeah, it was sort of because like we have some example in APPM Probe but that's actually like a Java thing but then how we adapt it more towards what you say Maybe in the JavaScript world or even more information about how the Xcode instrument works. Those are some things which I wanted to make sure because like the current example which we have in the documentation is something like okay, we can use some specific things like activity monitor, time profile and things like that which are xcode specific instruments out of the box. What I wanted to do was write something which was useful for JavaScript as well as you know we can have, we have the capability to sort of add custom profilers. Let's say for this example I tried and like this was like a real life scenario which I wanted to do like in my app I wanted to make sure that I want to check the gpu, what is the FPS because those are the things which really matter as a performance metrics for an mobile app. So all of those things how we can, you know, sort of measure using a single what is the test. So that's the reason that was my memo. Motivation to write more in a more sort of detailed way.

Nice. Do you also have an Android version of the app as well?

We do. And then for Android we probably can get these things. Most of this information using adb, because one good thing about Android as a platform is that you can basically use the shell commands itself, maybe like FPS or top and all those sort of things. And then it's basically about getting those information in some CSV format or something like that. And then basically there is another thing for F, specifically the Dumpsys command which we can use for the GPU profile and based on that we can get the stats. So that's what I'm saying. The point is the Android is much more programmer friendly. I'm not saying whatever we have in iOS is bad. But the thing is that getting it, you know, it's almost like Apple's ideology is like, okay, hey, this is what the customer would want. And this is actually a fabulous tool. The Xcode Instruments, you know, you get everything. The output file is literally. It has a lot of graphs, it has a lot of statistics, you can drill down further and further and all those sort of things. But then when it comes to sort of like attest automation requirement, like where I want to know the trend of how that software is like behaving, typically what you would do is that you run performance test periodically, you collect the stats and then you store the key stats somewhere maybe. And like eventually you can retrieve that data and see how the trend is going, whether we are improving performance wise or degrading. So those sort of things, we really want to have that sort of programming flexibility, sort of something which is still missing in iOS. I feel the tool is fabulous. But then it's more or less what you say. Android gives you all the options to write the code where you want it, I mean how you design it. But iOS is like, okay, hey, this is the API which you have. These are the format of data you will have and you probably need to work backwards from there. Those are my personal views.

Though this might be a dumb question. I'm thinking of web testing. People say, well, Chrome performs the same as Firefox. So does performance vary for the same exact app on iOS compared to an Android? Could the performance be significantly different?

I haven't sort of worked on specific native apps. So on both iOS and Android for comparison, we have a code base where both the platform independence is guaranteed using React native. So the React native by itself, it sort of Takes care of how the app even displays itself, you know, manifests itself in both iOS and app. And I, of course, I have noticed that there are market differences, the way the API underlying API behaves. And surely it's kind of, it's more or less like JVM per se. When you say React native, you basically have the same code which runs on different environments, whether it is the iOS native or the Android native. But then how does it actually translate? And that's where we were like, really interested. No, because in the past, when I was working in one of my previous organizations, we had specific native clients for both the platforms and the performance tests for them were different. And over here we have React native. So it adds another layer of sort of performance measurement. The level of control we have and the level of measurements we need to do are really critical in that sense.

So you mentioned a few monitors already, GPU and FPS. I don't know how many people are familiar with those terms. Maybe we could dive in a little bit more of the main monitors you look at and maybe what each one does. That would be awesome.

Sure. So GPU is the graphical processing capability of the mobile. And specifically we want to make sure that, let's say whether it is Android or iOS, the top frames per second, which is basically the amount of frames of animation or the app rendering, it's a measure of how the app renders itself. If there is what you say. Sometimes what happens probably if you have played games or things like that, you see that suddenly the game slows up and, you know, people say, okay, there is a lag, there's a drag. There's a lot of sort of terminology people say about the way the app display is being shown. So it's a measure of that, you know, so it's a measure of how many frames per second we generally we are currently showing. And the best case is 60. But then obviously human eyes don't need that. We ideally need to make sure that, you know, we have an on average around 24 sort of frames per second. If it's more, maybe 30, 32, it's actually pretty good. So we want to make sure that, you know, whenever we execute a particular performance scenario, you go through when you're playing a game or maybe you navigating from pages to pages. We want to make sure that the GPU information is captured and Xcode Instruments has a GPU measure which you can basically add to one of your instruments. And we want to make sure that it's not really pushed to the GPU percentages around less than 50%. That's actually pretty good. And then you want to make sure that you have on an average around 30 frames per second.

Nice. You did mention that Appium gives you some information and Apple itself gives you some information. Do you have a report at the end? They're able to merge both of them. So you get a good performance kind of overview.

No. So just to explain how Appium does this. So Appium as you can know, it's actually something like it's a small web server which sits in between the mobile app and you know, the client. Whichever test which you are like, you know, sort of running. But in order to get to the app, it needs a driver. So for iOS they have something called Xqui testserver. It's basically Xcode uitest server. It used to be a separate project in itself, but now I think over the last couple of years, since there was no new development happening, APM sort of, you know, has added it to one of their own open source projects. So this driver basically, so APM runs on a set of what you say W3C standard common commands. So that gets translated into what exactly what you want to do. Let's say when you say apm, click it actually gets translated into the driver, into the iOS code. So this driver has an integration with the Xcode instruments. So and they have a utility they have opened up. Like in an example I have given couple of JavaScript commands which says that okay, start the performance and you know, stop the performance and basically give what is the instrument which you are going to run. And then basically this format, the data format is sort of, it's like a buffer. You just need to. And like it collects all this data and then you. We just need to decode it to 64 bit and then it gets translated into a zip format which is more or less like the dot trace file, which you can sort of, you know, if you have Xcode installed on your Mac, you can basically double click on it. This file would render itself how the app was behaving and things like that. It's more or less like let's say if we didn't have apm, you just wanted to profile your app. So what you do is you open the project in Xcode and then you click on one of the top menu you click profile. So what it does, it builds it. When you are doing this, you can choose which are the things you're going to profile and then the app is sort of ready with that information. It has all those what do you say monitors triggered and attached to it? And basically you can do your operations and then stop this activity and then you have this information. It's basically Appium sort of provided you that sort of an information that okay, hey, programmatically, if you want to do this using appm, you can do this. You don't need to, you know, go through the Xcode method and you just need to, you know, it'll be like if it's like I don't know anything about this thing as a tester, probably as a test automation engineer probably will think, okay, hey, I'll try to automate the Xcode. I know opening the project and things like that. You don't need to do this, you can do this like this. So that is their strategy.

Very good. I'll have a link to your blog post in the show notes. Because it's just four steps, so it seems fairly easy. I don't know, could you talk at a high level what those four steps are to get someone started if they're like too hard for me or I don't know if I can do this?

Yeah, I mean that is the whole point that I really wanted them to sort of. It's that easy because like obviously we don't know much about Xcode instruments in general or maybe how easy to sort of to convert your existing because existing test automation code into a performance test using that. In my example, it's a simple login test which typically how you would be writing instead of that, I'm just saying, okay, add couple of hooks during whatever you want to really measure. You just need to start the performance and stop the performance once it's done. The only thing is that probably whatever template you are using for the Xcode instrument that needs to capture whatever you want. So. So the first thing which you need to do is you open up the Xcode, then add a blank sort of what you say, performance template. Add all the counters which you want. Maybe like you want activity monitor, you need the gpu, you need the allocations. Allocations are for memory, the disk I O and like one crucial thing which people don't realize is the thermal State of the iOS to be honest. Like Xcode really gives lot of good features, you know, because a lot of times you see that the mobile just heats up. Because when you're playing an app or then you are using an app for a long time, all of those things can be measured. So about how you can add all those instruments which you say together create a template. Set that aside, note down Its name and then create a small utility which would basically just call this APM script. What is the apm? The driver execute script. Start the performance with this and make sure that the PID which you are, the process ID which you are using is the current. Just that's something they have written in their code like that. If you don't pass that process ID PID is current, what would happen is that it would do the complete fonts. What you say data, you know, that will be huge, which you probably are not interested. So. And then once your test is done, probably you just need to make sure that, you know, stop the recording and like, you know, in your APM what you say logs, you can actually find that this many data, this much bytes have been written. You just need to make sure that that data is captured, converted to a format which you. And then you're good to go. It's that easy.

Very cool. So do you have like separate tests just for performance or do you instrument some of your tests that are normal functional tests and you just add extra features to capture the performance as well?

The current strategy we have is that we have some functional tests which are like single user use cases. Or maybe I'm not really logging into a folder or place like that where I don't have multiple data. But here maybe it could be just like, you know, if we open, maybe let's say we are trying to automate a feature like a gallery. So as a functionality functional test would be just, okay, I have just a single picture or a single video there. How quickly that gallery app opens up, that would be more or less like a functional test. But then for a performance test, I really want to make sure that, okay, maybe there are like hundreds of images there or hundreds of videos there, or if I scroll through, if I zoom through all the sort of things. So some of the tests, like the functional tests, we definitely want to make sure that they are single user or basic things like, you know, the performance can be, you know, it can, it can't go beyond that. So for those sort of things, we use bu reuse some of the functional tests. But otherwise it's basically like, you know, we have some extra things which probably that's a good place where you can. You really need to have a discussion with your app developers too. Maybe they know some part of the. Of course, in an ideal world we all have enough time and you have time to open and read the code which the developer has written. But of course that's where like, you know, good teams really, they Help each other and say that, okay, hey, these are the performance scenarios which I can, can foresee. How is it good enough? Or is it like, do you want extra features to be tested? Or even we can say, okay, hey, this is a small code, some trick which I have done probably. I'm not so sure the developer can themselves say that. And so the whole point is we want to try and avoid bugs rather than okay, hey, I caught this bug or like I didn't catch that bug, that sort of a thing.

Nice. How often do you run these tests then? Is it on every check in? Is it like every quarter? Like how often?

We are. The company where I'm working is very small and we are, we want to really push automation to the core and like we want to shift as much as possible to the left. Shift left is the. They're really, they really mean business there. So of course functional tests are run almost every day. These are part of the sort of release test where we have sort of, you know, I mean, even before our product actually goes out to the market, we want to make sure that, you know, we do performance tests where cycle. But then the ideal candidate for performance tests would be like more or less like a staging environment pre prod. We really are sure. But of course when you write smart code, we want to make sure that we can make sure that it runs on maybe on a developed branch too. So that's the strategy we are currently taking. But ideally it should be on a staging environment pre prod.

Very nice. So are there any lessons learned? Like maybe you started the project and then after you're like, oh, I should have done this differently or maybe we should have. I wish I knew this before I started this particular project.

That is actually sort of interesting because when I, I started my career as a performance test engineer, almost like way back in 2004, I was just doing load runner and like at that time when my team asked me to do I could. It didn't really make much sense, you know, how critical some of the steps which we have are. Things like that. Okay, so at that time back in the day I was doing server performance test, but out here when I did the app performance, I forgot some of this. Of course we are all humans, we. I mean most of the time I was doing development work and automation so of course I forgot some of the performance concepts. So I was doing scenarios one by one and most of the time I was just noticing that the app was. The performance of the app was really going bad and I was like really worried Whoa, that can't be true. Then obviously I understood that each of the scenarios are sort of different. We want to make sure that they are independent and we want to give the best chance for the app every time you are doing the test. So that was the lesson which I learned basically the next time what I did, I make sure that the system has all the hardware capability. Maybe it could be through rebooting, reinstalling the app, making sure that the connectivity is perfect, all those sort of things. So just to make sure that, you know, the performance data which you collect are really valid. Otherwise if I just took this data, the dev, they probably would panic, you know, oh, come on, it can't be that bad. So it was sort of a nice experience and probably, you know, it is good to learn that early rather than, you know, give all this data, publish it to the VP and they get the feed. Okay. Hey, the product is really bad. So we want to make sure that we are doing the right things at the right time.

Absolutely. So I guess another thing, I come from a similar background. I did mostly performance testing earlier in my career, but back then it was putting a load on a server. So you had multiple users and that was the performance, how well it did under load. When you're doing mobile testing, do you ever do concurrent users or is it always a one user type of performance test?

Yeah, I mean it purely depends on the app. Most of the app apps which we have, they don't support concurrency. Maybe Google Chrome is sort of an. Or any browser is sort of a concurrency example where you can have multiple tabs and each tab remains as a separate process in itself. But most of the apps, if you open one, maybe the second app, they don't generally allow because they somehow feel that maybe that's not right thing to do. But yes, the major concern when we are doing performance tests for apps are basically how much memory it is consuming, how much of CPU time it is being using, how many disk, you know, all sorts of things and all those sort of system IO usage. And of course the gpu. GPU is so critical. I can't quite emphasize, you know, because previously I was working for game testing as well and back in the day I was doing slots testing. Slots are like, you know, if the reels are really there, they became suddenly in slow motion, the customers are going to be super, super pissed. So we really want to avoid that. So the major thing we need to worry about this are these critical features. At the same time, we want to also test some things like, you know, when the app goes into background, how much of memory is it releasing back? Because in an ideal world, probably the app doesn't reside on its own. People have multiple apps, you know, they're saying multiple notifications. How does the app prioritize all those sort of things? You want to make sure that that customer experience is good. So that's where like you said, like it's. The major difference is that, you know, you don't have, most of the time you don't have multiple users. I'm sure maybe the browser, maybe the testing team, they really need to worry about, they might be doing that sort of, you know, concurrent multiple user scenarios. That will be interesting too. Maybe that's something which I will also be interested. Maybe I'll try it out. Let's see.

That'd be awesome. Have you back on the show for that. So do you actually test on a real device? Is it a simulator? Is it an emulator? Is it all the same? Does it matter what you're testing against?

Oh no, no, no. I forgot to mention that. Thanks for reminding Joe. So simulators are the sort of the real thing, but not the real thing. It's sort of what you say you're same, you know, test as well as the test driver are the same. It's sort of like not the perfect thing. It's a good thing probably which we use for test automation because the simulator or the emulator which we have, they generally don't come in with, you know, the real life problems of actually having a device farm or maybe a real device, as in it doesn't do, you know, system updates on the fly. It doesn't get notification from other apps and things like that. And also, at least for the Xcode thing, a lot of features like the GPU or even thermal stats or things like that, they are not really. The instruments would simply fail. It says that. Okay, I can't collect this because it's a simulator. But the good thing is if you have a decent one of the latest iOS devices using Xcode Instruments, you can get most of the information which you want.

Now this is a weird question. I've seen this a lot, but I've never asked anyone about it. You're using Codecep JS. What is Codecep JS?

Oh yeah, I'm not really a JavaScript expert. I mean it's one of the framework which it's more or less like the test ng the pytest or whatever we have. It has certain, what you say, good features. That's the reason our Team uses this. From my observation and usage, it's more or less like a set of instructions which you can give. Plus things like the automatic retrial, the skipping of certain scenarios and things like those are much more. It's handled in a much more cleaner way. It has a very good integration with most of the current technologies like Puppeteer, WebDriver, Selenium, even APM and Detox. In that sense, it's actually very good. And then the fact that our code is react native and we are a react native shop, it's actually works very well for our technology. And of course there are some things which I find that, you know, because coming from Java and Python background that, you know, some things are tough getting it done there because it just goes in like a marching order. Okay, I see this, I see this, I see this. And if I have seen maybe step A once, it won't do the step A again. I don't know. That's why it has a set of orders only that it follows that if there is a error, if suddenly it popped up, it just fails. It just retries the same stuff. But then you can't really write a sort of. In a loopy code which we, of course most of our test automation code is. We need to put a lot of bandages to make sure it works. Because you know, at the end of the day maybe if a login fails, we just don't, we can't just stop there. You know, we try to make sure that we get the best chance for the code to go forward. But in that sense, concept is very straight. But of course that's one. Another way of looking at it is saying that okay, if you have a bug there, what's the point in going further? Right. So it's like that.

Okay, Cindy, before we go, is there one piece of actionable advice you can give to someone to help them with their iOS performance testing efforts or strategies and what's the best way to find or contact you?

Sure. So definitely, if you are sort of using maybe APM or any technology, try to use this. There are other methods. True. By which we can get the same thing done. Because I was exploring further and recently I was working more on the webview automation. You know, because what happens is that, you know, you might not get that sort of info performance matrices from the app process. Sometimes, sometimes it gets delegated to the app webview, the Apple Webview which you have. So specifically that process needs to be instrumented and follow. That would definitely be a separate process. I mean I would say A separate blog in itself maybe. Currently I'm working on it. I can't really share much details because I am still learning and probably whatever I have shared with my developers, they're quite happy with what they have seen. It's basically you need to use at that point of time maybe the Xcode command line. By itself, of course in iOS I would say it has lesser flexibility. Let's say using adb. Let's say you can say ADB top command and then correct this stats but then they still have some, you know, some information are like not really exposed or maybe advertised that well. But then they still have some things like Xcode XCTrace, Xcode trace by which you can achieve the same thing and you probably can automate and you can attach. It's actually very neat as well. You know, you probably. You give the process name, not really the process id. You can give either of it and attach itself and it will still run on the same what you say instruments. So that's something which I would say probably don't probably lose hope and probably just try to do this manually. Listen to podcasts and talk to people. I would say ask these questions in web forums and things like that. Maybe that will definitely help because I for sure learned a lot from, I mean some of the other bloggers or people who actually worked on this and that's why most of us are like really happy to share this information and I know that this information is less but then maybe try to ask that question and maybe seek help. Probably we might. I can also learn of new things simply a lot of people would be able to share that information.

Awesome. Sandeep, the best way to find or contact you?

Sure. My LinkedIn or my Medium and my email address please. I have shared that information out there now, but a lot of people actually directly talk to me as well. Like maybe I did a lot of GitHub Action integration for iOS emulators and Android emulators. They did reach out to me and like I will try to. I'll try my best to help or at least point them in the right direction so that that you know, they try to solve that issue.

Thanks again for your performance testing awesomeness links to everything of value we covered in this episode. Head on over to testguild.com P267 and while you're there make sure to click on the try them both today link under the exclusive sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UiPro and if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.